Intelligent quality evaluation method and device for package design drawing, storage medium and equipment

Abstract

The invention discloses an intelligent quality evaluation method and device for a package design drawing, a storage medium and equipment. The method comprises the steps of obtaining a to-be-evaluated package design drawing; inputting the to-be-evaluated package design drawing into a preset multi-modal embedding model for vector retrieval, and outputting a strong correlation vector carrying correlation scores; restoring each strong correlation vector carrying the correlation score into a picture or text format to obtain an original package design drawing or an original design quality evaluation text and the correlation score thereof; inputting the original package design drawing or the original design quality evaluation text and the correlation score thereof into a preset correlation filter for enhancement processing to obtain cue words; and inputting the prompt words into a quality intelligent evaluation model which is finely adjusted in advance, and outputting a design quality evaluation text corresponding to the package design drawing to be evaluated. Therefore, by adopting the embodiment of the invention, the illusion and timeliness problems of a large model are solved, the accurate evaluation of the package design can be effectively realized, the efficiency is greatly improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and equipment for intelligent quality evaluation of packaging design drawings. Background Art

[0002] In the multimodal field, especially in the packaging design evaluation task, the traditional manual design feature aesthetic evaluation method is costly and inefficient. With the development of multimodal and large model technology, the use of artificial intelligence to automate packaging design evaluation has become an emerging development trend. This automated task involves converting packaging design drawings into text descriptions (picture-to-text problem) and generating corresponding quality evaluation texts.

[0003] In the related technologies, the current large models have shown strong performance in different tasks and fields and have supported multimodal tasks. However, there are still problems in dealing with tasks in specific fields, such as hallucinations and timeliness issues. In the process of the quality evaluation test task of packaging design, the following problems occurred when the packaging design drawings were input and the large model generated the evaluation: (1) When the questions were constructed for a long time, there would still be a situation where some parts of the answers were missing from time to time, such as only giving scores without reasons, not giving final conclusions, etc.; (2) Occasionally, the scoring would contradict itself; (3) The same packaging and questions would result in different scoring results when asked consecutively. The above situations made it impossible to accurately evaluate the packaging design. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and equipment for intelligent quality evaluation of packaging design drawings. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for intelligently evaluating the quality of a packaging design drawing, the method comprising: Get the packaging design to be reviewed; Input the packaging design drawing to be reviewed into a preset multimodal embedding model, perform vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and output a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed. The strongly correlated vector is selected by calculating the cosine similarity between the target vector and each original vector in the preset vector database. The target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; Each strongly correlated vector carrying a correlation score is restored to a picture or text format to obtain the original packaging design picture or the original design quality review text and its correlation score; Inputting the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and performing enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; The prompt words are input into a pre-fine-tuned quality intelligent review model, and the design quality review text corresponding to the packaging design drawing to be reviewed is output. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning the pre-trained multimodal large language model.

[0006] Optionally, perform vector search on the package design to be reviewed, including: Vectorize the packaging design drawing to be reviewed to obtain a target vector; Calculate the cosine similarity between the target vector and each original vector in the preset vector database; Obtain a preset number of original vectors in descending order of cosine similarity; Normalize the cosine similarity between each original vector and the target vector to obtain a correlation score between each original vector and the target vector; The correlation score between each obtained original vector and the target vector is marked on the corresponding original vector as a strong correlation vector carrying the correlation score.

[0007] Optionally, the preset multimodal embedding model includes a text encoder and an image encoder; Follow these steps to generate a preset vector database, including: Obtain each historical authentic packaging design drawing and its corresponding design quality review text; A text encoder is used to convert each historical real packaging design image into a vector representation; An image encoder is used to convert the design quality review text corresponding to each historical real packaging design image into a vector representation; The vector representation of each historical real packaging design drawing and the vector of the corresponding design quality review text are stored to obtain a preset vector database.

[0008] Optionally, the original packaging design image or the original design quality review text and its relevance score are enhanced, including: For the original packaging design drawing and its relevance score, the design quality review text is supplemented, and for the original design quality review text and its relevance score, the packaging design drawing is supplemented to obtain the image text pair of the packaging design drawing and the corresponding design quality review text and its relevance score; Delete the overlapping image-text pairs and their relevance scores; The remaining image-text pairs after deletion, their relevance scores, and the packaging design drawings to be reviewed are used to construct prompt words in order.

[0009] Optionally, follow the steps below to generate a pre-tuned quality intelligence review model, including: Obtain a preset first quantity of packaging design drawings; From a preset first number of packaging design drawings, a preset second number of packaging design drawings are obtained for manual annotation to obtain a plurality of first packaging design drawings with manually annotated design quality review texts, wherein the preset second number is less than the preset first number; Inputting the second packaging design drawings other than the preset second number of packaging design drawings among the preset first number of packaging design drawings into the preset large language model to obtain a plurality of second packaging design drawings with machine-annotated design quality review texts; Selecting, from among the plurality of second packaging design drawings of the machine-annotated design quality review text, a plurality of third packaging design drawings of the machine-annotated design quality review text that meet a preset quality standard; combining a plurality of first packaging design drawings with manually annotated design quality review texts and a plurality of third packaging design drawings with machine annotated design quality review texts into fine-tuning data; Based on the fine-tuning data, a pre-fine-tuned quality intelligent review model is generated.

[0010] Optionally, based on the fine-tuning data, a pre-fine-tuned quality intelligent review model is generated, including: Load pre-trained multimodal large language model; Input the fine-tuning data into the pre-trained multimodal large language model for model fine-tuning to obtain a pre-fine-tuned large language model; Among them, the loss function of the pre-trained multimodal large language model is the cross entropy loss function, and the cross entropy loss function is: ; Among them, p represents the true label probability distribution, q represents the label probability distribution predicted by the model, and i represents each word in the vocabulary; The model parameters of the pre-tuned quality intelligent evaluation model are:

[0011] in, The model parameters of the pre-fine-tuned quality intelligent assessment model, are the model parameters of the pre-trained multimodal large language model, To fine-tune the parameters;

[0012] in, and is a fine-tuning parameter, , represents the rank in the low-rank approximation, for the number of rows, for The number of columns.

[0013] Optionally, the method further includes: Get the model weights and configuration files of the pre-fine-tuned large language model; Stores model weights and configuration files.

[0014] In a second aspect, an embodiment of the present application provides a device for intelligently reviewing the quality of a packaging design drawing, the device comprising: An acquisition module is used to acquire the packaging design drawing to be reviewed; A vector retrieval module is used to input the packaging design drawing to be reviewed into a preset multimodal embedding model, perform vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and output a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed. The strongly correlated vector is selected by calculating the cosine similarity between the target vector and each original vector in the preset vector database. The target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; A restoration module is used to restore each strongly correlated vector carrying a correlation score into a picture or text format to obtain the original packaging design image or the original design quality review text and its correlation score; An enhancement processing module is used to input the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and enhance the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; The design quality review text output module is used to input prompt words into a pre-fine-tuned quality intelligent review model, and output the design quality review text corresponding to the packaging design drawing to be reviewed. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning the pre-trained multimodal large language model.

[0015] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0016] In a fourth aspect, an embodiment of the present application provides a device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0017] The technical solution provided by the embodiments of the present application may have the following beneficial effects: In an embodiment of the present application, quality intelligent review is performed by combining a preset multimodal embedding model and a pre-fine-tuned quality intelligent review model. The preset multimodal embedding model can reduce the "hallucination" phenomenon that the model generates is inconsistent with the facts or is inaccurate by retrieving relevant knowledge. The pre-fine-tuned quality intelligent review model is fine-tuned on data sets in specific fields. The model can better understand and adapt to the specific needs of packaging design evaluation tasks. The combination of the two can effectively realize the quality intelligent review of packaging design drawings, thereby improving the accuracy and efficiency of quality review and reducing labor costs.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 It is a flow chart of a method for intelligently evaluating the quality of a packaging design drawing provided in an embodiment of the present application; Figure 2 It is a process diagram of a processing process of a preset correlation filter provided by the present application; Figure 3 It is a schematic diagram of a prompt word provided by this application; Figure 4 It is a process diagram of a quality intelligent evaluation process of a packaging design drawing provided by the present application; Figure 5 It is a flowchart of a model training method of a pre-fine-tuned quality intelligent review model provided by the present application; Figure 6 It is a schematic diagram of the quality intelligent evaluation process of a packaging design drawing provided by this application; Figure 7 It is a structural schematic diagram of a quality intelligent evaluation device for a packaging design drawing provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0022] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0023] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0024] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0025] The present application provides a method, device, storage medium and equipment for intelligent quality evaluation of packaging design drawings to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, intelligent quality evaluation is performed by combining a preset multimodal embedding model and a pre-fine-tuned quality intelligent evaluation model. The preset multimodal embedding model can reduce the "hallucination" phenomenon that the model generates inconsistent with the facts or is inaccurate by retrieving relevant knowledge. The pre-fine-tuned quality intelligent evaluation model can better understand and adapt to the specific needs of packaging design evaluation tasks by fine-tuning on data sets in specific fields. The combination of the two can effectively realize the intelligent quality evaluation of packaging design drawings, thereby improving the accuracy and efficiency of quality evaluation and reducing labor costs. The following uses an exemplary embodiment for detailed description.

[0026] The following will be combined with the attached Figure 1 -Attached Figure 6 , a method for intelligent quality evaluation of packaging design drawings provided in an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a quality intelligent evaluation device for packaging design drawings based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0027] See also Figure 1, which is a flow chart of a method for intelligently evaluating the quality of a packaging design drawing provided in the embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps: S101, obtaining a packaging design drawing to be reviewed; The packaging design drawing to be reviewed refers to the visual representation of the packaging design that needs to be evaluated for quality, usually in the form of an image file, such as JPEG, PNG, etc. This design drawing contains all the visual elements of the packaging that needs to be evaluated for quality, such as color, shape, pattern, text, etc.

[0028] In some embodiments of the present application, a designer uploads a packaging design drawing and performs preprocessing on it, such as resizing, cropping, and denoising, to ensure that the quality of the image meets the requirements of the evaluation system, and obtains a packaging design drawing to be reviewed.

[0029] S102, inputting the packaging design drawing to be reviewed into a preset multimodal embedding model, performing vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and outputting a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed, wherein the strongly correlated vector is selected by calculating the cosine similarity between a target vector and each original vector in a preset vector database, wherein the target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; Among them, the preset multimodal embedding model is a pre-built model that can process multiple types of data (such as images and text) and convert these data into a unified vector representation for comparison and retrieval. Vector retrieval is a search technology in vector space that calculates the similarity between vectors to find the vector that best matches the query vector. Strongly correlated vectors are vectors that have a high correlation score with the query vector during vector retrieval.

[0030] In some embodiments of the present application, the designer uploads a new packaging design to the system, and the system converts the design into a format that can be processed by the model through an image processing module. The preset multimodal embedding model receives the packaging design to be reviewed as input for processing, and after the processing is completed, a strong correlation vector carrying a correlation score corresponding to the packaging design to be reviewed can be obtained.

[0031] The multimodal embedding model is implemented using CLIP, a multimodal pre-trained neural network whose core is to train data based on image text to learn the correspondence between images and text.

[0032] In some embodiments of the present application, the specific process of performing vector retrieval on the packaging design drawing to be reviewed includes: vectorizing the packaging design drawing to be reviewed to obtain a target vector; calculating the cosine similarity between the target vector and each original vector in a preset vector database; obtaining a preset number of original vectors in order of cosine similarity from large to small; normalizing the cosine similarity between each original vector obtained and the target vector to obtain a correlation score between each original vector obtained and the target vector; marking the correlation score between each original vector obtained and the target vector on the corresponding original vector as a strongly correlated vector carrying the correlation score.

[0033] Among them, the target vector is a numerical vector representation of the packaging design drawing to be reviewed after vectorization. The preset vector database is a database that stores multiple original vectors, which are pre-calculated and stored for comparison with the target vector. Cosine similarity is a measure of the similarity between two vectors, which is determined by calculating the ratio of the dot product of the two vectors and the product of their moduli. Normalization is the process of scaling data so that it falls within a specific interval (such as between 0 and 1). The correlation score represents the numerical value of the similarity between two vectors, which is obtained by normalizing the cosine similarity.

[0034] In some embodiments of the present application, the system uses a preset multimodal embedding model to extract features from the design drawing and converts the design drawing into a target vector that contains the key visual features of the design drawing. The system accesses a pre-built vector database that contains vector representations of all packaging design drawings in history. For the target vector, the system calculates the cosine similarity between it and each original vector in the database. The system selects a preset number (e.g., k) of original vectors that are most similar to the target vector in order of cosine similarity from large to small. The system normalizes the cosine similarity between each selected original vector and the target vector to obtain a correlation score that represents the similarity between them. The system marks the normalized correlation scores on the corresponding original vectors, which are now regarded as strongly correlated vectors carrying correlation scores.

[0035] Among them, the preset multimodal embedding model includes a text encoder and an image encoder.

[0036] In some embodiments of the present application, the specific process of generating a preset vector database includes: obtaining each historical real packaging design drawing and its corresponding design quality review text; using a text encoder to convert each historical real packaging design drawing into a vector representation; using an image encoder to convert the design quality review text corresponding to each historical real packaging design drawing into a vector representation; storing the vector representation of each historical real packaging design drawing and the vector of its corresponding design quality review text to obtain a preset vector database.

[0037] It should be noted that the top-k vectors with the strongest correlation may not be related to the input packaging diagram. In other words, there is no data in the database that is strongly related to the input packaging diagram. Or among the top-k vectors retrieved, the first n correlations are particularly high, while the rest are very low, resulting in a gap. At this time, if the retrieved data is directly fed into the large model, it is obviously not accurate enough. In order to make the model aware of this problem, more reference is made to information with strong correlation and less reference is made to information with weak correlation, so the correlation score attribute is introduced. That is, the previously calculated cosine similarity is saved and normalized as the correlation score.

[0038] S103, restoring each strongly correlated vector carrying a correlation score into a picture or text format to obtain an original packaging design image or an original design quality review text and its correlation score; Among them, strongly related vectors carrying relevance scores refer to vectors that have high similarity with the target vector during vector retrieval and comparison, and these vectors are accompanied by a numerical value representing the similarity, namely the relevance score. Restoration is the process of converting data in vector form back to its original form, such as converting from vector representation back to image or text.

[0039] In some embodiments of the present application, for each strongly correlated vector carrying a correlation score, the system needs to restore it to its original format. If the vector represents image data, the system will use inverse vectorization technology (for example, through a decoder) to convert the vector back to a picture format to obtain the original packaging design. The system annotates each restored picture or text with its correlation score. For pictures, the score may be displayed directly next to the picture or attached to the picture file as metadata; for text, the score may be attached to the beginning or end of the text as a rating or similarity indicator.

[0040] S104, inputting the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and performing enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; Among them, the preset relevance screening is a pre-set algorithm or model used to screen and process data carrying relevance scores to enhance the usefulness and accuracy of the data.

[0041] In some embodiments of the present application, the specific process of enhancing the original packaging design drawing or the original design quality review text and its relevance score includes: completing the design quality review text for the original packaging design drawing and its relevance score, completing the packaging design drawing for the original design quality review text and its relevance score, and obtaining image-text pairs of the packaging design drawing and the corresponding design quality review text and their relevance scores; deleting the overlapping image-text pairs and their relevance scores; and constructing prompt words for the remaining image-text pairs and their relevance scores after the deletion and the packaging design drawing to be reviewed in chronological order.

[0042] For example Figure 2 As shown, Figure 2 This is a schematic diagram of the processing process of a preset relevance filter provided by the present application. The top-k most relevant retrieved vectors can be decoded to restore them to the original image and text content, and the restored image is supplemented with its corresponding evaluation text label, and the restored text is supplemented with its corresponding packaging design drawing, forming an image-text pair of the packaging design drawing and the corresponding packaging design evaluation. The overlapping image-text pairs and the corresponding relevance scores are deleted, and the remaining image-text pairs and their relevance scores and the input packaging design drawing are used to construct prompt words in chronological order. The constructed prompt words are, for example, Figure 3 shown.

[0043] S105, inputting the prompt words into a pre-fine-tuned quality intelligent review model, and outputting the design quality review text corresponding to the packaging design drawing to be reviewed. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning the pre-trained multimodal large language model.

[0044] Among them, the prompt words are a series of carefully selected and constructed keywords or phrases, which contain relevant features and information of the packaging design drawings to be reviewed, and are used to guide or prompt the subsequent evaluation model to generate more accurate and specific evaluations. The pre-tuned quality intelligent evaluation model is a model that has been fine-tuned through a dataset in a specific field, enabling it to more accurately understand and evaluate the quality of packaging design. The design quality review text is a text description obtained after evaluating and analyzing the packaging design drawings, including descriptions, ratings, and suggestions for the pros and cons of the design.

[0045] In some embodiments of the present application, the constructed prompt words are input into a pre-fine-tuned quality intelligent review model. This model has been fine-tuned through a large amount of packaging design evaluation data, and can understand the key information in the prompt words and generate evaluations based on this information. The pre-fine-tuned quality intelligent review model receives the prompt words as input, and uses its learned knowledge and logic to generate a design quality review text for the packaging design drawing to be reviewed.

[0046] For example Figure 4 As shown, Figure 4 This is a process diagram of the intelligent quality review process of a packaging design drawing provided by the present application. First, the packaging design drawing is sent to a multimodal embedding model for vectorization processing. The model will search in a vector database, which is composed of a batch of real packaging design image data sets and corresponding evaluation texts after vectorization. The embedding model will retrieve the top-k vectors with the strongest correlation, restore the retrieved vectors to image or text format, and send them to the correlation filter together with their correlation scores. Since the pictures and evaluation texts will be extracted separately in the process of constructing the database through vectorization of the data document, the restored pictures will lack the corresponding evaluation text labels, and the restored text will lack the corresponding packaging design images. In the correlation filter, the missing parts should be filled and the overlapping image-text pairs should be deleted. Finally, the prompt words are constructed based on these images and their corresponding packaging design evaluation texts, their correlation scores, and the input packaging design images. Finally, the prompt words are input into the fine-tuned large language model to generate the final answer.

[0047] In an embodiment of the present application, the specific steps of generating a pre-fine-tuned quality intelligent review model include: obtaining a preset first number of packaging design drawings; obtaining a preset second number of packaging design drawings from the preset first number of packaging design drawings for manual annotation, and obtaining multiple first packaging design drawings with manually annotated design quality review texts, the preset second number being less than the preset first number; inputting the other second packaging design drawings except the preset second number of packaging design drawings from the preset first number of packaging design drawings into a preset large language model, and obtaining multiple second packaging design drawings with machine-annotated design quality review texts; from the multiple second packaging design drawings with machine-annotated design quality review texts, screening out multiple third packaging design drawings with machine-annotated design quality review texts that meet preset quality standards; combining the multiple first packaging design drawings with manually annotated design quality review texts and the multiple third packaging design drawings with machine-annotated design quality review texts into fine-tuning data; and generating a pre-fine-tuned quality intelligent review model based on the fine-tuning data.

[0048] For example Figure 6 As shown, Figure 6This is a schematic diagram of the quality intelligent review process of a packaging design drawing provided by the present application. First, the packaging design drawing is sent to a multimodal embedding model for vectorization and retrieved in a vector database. The vector database is composed of a batch of real packaging design image data sets and corresponding evaluation texts after vectorization. The embedding model will retrieve the top-k vectors with the strongest correlation. The retrieved vectors are restored to image or text format and sent to the relevance filter together with their relevance scores. Since the images and evaluation texts will be extracted separately in the process of vectorizing the data document to build the database, the restored images will lack the corresponding evaluation text labels, and the restored text will lack the corresponding packaging design images. In the relevance filter, the missing parts should be filled and the overlapping image-text pairs should be deleted. Finally, the prompt words are constructed based on these images and their corresponding packaging design evaluation texts, their relevance scores, and the input packaging design images. The prompt words are input into the fine-tuned large language model to output the final evaluation of the packaging design. In the preprocessing stage, the images and generated evaluation texts were further screened to select 2,000 high-quality images and corresponding packaging design labels with accurate evaluations and moderate word counts, resulting in a high-quality dataset of 10,000 images and corresponding evaluation labels that met our requirements. In the end, the real dataset and the artificial dataset constituted the data for fine-tuning in a ratio of 8:2. This dataset was used to fine-tune the large language model to adapt it to tasks in the field of packaging design.

[0049] In an embodiment of the present application, quality intelligent review is performed by combining a preset multimodal embedding model and a pre-fine-tuned quality intelligent review model. The preset multimodal embedding model can reduce the "hallucination" phenomenon that the model generates is inconsistent with the facts or is inaccurate by retrieving relevant knowledge. The pre-fine-tuned quality intelligent review model is fine-tuned on data sets in specific fields. The model can better understand and adapt to the specific needs of packaging design evaluation tasks. The combination of the two can effectively realize the quality intelligent review of packaging design drawings, thereby improving the accuracy and efficiency of quality review and reducing labor costs.

[0050] See also Figure 5 , which is a flow chart of a model training method for a pre-fine-tuned quality intelligent evaluation model provided in the embodiment of the present application. Figure 5 As shown, the method of the embodiment of the present application may include the following steps: S201, obtaining a preset first number of packaging design drawings; In some embodiments of the present application, the preset first number of packaging design drawings may be 100,000 real packaging design drawings.

[0051] S202, from the preset first number of packaging design drawings, a preset second number of packaging design drawings are obtained for manual annotation to obtain a plurality of first packaging design drawings with manually annotated design quality review texts, wherein the preset second number is less than the preset first number; In some embodiments of the present application, 8,000 of the 100,000 packaging design drawings are selected for manual annotation and packaging design evaluation, and gpt-4o generates packaging design evaluations for the remaining packaging design drawings. S203, inputting the second packaging design drawings other than the preset second number of packaging design drawings among the preset first number of packaging design drawings into a preset large language model to obtain a plurality of second packaging design drawings with machine-annotated design quality review texts; S204, selecting a plurality of third packaging design drawings of the machine-annotated design quality review texts that meet a preset quality standard from the plurality of second packaging design drawings of the machine-annotated design quality review texts; In some embodiments of the present application, 2,000 high-quality pictures and corresponding packaging design labels with accurate evaluations and a moderate number of words are manually screened out by considering aesthetic elements and factual factors.

[0052] S205, combining the plurality of first packaging design drawings with manually annotated design quality review texts and the plurality of third packaging design drawings with machine annotated design quality review texts into fine-tuning data; S206, generating a pre-fine-tuned quality intelligent evaluation model based on the fine-tuning data.

[0053] In some embodiments of the present application, the specific process of generating a pre-fine-tuned quality intelligent review model based on the fine-tuning data is as follows: loading a pre-trained multimodal large language model; inputting the fine-tuning data into the pre-trained multimodal large language model for model fine-tuning to obtain a pre-fine-tuned large language model; Among them, the loss function of the pre-trained multimodal large language model is the cross entropy loss function, and the cross entropy loss function is: ; Among them, p represents the true label probability distribution, q represents the label probability distribution predicted by the model, and i represents each word in the vocabulary; The model parameters of the pre-tuned quality intelligent evaluation model are:

[0054] in, The model parameters of the pre-fine-tuned quality intelligent assessment model, are the model parameters of the pre-trained multimodal large language model, To fine-tune the parameters;

[0055] in, and is a fine-tuning parameter, , represents the rank in the low-rank approximation, for the number of rows, for The number of columns.

[0056] In some embodiments of the present application, the method further includes: obtaining model weights and a configuration file of a pre-fine-tuned large language model; and storing the model weights and the configuration file.

[0057] Among them, the pre-trained multimodal large language model Qwen2-VL-7B. Qwen2-VL-7B is an advanced multimodal large language model that focuses on visual and language tasks and can understand and generate content based on images.

[0058] In an embodiment of the present application, quality intelligent review is performed by combining a preset multimodal embedding model and a pre-fine-tuned quality intelligent review model. The preset multimodal embedding model can reduce the "hallucination" phenomenon that the model generates is inconsistent with the facts or is inaccurate by retrieving relevant knowledge. The pre-fine-tuned quality intelligent review model is fine-tuned on data sets in specific fields. The model can better understand and adapt to the specific needs of packaging design evaluation tasks. The combination of the two can effectively realize the quality intelligent review of packaging design drawings, thereby improving the accuracy and efficiency of quality review and reducing labor costs.

[0059] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0060] See also Figure 7 , which shows a schematic diagram of the structure of a packaging design quality intelligent review device provided by an exemplary embodiment of the present application. The packaging design quality intelligent review device can be implemented as all or part of a device through software, hardware, or a combination of both. The device 1 includes an acquisition module 10, a vector retrieval module 20, a restoration module 30, an enhancement processing module 40, and a design quality review text output module 50.

[0061] An acquisition module 10 is used to acquire a packaging design drawing to be reviewed; A vector retrieval module 20 is used to input the packaging design drawing to be reviewed into a preset multimodal embedding model, perform vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and output a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed, wherein the strongly correlated vector is selected by calculating the cosine similarity between the target vector and each original vector in a preset vector database, wherein the target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; Restoration module 30, used to restore each strongly correlated vector carrying a correlation score into a picture or text format, to obtain the original packaging design image or the original design quality review text and its correlation score; The enhancement processing module 40 is used to input the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and perform enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; The design quality review text output module 50 is used to input the prompt words into the pre-fine-tuned quality intelligent review model, and output the design quality review text corresponding to the packaging design drawing to be reviewed. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning the pre-trained multimodal large language model.

[0062] It should be noted that the intelligent quality review device for packaging design drawings provided in the above embodiment only uses the division of the above functional modules as an example when executing the intelligent quality review method for packaging design drawings. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the intelligent quality review device for packaging design drawings provided in the above embodiment and the intelligent quality review method embodiment for packaging design drawings belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0063] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0064] In an embodiment of the present application, quality intelligent review is performed by combining a preset multimodal embedding model and a pre-fine-tuned quality intelligent review model. The preset multimodal embedding model can reduce the "hallucination" phenomenon that the model generates is inconsistent with the facts or is inaccurate by retrieving relevant knowledge. The pre-fine-tuned quality intelligent review model is fine-tuned on data sets in specific fields. The model can better understand and adapt to the specific needs of packaging design evaluation tasks. The combination of the two can effectively realize the quality intelligent review of packaging design drawings, thereby improving the accuracy and efficiency of quality review and reducing labor costs.

[0065] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the intelligent quality evaluation method for packaging design drawings provided by the above-mentioned various method embodiments.

[0066] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method for intelligent quality evaluation of packaging design drawings of the above-mentioned various method embodiments.

[0067] See also Figure 8 , which is a schematic diagram of the structure of a device provided in the embodiment of the present application. Figure 8 As shown, the device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0068] The communication bus 1002 is used to realize the connection and communication between these components.

[0069] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0070] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0071] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts within the entire device 1000, and executes various functions and processes data of the device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes operating devices, user interfaces and applications, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and can be implemented separately through a chip.

[0072] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing the operating device, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage device located away from the aforementioned processor 1001. As Figure 8 As shown, the memory 1005 as a computer storage medium may include an operating device, a network communication module, a user interface module, and an intelligent quality review application for a packaging design drawing.

[0073] exist Figure 8In the device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1001 can be used to call the quality intelligent review application of the packaging design stored in the memory 1005, and specifically perform the following operations: Get the packaging design to be reviewed; Input the packaging design drawing to be reviewed into a preset multimodal embedding model, perform vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and output a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed. The strongly correlated vector is selected by calculating the cosine similarity between the target vector and each original vector in the preset vector database. The target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; Each strongly correlated vector carrying a correlation score is restored to a picture or text format to obtain the original packaging design picture or the original design quality review text and its correlation score; Inputting the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and performing enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; The prompt words are input into a pre-fine-tuned quality intelligent review model, and the design quality review text corresponding to the packaging design drawing to be reviewed is output. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning the pre-trained multimodal large language model.

[0074] In one embodiment, when the processor 1001 performs a vector search on the packaging design drawing to be reviewed, the processor 1001 specifically performs the following operations: Vectorize the packaging design drawing to be reviewed to obtain a target vector; Calculate the cosine similarity between the target vector and each original vector in the preset vector database; Obtain a preset number of original vectors in descending order of cosine similarity; Normalize the cosine similarity between each original vector and the target vector to obtain a correlation score between each original vector and the target vector; The correlation score between each obtained original vector and the target vector is marked on the corresponding original vector as a strong correlation vector carrying the correlation score.

[0075] In one embodiment, when the processor 1001 generates a preset vector database, the processor 1001 specifically performs the following operations: Obtain each historical authentic packaging design drawing and its corresponding design quality review text; A text encoder is used to convert each historical real packaging design image into a vector representation; An image encoder is used to convert the design quality review text corresponding to each historical real packaging design image into a vector representation; The vector representation of each historical real packaging design drawing and the vector of the corresponding design quality review text are stored to obtain a preset vector database.

[0076] In one embodiment, when the processor 1001 performs enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score, the following operations are specifically performed: For the original packaging design drawing and its relevance score, the design quality review text is supplemented, and for the original design quality review text and its relevance score, the packaging design drawing is supplemented to obtain the image text pair of the packaging design drawing and the corresponding design quality review text and its relevance score; Delete the overlapping image-text pairs and their relevance scores; The remaining image-text pairs after deletion, their relevance scores, and the packaging design drawings to be reviewed are used to construct prompt words in order.

[0077] In one embodiment, when the processor 1001 generates a pre-fine-tuned quality intelligent review model, the processor 1001 specifically performs the following operations: Obtain a preset first quantity of packaging design drawings; From a preset first number of packaging design drawings, a preset second number of packaging design drawings are obtained for manual annotation to obtain a plurality of first packaging design drawings with manually annotated design quality review texts, wherein the preset second number is less than the preset first number; Inputting the second packaging design drawings other than the preset second number of packaging design drawings among the preset first number of packaging design drawings into the preset large language model to obtain a plurality of second packaging design drawings with machine-annotated design quality review texts; Selecting, from among the plurality of second packaging design drawings of the machine-annotated design quality review text, a plurality of third packaging design drawings of the machine-annotated design quality review text that meet a preset quality standard; combining a plurality of first packaging design drawings with manually annotated design quality review texts and a plurality of third packaging design drawings with machine annotated design quality review texts into fine-tuning data; Based on the fine-tuning data, a pre-fine-tuned quality intelligent review model is generated.

[0078] In one embodiment, when the processor 1001 generates a pre-fine-tuned quality intelligent review model based on the fine-tuning data, the processor 1001 specifically performs the following operations: Load pre-trained multimodal large language model; The fine-tuning data is input into the pre-trained multimodal large language model for model fine-tuning to obtain a pre-fine-tuned large language model; the loss function of the pre-trained multimodal large language model is a cross entropy loss function, and the cross entropy loss function is: ; Among them, p represents the true label probability distribution, q represents the label probability distribution predicted by the model, and i represents each word in the vocabulary; among them, The model parameters of the pre-tuned quality intelligent evaluation model are:

[0079] in, The model parameters of the pre-fine-tuned quality intelligent assessment model, are the model parameters of the pre-trained multimodal large language model, To fine-tune the parameters;

[0080] in, and is a fine-tuning parameter, , represents the rank in the low-rank approximation, for the number of rows, for The number of columns.

[0081] In one embodiment, the processor 1001 further performs the following operations: Get the model weights and configuration files of the pre-fine-tuned large language model; Stores model weights and configuration files.

[0082] In an embodiment of the present application, quality intelligent review is performed by combining a preset multimodal embedding model and a pre-fine-tuned quality intelligent review model. The preset multimodal embedding model can reduce the "hallucination" phenomenon that the model generates is inconsistent with the facts or is inaccurate by retrieving relevant knowledge. The pre-fine-tuned quality intelligent review model is fine-tuned on data sets in specific fields. The model can better understand and adapt to the specific needs of packaging design evaluation tasks. The combination of the two can effectively realize the quality intelligent review of packaging design drawings, thereby improving the accuracy and efficiency of quality review and reducing labor costs.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for intelligent quality evaluation of packaging design drawings can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program for intelligent quality evaluation of packaging design drawings can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0084] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for intelligently evaluating the quality of a packaging design drawing, characterized in that: The method comprises: Get the packaging design to be reviewed; Input the packaging design drawing to be reviewed into a preset multimodal embedding model, perform vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and output a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed, wherein the strongly correlated vector is selected by calculating the cosine similarity between a target vector and each original vector in a preset vector database, wherein the target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; Each strongly correlated vector carrying a correlation score is restored to a picture or text format to obtain the original packaging design picture or the original design quality review text and its correlation score; Inputting the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and performing enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; The prompt words are input into a pre-fine-tuned quality intelligent review model, and a design quality review text corresponding to the packaging design drawing to be reviewed is output. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning a pre-trained multimodal large language model.

2. The method according to claim 1, characterized in that Performing vector search on the packaging design drawing to be reviewed, including: Vectorizing the packaging design drawing to be reviewed to obtain a target vector; Calculating the cosine similarity between the target vector and each original vector in a preset vector database; Obtaining a preset number of original vectors in descending order of the cosine similarity; Normalizing the cosine similarity between each original vector obtained and the target vector to obtain a correlation score between each original vector obtained and the target vector; The correlation score between each obtained original vector and the target vector is marked on the corresponding original vector as a strongly correlated vector carrying the correlation score.

3. The method according to claim 1, characterized in that: The preset multimodal embedding model includes text encoder and image encoder; Follow these steps to generate a preset vector database, including: Obtain each historical authentic packaging design drawing and its corresponding design quality review text; Using the text encoder to convert each historical real packaging design image into a vector representation; The image encoder is used to convert the design quality review text corresponding to each historical real packaging design image into a vector representation; The vector representation of each historical real packaging design drawing and the vector of the corresponding design quality review text are stored to obtain a preset vector database.

4. The method according to claim 1, characterized in that: The enhancement processing of the original packaging design drawing or the original design quality review text and its relevance score includes: The original packaging design drawing and its relevance score are supplemented with the design quality review text, and the original design quality review text and its relevance score are supplemented with the packaging design drawing to obtain an image-text pair of the packaging design drawing and the corresponding design quality review text and its relevance score; Delete the overlapping image-text pairs and their relevance scores; The remaining image-text pairs after the deletion and their relevance scores as well as the packaging design drawing to be reviewed are used to construct prompt words in a sequential order.

5. The method according to claim 1, characterized in that Follow these steps to generate a pre-tuned quality intelligence model, including: Obtain a preset first quantity of packaging design drawings; From the preset first number of packaging design drawings, a preset second number of packaging design drawings are obtained for manual annotation to obtain a plurality of first packaging design drawings with manually annotated design quality review texts, wherein the preset second number is less than the preset first number; Inputting the other second packaging design drawings among the preset first number of packaging design drawings except the preset second number of packaging design drawings into a preset large language model to obtain a plurality of second packaging design drawings with machine-annotated design quality review texts; Selecting, from among the plurality of second packaging design drawings of the machine-annotated design quality review text, a plurality of third packaging design drawings of the machine-annotated design quality review text that meet a preset quality standard; combining a plurality of first packaging design drawings with manually annotated design quality review texts and a plurality of third packaging design drawings with machine annotated design quality review texts into fine-tuning data; Based on the fine-tuning data, a pre-fine-tuned quality intelligent evaluation model is generated.

6. The method according to claim 5, characterized in that The step of generating a pre-fine-tuned quality intelligent evaluation model based on the fine-tuning data includes: Load pre-trained multimodal large language model; The fine-tuning data is input into a pre-trained multimodal large language model for model fine-tuning to obtain a pre-fine-tuned large language model; wherein, The loss function of the pre-trained multimodal large language model is the cross entropy loss function, which is: ; Among them, p represents the true label probability distribution, q represents the label probability distribution predicted by the model, and i represents each word in the vocabulary; The model parameters of the pre-tuned quality intelligent evaluation model are: in, The model parameters of the pre-fine-tuned quality intelligent assessment model, are the model parameters of the pre-trained multimodal large language model, To fine-tune the parameters; in, and is a fine-tuning parameter, , represents the rank in the low-rank approximation, for the number of rows, for The number of columns.

7. The method according to claim 6, characterized in that The method further comprises: Get the model weights and configuration files of the pre-fine-tuned large language model; The model weights and configuration files are stored.

8. An intelligent quality evaluation device for packaging design drawings, characterized in that: The device comprises: An acquisition module, used to acquire the packaging design drawing to be reviewed; A vector retrieval module is used to input the packaging design drawing to be reviewed into a preset multimodal embedding model, perform vector retrieval on the packaging design drawing to be reviewed through the preset multimodal embedding model, and output a strongly correlated vector carrying a correlation score corresponding to the packaging design drawing to be reviewed, wherein the strongly correlated vector is selected by calculating the cosine similarity between a target vector and each original vector in a preset vector database, wherein the target vector is obtained by vectorizing the packaging design drawing to be reviewed, and each original vector is obtained by vectorizing each historical real packaging design drawing and its corresponding design quality review text; A restoration module is used to restore each strongly correlated vector carrying a correlation score into a picture or text format to obtain the original packaging design image or the original design quality review text and its correlation score; An enhancement processing module, used for inputting the original packaging design drawing or the original design quality review text and its relevance score into a preset relevance filter, and performing enhancement processing on the original packaging design drawing or the original design quality review text and its relevance score through the preset relevance filter to obtain a prompt word; The design quality review text output module is used to input the prompt words into a pre-fine-tuned quality intelligent review model, and output the design quality review text corresponding to the packaging design drawing to be reviewed. The pre-fine-tuned quality intelligent review model is obtained by fine-tuning the pre-trained multimodal large language model.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

10. A device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 7.

Images