Batch code identification method and device, equipment and medium

Abstract

The invention relates to the technical field of image detection, and discloses a batch code recognition method, device and equipment and a medium, and the method comprises the steps: collecting different package images, recognizing a type label and a batch code text in each package image, carrying out the image enhancement of the package images, and obtaining an enhanced image set; performing fine adjustment on the image visual model by utilizing the enhanced image set, the type label and the batch code text; extracting a semantic vector of each package image in the enhanced data set, and mapping the semantic vector to an input space of a large language model to obtain a spatial feature vector; building a batch code cue word of the large language model, and finely adjusting the large language model according to the batch code cue word, the spatial feature vector and the batch code text; training the fine-tuned image vision model and the fine-tuned large language model to obtain a multi-modal recognition model; and obtaining a to-be-identified packaging image, and identifying the to-be-identified packaging image by using the multi-modal identification model to obtain a batch code. According to the invention, the accuracy of batch code identification can be improved.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a batch code recognition method, apparatus, device, and medium. Background Technology

[0002] Multimodal models have been widely applied across various business sectors, facilitating automation upgrades and reducing manpower. In industries such as logistics, warehousing, and pharmaceuticals, the automatic identification of product specification batch codes (such as drug batch numbers and logistics codes) is a crucial link in cargo tracking, quality control, and supply chain management. With industry development, the increasing variety of packaging types and the complex and ever-changing imaging environments place higher demands on the efficiency, robustness, and scalability of batch code recognition. Therefore, to meet the precise needs of automated management in logistics, warehousing, and pharmaceuticals, it is necessary to innovate and optimize product specification batch code recognition technology to improve the accuracy of batch code identification.

[0003] Existing technologies use deep learning detection models such as YOLO and Faster R-CNN to locate packaging batch code regions, combined with traditional OCR modules to extract text, and require manual rule matching to assist in batch code region detection. This method requires re-collecting data and retraining the model when packaging is updated, resulting in high deployment costs and limited generalization ability; in complex environments, it relies on only a single visual feature, and interference from low light and reflections can easily introduce noise, causing missed regions or misjudged characters; at the same time, it does not explore the semantic relationship between product specifications and batch codes, lacks cross-modal understanding capabilities, and cannot utilize encoding rules to assist in recognition, resulting in low accuracy in batch code recognition. Summary of the Invention

[0004] This invention provides a batch code recognition method, apparatus, device, and medium to solve the problem of low batch code recognition accuracy.

[0005] Firstly, a batch code recognition method is provided, including: Packaging images of different target objects within the target object cluster are collected, packaging type labels and batch code text in each packaging image are identified, and the packaging images are enhanced to obtain an enhanced image set. Using a preset image visual model, textual semantic features are extracted from the packaging type label and the batch code text, and visual features of each packaging image are extracted from the enhanced image set. The textual semantic features and the visual features are fused to obtain a fused feature matrix. Using a preset loss function and based on the error in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix, the parameters of the image visual model are iterated until the error is lower than a preset threshold, thus obtaining a fine-tuned image visual model. Extract the image semantic vectors of each packaged image in the augmented dataset, and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors; The batch code extraction prompt words of the large language model are constructed. The batch code extraction prompt words are converted into text embedding vectors through word embedding processing. The spatial feature vector and the text embedding vector are fused to obtain cross-modal training vectors. Training sample pairs are constructed by combining the batch code text and input into the large language model. The loss value between the batch code output text of the large language model and the batch code text is calculated. The model parameters of the large language model are iterated in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model. The fine-tuned image vision model and the fine-tuned large language model are co-adapted and trained to obtain a multimodal recognition model; A packaging image to be identified is obtained, and the multimodal recognition model is used to extract prompt words based on the batch code to identify the packaging image, thereby obtaining the batch code within the packaging image.

[0006] Secondly, a batch code recognition device is provided, comprising: The image enhancement module is used to acquire packaging images of different target objects within the target object cluster, identify packaging type labels and batch code text in each packaging image, and enhance the packaging images to obtain an enhanced image set. The image visual model fine-tuning module is used to extract textual semantic features from the packaging type label and the batch code text using a preset image visual model, extract visual features from each packaging image in the enhanced image set, fuse the textual semantic features and the visual features to obtain a fusion feature matrix, use a preset loss function and calculate the error in the batch code region and batch code text content in each packaging image based on the fusion feature matrix to perform parameter iteration on the image visual model until the error is lower than a preset threshold, and obtain the fine-tuned image visual model; The image semantic vector mapping module is used to extract the image semantic vectors of each packaged image in the augmented dataset and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors. The large language model fine-tuning module is used to construct batch code extraction prompt words for the large language model, convert the batch code extraction prompt words into text embedding vectors through word embedding processing, fuse the spatial feature vectors and the text embedding vectors to obtain cross-modal training vectors, and construct training sample pairs with the batch code texts as input to the large language model. The module calculates the loss value between the batch code output text of the large language model and the batch code texts, and iterates the model parameters of the large language model in reverse according to the loss value until the loss value meets the preset loss value condition, thereby obtaining the fine-tuned large language model. The multimodal recognition model training module is used to perform collaborative adaptation training on the fine-tuned image vision model and the fine-tuned large language model to obtain the multimodal recognition model. The packaging image recognition module is used to acquire the packaging image to be recognized, and to use the multimodal recognition model to extract prompt words based on the batch code to recognize the packaging image to obtain the batch code in the packaging image.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the batch code recognition method described above.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the batch code recognition method described above.

[0009] The scheme implemented by the above-mentioned batch code recognition method, device, equipment and medium can collect packaging images and identify relevant labels and batch codes through the client. After image enhancement, the image visual model and large language model are fine-tuned respectively. Through semantic vector mapping, prompt word construction and collaborative adaptation training, a multimodal recognition model is formed. It integrates dynamic packaging adaptation and robust recognition innovation in complex environments, realizes rapid adaptation and anti-interference recognition of new packaging, improves adaptation efficiency and recognition accuracy, and can solve the problem of low batch code recognition accuracy of traditional methods in dynamic packaging and complex environments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a schematic diagram of an application environment for a batch code recognition method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a batch code recognition method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a batch code recognition device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The batch code recognition method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can acquire packaging images from the client and identify relevant labels and batch codes. After image enhancement, the image visual model and the large language model are fine-tuned. Through semantic vector mapping, prompt word construction, and collaborative adaptation training, a multimodal recognition model is formed. This model integrates dynamic packaging adaptation and robust recognition in complex environments, achieving rapid adaptation to new packaging and anti-interference recognition, improving adaptation efficiency and recognition accuracy. It can solve the problem of low batch code recognition accuracy in dynamic packaging and complex environments using traditional methods. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the batch code recognition method provided in this embodiment of the invention includes the following steps: S1. Collect packaging images of different target objects within the target object cluster, identify the packaging type label and batch code text in each packaging image, and perform image enhancement on the packaging images to obtain an enhanced image set.

[0015] In this embodiment of the invention, the target cluster can be a collection of various goods on a shelf that require batch code identification in fields such as logistics, warehousing, and pharmaceuticals; the packaging image is an image corresponding to the packaging of different goods within the collection.

[0016] In detail, image data of the logistics scene can be collected by image acquisition equipment (such as cameras pre-installed on or around the shelves), and packaging images of different products with different materials and forms, including normal lighting, low lighting, reflective scenes, cardboard boxes, plastic bags, and glass bottles, can be obtained. Approximately 50 images of each type of packaging can be collected to obtain the original packaging images.

[0017] Specifically, each packaging image in the collected original packaging images is labeled with four types of annotations: packaging region bounding box, packaging type label, batch code region, and batch code text, to obtain an image-text annotation dataset.

[0018] In this embodiment of the invention, the packaging type label is a text label manually pre-marked in the packaging area frame of each packaging image; the batch code text is the character content within the area read according to the marked batch code area position in the packaging image.

[0019] In detail, when identifying packaging type labels and batch code text in each packaging image, the region content of an image-text annotation dataset containing packaging type labels, batch code text, and corresponding region bounding boxes can be constructed based on the collected packaging images using a preset image analysis network model (such as U-net, Mask2Former, etc.) to achieve the recognition of packaging type labels and batch code text in each packaging image.

[0020] In this embodiment of the invention, the enhanced image set is typically performed on each packaged image in the image-text annotation dataset by performing three data augmentation processes: simulating reflection by randomly adding 2-5 specular noise points, simulating low illumination by adjusting the Gamma value to 0.4-0.7, and random perspective transformation with ±15° tilt and ±5% scaling. Three enhanced samples are generated for each original packaged image to keep the annotation information updated synchronously. The final dataset contains these enhanced samples.

[0021] In this embodiment of the invention, the step of enhancing the packaging image to obtain an enhanced image set includes: Identify the batch code area and the packaging area in the packaging image; Randomly add a preset range of highlight noise to the batch code area to obtain a reflective simulated image of the packaging image; The brightness parameter of the reflected simulated image is reduced to a random value point within a preset gamma value adjustment range to obtain a low-light simulated image; A preset angle tilt and preset scaling ratio are applied to the packaging area within the low-light simulated image to obtain a pose transformation image; The reflective simulated image, the low-light simulated image, and the posture transformation image are respectively bound and associated with the packaging type label and batch code text of the packaging image to obtain enhanced data samples; The augmented data samples are sorted in random order to obtain an augmented image set.

[0022] In detail, the batch code region is a specific area within the packaging area of ​​the packaging image that contains the batch code, and its bounding box coordinates are recorded; the packaging area is the area in the packaging image that contains the product packaging, and its corresponding bounding box is also marked; based on the acquired original packaging image, all packaging areas in the image are marked and their bounding box data is recorded, and at the same time, the area where the batch code is located is selected within each packaging area and the bounding box coordinates of that area are recorded, thereby completing the identification of the batch code region and the packaging area in the packaging image.

[0023] Specifically, the number of specular noise points within the preset range can be 2-5, the noise area is usually less than or equal to 10% of the batch code area, and the brightness value is between 200-255. The reflective simulation image is a packaging image with a noisy batch code obtained by adding the specular noise points within the preset range to the batch code area of ​​the packaging image. The corresponding packaging image is obtained from the image-text annotation dataset, and specular noise points are randomly added to each batch code area of ​​the packaging image according to the above preset range to obtain the reflective simulation image of the packaging image.

[0024] Furthermore, the preset gamma value adjustment range can be 0.4-0.7; the low-light simulation image is a corrected packaging image obtained by adjusting the gamma value of the reflective simulation image to the preset range through image gamma correction, thereby reducing the overall brightness; the reflective simulation image is obtained, and the gamma value is randomly selected within the preset 0.4-0.7 gamma value adjustment range and the brightness parameters are adjusted through image gamma correction technology to obtain the low-light simulation image.

[0025] Furthermore, the preset tilt angle is a tilt adjustment of ±15°; the preset scaling ratio is a scaling adjustment of ±5%; the pose transformation image is a transformed packaging image obtained by applying the preset tilt angle and preset scaling ratio to the packaging area within the low-light simulation image, simulating the actual tilt scene; the low-light simulation image is obtained, and the packaging area inside it is tilted by ±15° and scaled by ±5% respectively through perspective transformation, thereby obtaining the pose transformation image.

[0026] In addition, the enhanced data samples are new data samples formed by binding the simulated reflection image, simulated low light image, and pose transformation image with the corresponding original packaging image's packaging area bounding box, packaging type label, batch code area, and batch code text annotation information, respectively. Based on each packaging image in the image-text annotation dataset, the three images obtained by simulating reflection, simulating low light, and random perspective transformation are bound and associated with the packaging type label, batch code text, and corresponding packaging area bounding box and batch code area annotation information of the packaging image, respectively, to generate three enhanced data samples. Then, all the generated enhanced data samples are arranged in random order to obtain the enhanced image set.

[0027] S2. Using a preset image visual model, extract textual semantic features from the packaging type label and the batch code text, extract visual features from each packaging image in the enhanced image set, fuse the textual semantic features and the visual features to obtain a fused feature matrix, use a preset loss function and calculate the error in the batch code region and batch code text content in each packaging image based on the fused feature matrix, perform parameter iteration on the image visual model until the error is lower than a preset threshold, and obtain a fine-tuned image visual model.

[0028] In this embodiment of the invention, the preset image vision model can be a GroundingDINO model. When fine-tuning the image vision model, the visual encoding layer (such as a convolutional layer) of the preset image vision model can be frozen to prevent the fine-tuning dataset from being trained using the text encoder in the image vision model. When the preset training conditions are met, training is stopped to obtain the fine-tuned image vision model.

[0029] In this embodiment of the invention, the step of extracting textual semantic features from the packaging type label and the batch code text using a preset image visual model, extracting visual features from each packaging image in the enhanced image set, fusing the textual semantic features and the visual features to obtain a fused feature matrix, and using a preset loss function and based on the error in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix, iterating the parameters of the image visual model until the error is lower than a preset threshold, to obtain a fine-tuned image visual model, includes: The packaging type label and the batch code text are semantically encoded using the encoding layer in the preset image visual model to obtain text semantic features; Visual features of each package image in the enhanced image set are extracted using the convolutional layers in the preset image vision model; The fully connected layer is used to fuse the text semantic features and the visual features to obtain a fused feature matrix. Calculate the output batch code region location and output batch code text content of each packaging image in the enhanced image set based on the fusion feature matrix; A joint loss function is constructed using a preset first-type loss function and a preset second-type loss function; The first error between the position of the batch code region and the output batch code region is calculated based on the joint loss function, and the second error between the output batch code text content and the batch code text is calculated. The image visual model is iterated according to a preset gradient descent algorithm until both the first error and the second error are reduced to below the preset threshold, thus obtaining the fine-tuned image visual model.

[0030] In detail, the fully connected layer can be a learnable layer with a dimension of 768×4096, adapting to the input space requirements of large language models (such as the Vicuna-7B model); the textual semantic features are usually high-dimensional vectors obtained by semantically encoding packaging type labels (such as "pharmaceutical aluminum-plastic board" and "logistics cardboard box") and batch code text (such as "HK20230715"), which can reflect the semantic information of the text; the visual features can be 224×224 size feature maps output by packaging images in the augmented image set after processing by a preset image vision model (such as the GroundingDINO model), and then 768-dimensional object tokens (visual feature vectors) generated by dimensionality reduction operation through 1×1 convolution. The linear fully connected layer in the image vision model performs semantic encoding on the packaging type labels and batch code text, transforming the text information into a high-dimensional vector that adapts to the model input. At the same time, the packaging images in the augmented image set are input into the image vision model. After the model processes the images, it outputs a feature map, and then performs dimensionality reduction operation on the feature map through 1×1 convolution to extract the visual features of the packaging images.

[0031] Specifically, the fused feature matrix can be a high-dimensional matrix that simultaneously contains textual semantic information and image visual information, providing comprehensive feature support for subsequent batch code recognition. Textual semantic features and visual features are input together into a linear fully connected layer. The linear fully connected layer performs dimensionality adaptation and information integration on the two types of features, fusing the semantic association information from the text end with the visual localization information from the image end, and finally outputting the fused feature matrix.

[0032] Furthermore, the output batch code region location can be a set of bounding box coordinates of the region where the batch code is located in the enhanced image marked by the image visual model, typically in the format [[x1_1, y1_1, x2_1, y2_1], [x1_2, y1_2, x2_2, y2_2],… [x1_n, y1_n, x2_n, y2_n]]; the output batch code text content can be a set of characters corresponding to the batch code region predicted by the model based on the fused feature matrix, typically in the format [text_1, text_2, …text_n] (e.g., “HK20230715”). The fused feature matrix is ​​input into the image vision model. The image vision model uses the visual feature extraction capability preserved by the frozen visual encoder to locate the region where the batch code is located. The bounding box coordinates are calculated by combining the three fully connected prompt word mapping layers that have been unfrozen with text semantic information. At the same time, the batch code text annotation information in the fine-tuning dataset is matched to generate the batch code character content corresponding to each packaging image. Finally, the output batch code region location and output batch code text content are obtained.

[0033] Furthermore, the first type of loss function can be bounding box regression loss (IoU loss); the second type of loss function can be region classification loss (cross-entropy loss); and the joint loss function can be a loss calculation function that fuses these two loss functions with fixed weights, used for parameter optimization during the fine-tuning of the visual image model. During execution, the prediction error calculation for the batch code region location corresponding to the first type of loss function is first defined, and the prediction error calculation for the packaging type label and batch code region classification corresponding to the second type of loss function is defined. Then, a 1:1 weight ratio is set for both, and the two loss functions are fused by direct addition to construct the joint loss function. This joint loss function combines the batch code region coordinates [[x1_1, y1_1, x2_1, y2_1],…] and packaging type labels [class_label_1, class_label_2,…] in the fine-tuning dataset, simultaneously measuring the accuracy of region localization and classification, providing a unified basis for loss calculation for model parameter adjustment.

[0034] Furthermore, the first error can be the IoU loss value between the batch code region location predicted by the image vision model and the coordinates of the labeled batch code region in the image-text annotation dataset [[x1_1, y1_1, x2_1, y2_1], [x1_2, y1_2, x2_2, y2_2],… [x1_n, y1_n, x2_n, y2_n]]; the second error can be the cross-entropy loss value between the batch code text [text_1, text_2, … text_n] predicted by the image vision model and the labeled batch code text [text_1, text_2, … text_n] in the image-text annotation dataset; the joint loss function is the loss calculation function constructed during the fine-tuning of the image vision model, which is "bounding box regression loss (IoU loss) + region classification loss (cross-entropy loss)" (weight ratio 1:1). During execution, the location of the batch code region output by the image vision model and the coordinates of the labeled batch code region in the image-text annotation dataset are input into the bounding box regression loss part of the joint loss function to calculate the first error. At the same time, the batch code text output by the image vision model and the labeled batch code text in the image-text annotation dataset are input into the region classification loss part of the joint loss function to calculate the second error.

[0035] In detail, the preset gradient descent algorithm can be a gradient optimization algorithm commonly used in deep learning model training, used to adjust model parameters to reduce loss; the preset threshold can be a criterion such as IoU ≥ 0.8, used to measure whether the error of batch code region localization meets the standard. During execution, the first error and the second error are substituted into the preset gradient descent algorithm. The algorithm continuously adjusts the parameters of the three fully connected prompt word mapping layers of the unfrozen text encoder in the image vision model over 10 training cycles with a learning rate of 1e-5 and a batch size of 8. After each training cycle, the error is verified using a fine-tuned dataset. When the IoU corresponding to the first error is ≥ 0.8 and the second error drops to an acceptable range for model training, parameter adjustment is stopped, and the fine-tuned image vision model is obtained.

[0036] S3. Extract the image semantic vectors of each packaged image in the augmented dataset, and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors.

[0037] In this embodiment of the invention, the image semantic vector is a vector that carries image semantic information such as packaging shape, region boundaries, lighting conditions and / or material texture of the packaging image.

[0038] In detail, during the extraction operation, feature extraction processing is performed on the input packaging image to generate a feature map of size 224×224. A 1×1 convolution is used to perform a dimensionality reduction transformation operation on the generated feature map, resulting in a 768-dimensional vector. This vector is the image semantic vector of the corresponding packaging image in the augmented dataset.

[0039] In this embodiment of the invention, the preset large language model can be the Vicuna-7B large language model; the input space can be the 4096-dimensional vector space corresponding to the large language model when receiving input data; the spatial feature vector can be the 4096-dimensional high-dimensional object token vector obtained by mapping the 768-dimensional object token (image semantic vector) to the input space of the large language model through a learnable linear fully connected layer (768×4096 dimensions).

[0040] In this embodiment of the invention, mapping the image semantic vector to the input space corresponding to a preset large language model to obtain a spatial feature vector includes: Obtain the spatial dimension parameters of the preset large language model, and perform a linear transformation on the image semantic vector according to the spatial dimension parameters to obtain a high-dimensional feature vector; The high-dimensional feature vector is transformed to the same data dimension as the input space corresponding to the large language model using a preset fully connected function to obtain the transformed vector; Obtain the input format of the large language model, convert the data format of the transform vector into the input format, and obtain the input feature data; The input feature data is mapped to the input space of the large language model using a preset spatial mapping algorithm to obtain spatial feature vectors.

[0041] In detail, the spatial dimension parameter can be the input spatial dimension parameter of the large language model, typically corresponding to a 4096-dimensional value; the high-dimensional feature vector can be a feature vector adapted to the input space of the large language model, typically a 4096-dimensional high-dimensional object token vector. The execution process can be as follows: define the input spatial dimension parameter of the large language model, build a learnable linear fully connected layer with a dimension set to 768×4096, input the 768-dimensional object token into the linear fully connected layer, perform dimension transformation operations through the linear fully connected layer, and finally generate a 4096-dimensional high-dimensional object token vector. This high-dimensional object token vector allows the large language model to process both visual and textual information simultaneously.

[0042] Specifically, the transformation vector can be a feature vector that achieves semantic matching with the input space of a large language model, enabling the large language model to process both visual and textual information simultaneously. The execution process can be as follows: the high-dimensional feature vector is typically a 4096-dimensional high-dimensional object token vector. A learnable linear fully connected layer with a dimension of 768×4096 is constructed. One end of this linear fully connected layer matches the 768-dimensional object token output by the image visual model, and the other end matches the 4096-dimensional input space of the large language model. The 768-dimensional object token is input into the linear fully connected layer, where dimensional transformation and semantic mapping operations are performed, converting the 768-dimensional object token into a 4096-dimensional high-dimensional object token vector. This completes the semantic feature alignment and calibration between the high-dimensional feature vector and the input space of the large language model, ultimately yielding the transformation vector, which fully adapts to the input requirements of the large language model.

[0043] Furthermore, the input format can be a vector input requirement that the large language model can accept, usually a 4096-dimensional vector format standard; the data format can be the dimensional structure and presentation of the transformed vector, usually a 4096-dimensional high-dimensional object token vector structure; the input feature data can be feature data that conforms to the input format of the large language model and can be directly processed by the large language model, usually a 4096-dimensional vector data that has been transformed and adapted. The execution process can be as follows: First, clarify the input format of the large language model and confirm that its required vector dimension is 4096. The transformed vector is usually a 4096-dimensional high-dimensional object token vector after linear transformation. Construct a learnable linear fully connected layer with a dimension of 768×4096. Input the 768-dimensional object token output by the image vision model into this linear fully connected layer. The linear fully connected layer performs dimension transformation and format adaptation to adjust the data format of the 768-dimensional object token to the 4096-dimensional input format required by the large language model. Finally, the input feature data is obtained. This input feature data can be directly received by the large language model and used for subsequent batch code text training.

[0044] Furthermore, the preset spatial mapping algorithm can be a learnable linear fully connected layer operation algorithm, typically corresponding to a linear transformation operation logic with a dimension of 768×4096. The 768-dimensional object token is input into a learnable linear fully connected layer with a dimension of 768×4096. The linear fully connected layer performs dimension transformation and spatial mapping operations, accurately mapping the 768-dimensional object token to the 4096-dimensional input space of the large language model, obtaining a spatial feature vector. This spatial feature vector is typically a high-dimensional object token vector of 4096 dimensions, enabling the large language model to successfully receive and process visual and textual association information.

[0045] S4. Construct batch code extraction prompts for the large language model, convert the batch code extraction prompts into text embedding vectors through word embedding processing, fuse the spatial feature vectors and the text embedding vectors to obtain cross-modal training vectors, and construct training sample pairs based on the batch code texts and input them into the large language model. Calculate the loss value between the batch code output texts of the large language model and the batch code texts, and iterate the model parameters of the large language model in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model.

[0046] In this embodiment of the invention, the batch code extraction prompts are usually text with a fixed format, which is a type of expression that can clearly and explicitly state the task objectives of a large language model.

[0047] In detail, when constructing batch code extraction prompts for a large language model, you can first clarify the fixed format requirements of the prompts, and then, in conjunction with the task objective of batch code extraction, write text that can clearly instruct the large language model to perform batch code extraction operations in the image, such as "Extract the batch code in the image".

[0048] In this embodiment of the invention, the fine-tuned large language model is typically combined with a fine-tuned image vision model. It utilizes batch code extraction prompts, spatial feature vectors converted from packaging image feature maps, and batch code text. The large language model, after being fine-tuned and trained using the cross-entropy loss function, can achieve end-to-end intelligent recognition of packaging area localization and batch code text extraction. It also has the ability to quickly adapt to new packaging and has high robustness in batch code recognition under complex environments.

[0049] In this embodiment of the invention, the step of converting the batch code extraction prompt words into text embedding vectors through word embedding processing, fusing the spatial feature vectors and the text embedding vectors to obtain cross-modal training vectors, and constructing training sample pairs with the batch code text as input to the large language model, calculating the loss value between the batch code output text of the large language model and the batch code text, and iterating the model parameters of the large language model in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model, includes: The batch code extraction prompt words are converted into text embedding vectors with the same dimension as the spatial feature vectors through word embedding processing; The spatial feature vector and the text embedding vector are fused to obtain a cross-modal training vector. The training sample pairs for the large language model are constructed using the cross-modal training vectors and the batch code text. The training sample pairs are input into the large language model to obtain the batch code output text output by the large language model; Calculate the loss value between the batch code text and the batch code output text; The model parameters of the large language model are iterated backward based on the loss value between the batch code text and the batch code output text until the text difference loss value meets the preset loss value condition, thus obtaining the fine-tuned large language model.

[0050] In detail, the text embedding vector can be a vector whose dimension matches the spatial feature vector output by the image vision model after the batch code extraction prompt word has undergone word embedding processing, typically 4096 dimensions. A fixed-format recognition text prompt, i.e., the batch code extraction prompt word, such as "Extract the batch code in the image", is constructed. Word embedding processing is performed on this recognition text prompt. A learnable linear fully connected layer (768×4096 dimensions) is used to perform dimension transformation on the word embedding vector, ensuring that the transformed text embedding vector maintains the same dimension as the high-dimensional object token vector (4096 dimensions) obtained by transforming the 768-dimensional object token vector obtained by 1×1 convolutional dimensionality reduction of the 224×224-size feature map output by the image vision model. This completes the generation of the text embedding vector.

[0051] In this embodiment of the invention, the cross-modal training vector can be a vector used for training a large language model after feature fusion of the spatial feature vector output by the image visual model and the text embedding vector corresponding to the recognized text prompt. It typically has both visual and textual features.

[0052] In this embodiment of the invention, the step of fusing the spatial feature vector with the text embedding vector to obtain a cross-modal training vector includes: A preset cross-modal association algorithm is used to perform cross-modal association calculations on the spatial feature vector and the text embedding vector to obtain a feature association weight matrix; Based on the feature association weight matrix, the spatial feature vector and the text embedding vector are fused element-wise to obtain a dual-modal information fusion vector. The bimodal information fusion vector is linearly transformed using the first layer of the fully connected network in the large language model to obtain bimodal mapping features; The bimodal mapping features are nonlinearly transformed using a preset activation function, and the nonlinearly transformed bimodal mapping features are subjected to redundancy optimization to obtain an optimized feature vector. The optimized feature vector is subjected to average pooling to obtain the pooled vector; The pooled vector is subjected to a quadratic linear transformation using the second layer of the fully connected network of the large language model to obtain a cross-modal training vector.

[0053] In detail, the pre-defined cross-modal association algorithm can be a cross-modal association mechanism based on the collaboration of an image visual recognition model and a large language model. It typically combines learnable linear fully connected layers with semantic mapping logic and is the core algorithm for constructing a cross-modal association model of packaged features and text semantics. The feature association weight matrix can be generated by this algorithm and is a data matrix used to characterize the strength of feature associations in each dimension between the spatial feature vectors output by the image visual recognition model and the text embedding vectors adapted by the large language model. The values ​​of each element in the matrix directly correspond to the semantic fit of different feature dimensions of the two types of vectors. During execution, the constructed fixed-format recognition text prompts are converted into text embedding vectors that can be processed by the large language model. Based on the pre-defined cross-modal association algorithm, the visual feature extraction capability of the image visual recognition model and the semantic understanding capability of the large language model are integrated to perform dimensional matching between the spatial feature vectors and the text embedding vectors, calculate the semantic fit and feature relevance of the two in each feature dimension, and generate a feature association weight matrix characterizing the association strength in different dimensions.

[0054] Specifically, the dual-modal information fusion vector can be a high-dimensional vector that combines the spatial feature vector (4096-dimensional high-dimensional object token vector) output by the image visual recognition model with the text embedding vector (4096-dimensional) generated by the large language model. It also contains packaging visual features and batch code text semantic features. This vector can be directly input into the large language model to participate in subsequent batch code recognition tasks. During execution, the generated feature association weight matrix, 4096-dimensional spatial feature vector, and 4096-dimensional text embedding vector are first obtained to ensure consistency in feature dimensions. Then, for each element of the spatial feature vector, a multiplication operation is performed with the corresponding weight value in the feature association weight matrix to obtain the weighted feature value of each dimension of the spatial feature vector. At the same time, for each element of the text embedding vector, a multiplication operation is also performed with the corresponding weight value in the feature association weight matrix to obtain the weighted feature value of each dimension of the text embedding vector. Then, the spatial weighted feature value and the text weighted feature value in the same dimension are accumulated. After traversing all 4096 feature dimensions to complete the above element-by-element weighting and accumulation operations, a dual-modal information fusion vector is generated.

[0055] Furthermore, the first layer of the fully connected network can be a fully connected layer in the large language model that is responsible for receiving the bimodal information fusion vector and performing the initial linear transformation. It typically contains a learnable weight matrix and bias terms adapted to the 4096-dimensional input and is the initial core layer of the large language model for processing cross-modal fusion features. The bimodal mapping features can be feature vectors that retain the core information of the packaging visual features and batch code text semantic features after the linear transformation of the first layer of the fully connected network of the large language model, and are adapted to the subsequent semantic processing modules of the image visual recognition model. During execution, a 4096-dimensional bimodal information fusion vector is input into the first layer of the fully connected network of the large language model. The pre-set learnable weight matrix (the dimension of which matches the 4096-dimensional input of the bimodal information fusion vector and the output feature dimension of the layer) and bias terms of the layer are retrieved. A weighted operation is performed on each feature dimension of the bimodal information fusion vector (i.e., matrix multiplication operation between the bimodal information fusion vector and the weight matrix of the layer). Then, a bias superposition operation is performed on all the feature dimension results after the weighted operation. Through this series of linear transformation operations, the bimodal mapping features can be obtained.

[0056] Furthermore, the preset activation function can be a non-linear transformation function adapted to large language models, typically the ReLU or GELU function, used to enhance the non-linear expressive power of bimodal mapping features. The optimized feature vector can be a vector that, after non-linear transformation by the activation function, removes invalid and duplicate features through redundancy optimization operations, retaining the core visual features of packaging and the core semantic features of batch code text. The pooled vector can be a low-dimensional condensed feature vector obtained by performing average pooling on the optimized feature vector, which can simplify the feature dimensions while retaining key information. During execution, the bimodal mapping features are input into the preset activation function. The activation function performs non-linear transformation operations on each element of the bimodal mapping features to filter out effective feature signals. Then, redundancy optimization operations are performed on the non-linearly transformed bimodal mapping features (e.g., setting a feature importance threshold, retaining core feature dimensions above the threshold, and removing redundant and invalid feature dimensions below the threshold) to obtain the optimized feature vector. Afterward, average pooling is performed on all feature dimensions of the optimized feature vector, and the average value of all feature dimension values ​​is calculated. This average value is used as the feature value of the pooled vector, finally obtaining the pooled vector.

[0057] Furthermore, the second layer of the fully connected network can be a fully connected layer that follows the first layer of the fully connected network in a large language model. It typically contains a learnable weight matrix and bias terms that match the dimensions of the pooled vector and the target dimensions of the cross-modal training vector. It is the core layer that further optimizes the feature representation of the pooled vector to adapt to subsequent training tasks. During execution, the pooled vector obtained by the average pooling operation is input into the second layer of the fully connected network of the large language model. The pre-set learnable weight matrix and bias terms of this layer are retrieved, and matrix multiplication is performed on all feature dimensions of the pooled vector (i.e., the pooled vector is multiplied element-wise by the learnable weight matrix of this layer and then summed). Then, a bias superposition operation is performed on each feature dimension result after the matrix multiplication operation. Through this series of quadratic linear transformation operations, the cross-modal training vector is finally obtained.

[0058] In this embodiment of the invention, the training sample pair can be a training data unit formed by combining cross-modal training vectors and batch code text according to their correspondence, typically used for fine-tuning training of large language models; the batch code output text can be the batch code character content predicted by the large language model from the input image after training. The constructed training sample pair is input into the large language model, allowing the large language model to continuously optimize during the training process, ultimately generating batch code output text that conforms to the semantic association reference standard.

[0059] Furthermore, the text difference loss value can be calculated using the cross-entropy loss function, typically reflecting the degree of difference between the labeled batch code text and the batch code output text generated by the large language model. Network parameters are adjustable weights, biases, and other parameters in the large language model, which generally affect the model's prediction output. The preset loss value condition can be the number of training epochs reaching a set number or the loss value stabilizing, typically using a training parameter configuration of a learning rate of 5e-6, a batch size of 4, and 8 training epochs. The text difference loss value is calculated using the cross-entropy loss function. Based on the calculated text difference loss value, the network parameters of the large language model are adjusted using the backpropagation algorithm. For example, iterative training is performed with a set learning rate of 5e-6, a batch size of 4, and 8 training epochs. This parameter adjustment and training process is repeated until the preset loss value condition is met, at which point training stops, resulting in a fine-tuned large language model.

[0060] S5. Perform collaborative adaptation training on the fine-tuned image vision model and the fine-tuned large language model to obtain a multimodal recognition model.

[0061] In this embodiment of the invention, the multimodal recognition model typically involves first fine-tuning the image visual model with small samples using an image-text annotation dataset and an augmented dataset. This involves freezing the visual encoder, unfreezing the text encoder, and training the cue word mapping layer to obtain the fine-tuned image visual model. Then, by combining the spatial feature vector converted from the feature map output by this fine-tuned image visual model, the recognition text cue, and the fine-tuned dataset, a large language model is fine-tuned and trained. The resulting model, formed through visual-language cross-modal collaborative adaptation, can receive pre-processed full-image packaging and text cue such as "Extract the batch code in the image," accurately locate the packaging area and the batch code area, extract the batch code text, and perform semantic association verification. It adapts to new packaging types and complex environments such as low light and reflection, achieving efficient and robust batch code recognition.

[0062] In this embodiment of the invention, a multimodal recognition model is obtained by co-adapting and training a fine-tuned image visual model and a fine-tuned large language model. First, the image semantic vector output by the fine-tuned image visual model and the language feature vector output by the fine-tuned large language model are extracted. The image semantic vector is then dimensionally transformed to obtain an adapted image semantic vector with the same dimension as the language feature vector. Next, the adapted image semantic vector and the language feature vector are weighted and fused according to preset fusion weights to obtain a fused feature vector. The model parameters of the fine-tuned large language model are then iteratively optimized based on the fused feature vector to generate feature outputs corresponding to the optimized model parameters. These feature outputs are then input into a preset output adaptation layer constructed for the multimodal recognition model for encapsulation processing, resulting in the multimodal recognition model.

[0063] In detail, the image semantic vector can be a 224×224 feature map generated by the fine-tuned image vision model after processing the input packaging image and packaging type text label, and a 768-dimensional object token obtained by dimensionality reduction transformation through a 1×1 convolution operation; the language feature vector can be the feature vector output by the fine-tuned large language model after processing the fixed-format recognition text prompt. During execution, the packaging images and corresponding packaging type text labels from the fine-tuning dataset are input into the fine-tuning image vision model. The fine-tuning image vision model processes these inputs to generate a 224×224 feature map. A 1×1 convolution is used to perform dimensionality reduction transformation on this feature map to obtain a 768-dimensional object token (image semantic vector). At the same time, a fixed-format recognition text prompt is constructed and input into the fine-tuned large language model. The fine-tuned large language model processes the recognition text prompt and outputs a language feature vector. Then, a learnable linear fully connected layer with a dimension of 768×4096 maps the 768-dimensional object token to the input space of the large language model (input vector dimension 4096), completing the extraction and adaptation of image semantic vectors and language feature vectors.

[0064] Specifically, the adapted image semantic vector can be a 4096-dimensional feature vector obtained by dimensionality transformation of the 768-dimensional object token, maintaining the same dimension as the input vector of the large language model. During execution, the 224×224 feature map output by the fine-tuned image visual model is first obtained. A 1×1 convolution is then performed on this feature map to obtain a 768-dimensional image semantic vector. The 768-dimensional object token is then mapped to complete the dimension transformation, resulting in a 4096-dimensional adapted image semantic vector. This ensures that the dimension of this vector is consistent with the dimension of the input vector (4096 dimensions) of the fine-tuned large language model.

[0065] Furthermore, the fused feature vector can be a 4096-dimensional adapted image semantic vector and a 4096-dimensional language feature vector. After weighting operations according to corresponding fusion weights, the resulting 4096-dimensional feature vector combines visual positioning information and language semantic information. First, the previously determined fusion weights (used to balance the contribution ratios of the adapted image semantic vector and the language feature vector) are obtained. Then, the 4096-dimensional adapted image semantic vector after dimensionality transformation and the 4096-dimensional language feature vector output by the fine-tuned large language model are obtained. Each element of the adapted image semantic vector is multiplied by its corresponding fusion weight ratio, and each element of the language feature vector is multiplied by its corresponding fusion weight ratio. The elements of the two weighted vectors are summed one-to-one to complete the weighted fusion process, ultimately obtaining a 4096-dimensional fused feature vector. This vector can fully integrate the advantages of visual positioning and language semantic understanding, providing support for accurate batch code recognition.

[0066] Furthermore, the model parameters can be learnable parameters such as weights and biases in the fine-tuned large language model, covering adjustable parameters of modules such as text encoders and fully connected layers; the feature output can be a feature representation generated by the large language model after collaborative iterative optimization, which includes semantic and visual integration information related to batch code recognition after processing and fusing feature vectors; the preset output adaptation layer can be a layer structure built to match the final output requirements of the multimodal recognition model, used to complete the formatting and logical adaptation of the feature output. During collaborative iterative optimization, a 4096-dimensional fused feature vector is input into the fine-tuned large language model, while batch code text is introduced as the real label. The cross-entropy loss function is used to calculate the error between the model's prediction and the real label. Based on the error, the model parameters of the fine-tuned large language model are adjusted using the backpropagation algorithm. The learning rate is set to 5e-6, the batch size to 4, and the number of training epochs to 8. Each epoch is validated based on the fine-tuned dataset. After multiple rounds of parameter adjustment, the optimized large language model processes the fused feature vector to generate feature output. During encapsulation processing, a preset output adaptation layer is first constructed. This layer contains format conversion and confidence verification logic. The feature output is input into the preset output adaptation layer, and the feature output is converted into a fixed format of "batch code: [text content]" through in-layer operations. At the same time, the confidence judgment rule is integrated (if the confidence is ≥0.85, it is directly output). After encapsulation processing, a multimodal recognition model is obtained. This model can receive the pre-processed full-image package and the text prompt "Extract the batch code in the image". Image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector image semantic vector.

[0067] S6. Obtain the packaging image to be identified, and use the multimodal recognition model to extract prompt words based on the batch code to identify the packaging image to obtain the batch code in the packaging image.

[0068] In this embodiment of the invention, the packaging image to be identified can be a product packaging image related to which the specification and batch code need to be extracted in the fields of logistics, warehousing, and pharmaceuticals.

[0069] In detail, when acquiring the packaging image to be identified, first input relevant packaging images containing product specification batch codes from fields such as logistics, warehousing, and pharmaceuticals. Perform standardization operations on the input images, scale the images to 640×640 pixels, and adjust the image pixel values ​​to the [0,1] range.

[0070] In this embodiment of the invention, the batch code can be character content on the packaging of goods in the fields of logistics, warehousing, and pharmaceuticals. It typically includes types such as drug batch number, production date, and logistics code, follows specific coding rules, and has a semantic association with product specification information.

[0071] In this embodiment of the invention, the step of using the multimodal recognition model to extract prompt words based on the batch code to identify the packaging image to be identified, and obtaining the batch code within the packaging image to be identified, includes: The multimodal recognition model is used to identify the target batch code region within the packaging image to be identified, and the regional feature vector of the target batch code region is extracted. Based on the batch code, extract prompt words to determine the semantic association features between the region feature vector and the preset packaging feature attributes; Based on the semantic association features and the preset batch code encoding strategy, multiple candidate batch codes are generated corresponding to the target batch code region; The confidence level of each candidate batch code is identified, and the candidate batch code with the highest confidence level is taken as the batch code of the packaging image to be identified.

[0072] In detail, the target batch code region can be the area containing the batch code within the bounding box of the packaging area in the image to be identified. Typically, the bounding box coordinates of each region are recorded in the format [[x1_1, y1_1, x2_1, y2_1], ...[x1_n, y1_n, x2_n, y2_n]]. The region feature vector can be the vector obtained after feature extraction of the target batch code region by the multimodal recognition model. Using a fine-tuned image vision model within the multimodal model, the pre-processed packaging image to be identified, along with text prompts such as "batch code region," is input. The open vocabulary detection function of the fine-tuned image vision model locates the target batch code region in the packaging image. Then, the fine-tuned image vision model extracts features from the target batch code region to generate a 224×224 feature map. This feature map is then converted into a 768-dimensional region feature vector through 1×1 convolution, completing the region localization and feature vector extraction.

[0073] Specifically, the preset packaging feature attributes can be information related to the type of product packaging in fields such as logistics, warehousing, and pharmaceuticals, such as text labels containing packaging types like "pharmaceutical aluminum-plastic composite" and "logistics cardboard box," as well as corresponding product specification information. Semantic association features can be the semantic correspondence between the preset packaging feature attributes and batch code encoding rules, typically manifested as specific patterns followed by batch codes of the same product specification (e.g., drug batch numbers containing a specific prefix). By leveraging the cross-modal understanding capabilities of a multimodal recognition model, the semantic association between the region feature vector and the preset packaging feature attributes is analyzed, automatically matching the batch code encoding rules corresponding to the preset packaging feature attributes to complete the extraction of semantic association features.

[0074] Furthermore, the preset batch code encoding strategy can be a specific encoding rule corresponding to different specifications in logistics, warehousing, pharmaceuticals and other fields. It usually includes the rule that batch codes of the same type of product should include a specific prefix, such as the requirement that drug batch numbers contain a specific prefix. The candidate batch code can be multiple possible batch code texts that conform to the preset batch code encoding strategy. It is usually a combination of characters that conform to the encoding rules and is generated based on semantic association features. By utilizing the cross-modal understanding capability of the multimodal recognition model, based on the semantic association features between the parsed regional feature vector and the preset packaging feature attributes, the preset batch code encoding strategy is called to perform rule matching on the character-related information corresponding to the target batch code area, generate multiple character combinations that conform to the encoding strategy, and obtain multiple candidate batch codes corresponding to the target batch code area.

[0075] Furthermore, the confidence score can be a credibility metric attached to the candidate batch codes output by the multimodal recognition model. It typically reflects the degree of matching between the candidate batch code and the actual batch code in the packaging image to be recognized. The preset confidence threshold can be a credibility standard for judging whether a candidate batch code is qualified, such as setting it to 0.85. While generating multiple candidate batch codes, the multimodal recognition model simultaneously outputs the confidence score corresponding to each candidate batch code. The confidence score of each candidate batch code is compared with the preset confidence threshold of 0.85. Candidate batch codes with a confidence score of 0.85 or higher are selected as the final batch code of the packaging image to be recognized, presented in the format of "Product Specification Batch Code: [Text Content]".

[0076] As can be seen, in the above scheme, packaging images are collected and relevant labels and batch codes are identified. After image enhancement, the image visual model and the large language model are fine-tuned respectively. Through semantic vector mapping, prompt word construction and collaborative adaptation training, a multimodal recognition model is formed. It integrates the innovation of dynamic packaging adaptation and robust recognition in complex environments, realizes rapid adaptation and anti-interference recognition of new packaging, improves adaptation efficiency and recognition accuracy, and can solve the problem of low batch code recognition accuracy of traditional methods in dynamic packaging and complex environments.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] In one embodiment, a batch code recognition device 100 is provided, which corresponds one-to-one with the batch code recognition method described in the above embodiments. For example... Figure 3 As shown, the batch code recognition device 100 includes an image enhancement module 101, an image visual model fine-tuning module 102, an image semantic vector mapping module 103, a large language model fine-tuning module 104, a multimodal recognition model training module 105, and a package image recognition module 106. Detailed descriptions of each functional module are as follows: Image enhancement module 101 is used to acquire packaging images of different target objects within the target object cluster, identify packaging type labels and batch code text in each packaging image, and enhance the packaging images to obtain an enhanced image set; The image visual model fine-tuning module 102 is used to extract textual semantic features from the packaging type label and the batch code text using a preset image visual model, extract visual features from each packaging image in the enhanced image set, fuse the textual semantic features and the visual features to obtain a fusion feature matrix, use a preset loss function and calculate the error in the batch code region and batch code text content in each packaging image based on the fusion feature matrix to perform parameter iteration on the image visual model until the error is lower than a preset threshold, and obtain the fine-tuned image visual model; The image semantic vector mapping module 103 is used to extract the image semantic vectors of each packaged image in the augmented dataset and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors. The large language model fine-tuning module 104 is used to construct batch code extraction prompt words for the large language model, convert the batch code extraction prompt words into text embedding vectors through word embedding processing, fuse the spatial feature vectors and the text embedding vectors to obtain cross-modal training vectors, and construct training sample pairs with the batch code texts and input them into the large language model. The loss value between the batch code output text of the large language model and the batch code texts is calculated, and the model parameters of the large language model are iterated in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model. The multimodal recognition model training module 105 is used to perform collaborative adaptation training on the fine-tuned image vision model and the fine-tuned large language model to obtain a multimodal recognition model. The packaging image recognition module 106 is used to acquire the packaging image to be recognized, and to use the multimodal recognition model to extract prompt words based on the batch code to recognize the packaging image to obtain the batch code in the packaging image to be recognized.

[0079] In one embodiment, the image enhancement module 101, when performing image enhancement on the packaging image to obtain an enhanced image set, is configured to: Identify the batch code area and the packaging area in the packaging image; Randomly add a preset range of highlight noise to the batch code area to obtain a reflective simulated image of the packaging image; The brightness parameter of the reflected simulated image is reduced to a random value point within a preset gamma value adjustment range to obtain a low-light simulated image; A preset angle tilt and preset scaling ratio are applied to the packaging area within the low-light simulated image to obtain a pose transformation image; The reflective simulated image, the low-light simulated image, and the posture transformation image are respectively bound and associated with the packaging type label and batch code text of the packaging image to obtain enhanced data samples; The augmented data samples are sorted in random order to obtain an augmented image set.

[0080] In one embodiment, the image visual model fine-tuning module 102, when performing operations such as extracting textual semantic features from the packaging type label and the batch code text using a preset image visual model, extracting visual features from each packaging image in the enhanced image set, fusing the textual semantic features and the visual features to obtain a fused feature matrix, and using a preset loss function and based on the error in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix to iterate the parameters of the image visual model until the error is lower than a preset threshold, thus obtaining a fine-tuned image visual model, is used to: The packaging type label and the batch code text are semantically encoded using the encoding layer in the preset image visual model to obtain text semantic features; Visual features of each package image in the enhanced image set are extracted using the convolutional layers in the preset image vision model; The fully connected layer is used to fuse the text semantic features and the visual features to obtain a fused feature matrix. Calculate the output batch code region location and output batch code text content of each packaging image in the enhanced image set based on the fusion feature matrix; A joint loss function is constructed using a preset first-type loss function and a preset second-type loss function; The first error between the position of the batch code region and the output batch code region is calculated based on the joint loss function, and the second error between the output batch code text content and the batch code text is calculated. The image visual model is iterated according to a preset gradient descent algorithm until both the first error and the second error are reduced to below the preset threshold, thus obtaining the fine-tuned image visual model.

[0081] In one embodiment, the image semantic vector mapping module 103, when performing the operation of mapping the image semantic vector to the input space corresponding to a preset large language model to obtain a spatial feature vector, is used to: Obtain the spatial dimension parameters of the preset large language model, and perform a linear transformation on the image semantic vector according to the spatial dimension parameters to obtain a high-dimensional feature vector; The high-dimensional feature vector is transformed to the same data dimension as the input space corresponding to the large language model using a preset fully connected function to obtain the transformed vector; Obtain the input format of the large language model, convert the data format of the transform vector into the input format, and obtain the input feature data; The input feature data is mapped to the input space of the large language model using a preset spatial mapping algorithm to obtain spatial feature vectors.

[0082] In one embodiment, the large language model fine-tuning module 104, when performing word embedding processing to convert the batch code extracted prompt words into text embedding vectors, fusing the spatial feature vectors and the text embedding vectors to obtain cross-modal training vectors, and constructing training sample pairs with the batch code text as input to the large language model, calculates the loss value between the batch code output text of the large language model and the batch code text, and iterates the model parameters of the large language model in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model, is used for: The batch code extraction prompt words are converted into text embedding vectors with the same dimension as the spatial feature vectors through word embedding processing; The spatial feature vector and the text embedding vector are fused to obtain a cross-modal training vector. The training sample pairs for the large language model are constructed using the cross-modal training vectors and the batch code text. The training sample pairs are input into the large language model to obtain the batch code output text output by the large language model; Calculate the text difference loss value between the batch code text and the batch code output text; The model parameters of the large language model are iterated backward based on the text difference loss value until the text difference loss value meets the preset loss value condition, thus obtaining the fine-tuned large language model.

[0083] In one embodiment, the large language model fine-tuning module 104, when performing feature fusion of the spatial feature vector and the text embedding vector to obtain a cross-modal training vector, is used to: A preset cross-modal association algorithm is used to perform cross-modal association calculations on the spatial feature vector and the text embedding vector to obtain a feature association weight matrix; Based on the feature association weight matrix, the spatial feature vector and the text embedding vector are fused element-wise to obtain a dual-modal information fusion vector. The bimodal information fusion vector is linearly transformed using the first layer of the fully connected network in the large language model to obtain bimodal mapping features; The bimodal mapping features are nonlinearly transformed using a preset activation function, and the nonlinearly transformed bimodal mapping features are subjected to redundancy optimization to obtain an optimized feature vector. The optimized feature vector is subjected to average pooling to obtain the pooled vector; The pooled vector is subjected to a quadratic linear transformation using the second layer of the fully connected network of the large language model to obtain a cross-modal training vector.

[0084] In one embodiment, the packaging image recognition module 106, when performing the process of using the multimodal recognition model to extract prompt words based on the batch code to recognize the packaging image and obtain the batch code within the packaging image, is configured to: The multimodal recognition model is used to identify the target batch code region within the packaging image to be identified, and the regional feature vector of the target batch code region is extracted. Based on the batch code, extract prompt words to determine the semantic association features between the region feature vector and the preset packaging feature attributes; Based on the semantic association features and the preset batch code encoding strategy, multiple candidate batch codes are generated corresponding to the target batch code region; The confidence level of each candidate batch code is identified, and the candidate batch code with the highest confidence level is taken as the batch code of the packaging image to be identified.

[0085] This invention provides a batch code recognition device that acquires packaging images through a client and identifies relevant labels and batch codes. After image enhancement, the image visual model and large language model are fine-tuned respectively. Through semantic vector mapping, prompt word construction, and collaborative adaptation training, a multimodal recognition model is formed. It integrates dynamic packaging adaptation and robust recognition in complex environments, achieving rapid adaptation to new packaging and anti-interference recognition, improving adaptation efficiency and recognition accuracy. It can solve the problem of low batch code recognition accuracy of traditional methods in dynamic packaging and complex environments.

[0086] Specific limitations regarding the batch code recognition device can be found in the limitations of the batch code recognition method above, and will not be repeated here. Each module in the aforementioned batch code recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0087] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a batch code identification method on the server side.

[0088] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a batch code identification method on the client side.

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Packaging images of different target objects within the target object cluster are collected, packaging type labels and batch code text in each packaging image are identified, and the packaging images are enhanced to obtain an enhanced image set. Using a preset image visual model, textual semantic features are extracted from the packaging type label and the batch code text, and visual features of each packaging image are extracted from the enhanced image set. The textual semantic features and the visual features are fused to obtain a fused feature matrix. Using a preset loss function and based on the error in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix, the parameters of the image visual model are iterated until the error is lower than a preset threshold, thus obtaining a fine-tuned image visual model. Extract the image semantic vectors of each packaged image in the augmented dataset, and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors; The batch code extraction prompt words of the large language model are constructed. The batch code extraction prompt words are converted into text embedding vectors through word embedding processing. The spatial feature vector and the text embedding vector are fused to obtain cross-modal training vectors. Training sample pairs are constructed by combining the batch code text and input into the large language model. The loss value between the batch code output text of the large language model and the batch code text is calculated. The model parameters of the large language model are iterated in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model. The fine-tuned image vision model and the fine-tuned large language model are co-adapted and trained to obtain a multimodal recognition model; A packaging image to be identified is obtained, and the multimodal recognition model is used to extract prompt words based on the batch code to identify the packaging image, thereby obtaining the batch code within the packaging image.

[0090] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Packaging images of different target objects within the target object cluster are collected, packaging type labels and batch code text in each packaging image are identified, and the packaging images are enhanced to obtain an enhanced image set. Using a preset image visual model, textual semantic features are extracted from the packaging type label and the batch code text, and visual features of each packaging image are extracted from the enhanced image set. The textual semantic features and the visual features are fused to obtain a fused feature matrix. Using a preset loss function and based on the error in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix, the parameters of the image visual model are iterated until the error is lower than a preset threshold, thus obtaining a fine-tuned image visual model. Extract the image semantic vectors of each packaged image in the augmented dataset, and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors; The batch code extraction prompt words of the large language model are constructed. The batch code extraction prompt words are converted into text embedding vectors through word embedding processing. The spatial feature vector and the text embedding vector are fused to obtain cross-modal training vectors. Training sample pairs are constructed by combining the batch code text and input into the large language model. The loss value between the batch code output text of the large language model and the batch code text is calculated. The model parameters of the large language model are iterated in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model. The fine-tuned image vision model and the fine-tuned large language model are co-adapted and trained to obtain a multimodal recognition model; A packaging image to be identified is obtained, and the multimodal recognition model is used to extract prompt words based on the batch code to identify the packaging image, thereby obtaining the batch code within the packaging image.

[0091] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0094] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0095] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A batch code recognition method, characterized in that, include: Packaging images of different target objects within the target object cluster are collected, packaging type labels and batch code text in each packaging image are identified, and the packaging images are enhanced to obtain an enhanced image set. Using a preset image visual model, textual semantic features are extracted from the packaging type label and the batch code text, and visual features of each packaging image are extracted from the enhanced image set. The textual semantic features and the visual features are fused to obtain a fused feature matrix. Using a preset loss function and based on the error in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix, the parameters of the image visual model are iterated until the error is lower than a preset threshold, thus obtaining a fine-tuned image visual model. Extract the image semantic vectors of each packaged image in the augmented dataset, and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors; The batch code extraction prompt words of the large language model are constructed. The batch code extraction prompt words are converted into text embedding vectors through word embedding processing. The spatial feature vector and the text embedding vector are fused to obtain cross-modal training vectors. Training sample pairs are constructed by combining the batch code text and input into the large language model. The loss value between the batch code output text of the large language model and the batch code text is calculated. The model parameters of the large language model are iterated in reverse according to the loss value until the loss value meets the preset loss value condition to obtain the fine-tuned large language model. The fine-tuned image vision model and the fine-tuned large language model are co-adapted and trained to obtain a multimodal recognition model; A packaging image to be identified is obtained, and the multimodal recognition model is used to extract prompt words based on the batch code to identify the packaging image, thereby obtaining the batch code within the packaging image.

2. The batch code recognition method as described in claim 1, characterized in that, The process of enhancing the packaging image to obtain an enhanced image set includes: Identify the batch code area and the packaging area in the packaging image; Randomly add a preset range of highlight noise to the batch code area to obtain a reflective simulated image of the packaging image; The brightness parameter of the reflected simulated image is reduced to a random value point within a preset gamma value adjustment range to obtain a low-light simulated image; A preset angle tilt and preset scaling ratio are applied to the packaging area within the low-light simulated image to obtain a pose transformation image; The simulated reflective image, the simulated low-light image, and the pose transformation image are respectively bound and associated with the packaging type label and batch code text of the packaging image to obtain enhanced data samples; The augmented data samples are sorted in random order to obtain an augmented image set.

3. The batch code recognition method as described in claim 1, characterized in that, The process involves extracting textual semantic features from the packaging type label and the batch code text using a preset image visual model, extracting visual features from each packaging image in the enhanced image set, fusing the textual semantic features and the visual features to obtain a fused feature matrix, iterating the parameters of the image visual model using a preset loss function and based on the errors in the batch code region and batch code text content in each packaging image calculated from the fused feature matrix, until the error is lower than a preset threshold, to obtain a fine-tuned image visual model, including: The packaging type label and the batch code text are semantically encoded using the encoding layer in the preset image visual model to obtain text semantic features; Visual features of each package image in the enhanced image set are extracted using the convolutional layers in the preset image vision model; The fully connected layer is used to fuse the text semantic features and the visual features to obtain a fused feature matrix. Calculate the output batch code region location and output batch code text content of each packaging image in the enhanced image set based on the fusion feature matrix; A joint loss function is constructed using a preset first-type loss function and a preset second-type loss function; The first error between the position of the batch code region and the output batch code region is calculated based on the joint loss function, and the second error between the output batch code text content and the batch code text is calculated. The image visual model is iterated according to a preset gradient descent algorithm until both the first error and the second error are reduced to below the preset threshold, thus obtaining the fine-tuned image visual model.

4. The batch code recognition method as described in claim 1, characterized in that, The step of mapping the image semantic vector to the input space corresponding to a preset large language model to obtain a spatial feature vector includes: Obtain the spatial dimension parameters of the preset large language model, and perform a linear transformation on the image semantic vector according to the spatial dimension parameters to obtain a high-dimensional feature vector; The high-dimensional feature vector is transformed to the same data dimension as the input space corresponding to the large language model using a preset fully connected function to obtain the transformed vector; Obtain the input format of the large language model, convert the data format of the transform vector into the input format, and obtain the input feature data; The input feature data is mapped to the input space of the large language model using a preset spatial mapping algorithm to obtain spatial feature vectors.

5. The batch code recognition method as described in claim 1, characterized in that, The process involves converting the batch code extraction prompts into text embedding vectors through word embedding processing, fusing the spatial feature vectors and text embedding vectors to obtain cross-modal training vectors, and constructing training sample pairs using the batch code text as input to the large language model. The loss value between the batch code output text of the large language model and the batch code text itself is calculated. The model parameters of the large language model are then iterated backward based on the loss value until the loss value meets a preset loss condition, resulting in a fine-tuned large language model. This process includes: The batch code extraction prompt words are converted into text embedding vectors with the same dimension as the spatial feature vectors through word embedding processing; The spatial feature vector and the text embedding vector are fused to obtain a cross-modal training vector. The training sample pairs for the large language model are constructed using the cross-modal training vectors and the batch code text. The training sample pairs are input into the large language model to obtain the batch code output text output by the large language model; Calculate the loss value between the batch code text and the batch code output text; The model parameters of the large language model are iterated backward based on the loss value between the batch code text and the batch code output text until the text difference loss value meets the preset loss value condition, thus obtaining the fine-tuned large language model.

6. The batch code recognition method as described in claim 5, characterized in that, The step of fusing the spatial feature vector with the text embedding vector to obtain a cross-modal training vector includes: A preset cross-modal association algorithm is used to perform cross-modal association calculations on the spatial feature vector and the text embedding vector to obtain a feature association weight matrix; Based on the feature association weight matrix, the spatial feature vector and the text embedding vector are fused element-wise to obtain a dual-modal information fusion vector. The bimodal information fusion vector is linearly transformed using the first layer of the fully connected network in the large language model to obtain bimodal mapping features; The bimodal mapping features are nonlinearly transformed using a preset activation function, and the nonlinearly transformed bimodal mapping features are subjected to redundancy optimization to obtain an optimized feature vector. The optimized feature vector is subjected to average pooling to obtain the pooled vector; The pooled vector is subjected to a quadratic linear transformation using the second layer of the fully connected network of the large language model to obtain a cross-modal training vector.

7. The batch code recognition method as described in claim 1, characterized in that, The step of using the multimodal recognition model to extract prompt words based on the batch code to identify the packaging image to obtain the batch code within the packaging image includes: The multimodal recognition model is used to identify the target batch code region within the packaging image to be identified, and the regional feature vector of the target batch code region is extracted. Based on the batch code, extract prompt words to determine the semantic association features between the region feature vector and the preset packaging feature attributes; Based on the semantic association features and the preset batch code encoding strategy, multiple candidate batch codes are generated corresponding to the target batch code region; The confidence level of each candidate batch code is identified, and the candidate batch code with the highest confidence level is taken as the batch code of the packaging image to be identified.

8. A batch code recognition device, characterized in that, include: The image enhancement module is used to acquire packaging images of different target objects within the target object cluster, identify packaging type labels and batch code text in each packaging image, and enhance the packaging images to obtain an enhanced image set. The image visual model fine-tuning module is used to extract textual semantic features from the packaging type label and the batch code text using a preset image visual model, extract visual features from each packaging image in the enhanced image set, fuse the textual semantic features and the visual features to obtain a fusion feature matrix, use a preset loss function and calculate the error in the batch code region and batch code text content in each packaging image based on the fusion feature matrix to perform parameter iteration on the image visual model until the error is lower than a preset threshold, and obtain the fine-tuned image visual model; The image semantic vector mapping module is used to extract the image semantic vectors of each packaged image in the augmented dataset and map the image semantic vectors to the input space corresponding to the preset large language model to obtain spatial feature vectors. The large language model fine-tuning module is used to construct batch code extraction prompt words for the large language model, convert the batch code extraction prompt words into text embedding vectors through word embedding processing, fuse the spatial feature vectors and the text embedding vectors to obtain cross-modal training vectors, and construct training sample pairs with the batch code texts as input to the large language model. The module calculates the loss value between the batch code output text of the large language model and the batch code texts, and iterates the model parameters of the large language model in reverse according to the loss value until the loss value meets the preset loss value condition, thereby obtaining the fine-tuned large language model. The multimodal recognition model training module is used to perform collaborative adaptation training on the fine-tuned image vision model and the fine-tuned large language model to obtain the multimodal recognition model. The packaging image recognition module is used to acquire the packaging image to be recognized, and to use the multimodal recognition model to extract prompt words based on the batch code to recognize the packaging image to obtain the batch code in the packaging image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the batch code recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the batch code recognition method as described in any one of claims 1 to 7.

Images