Image recognition method and system

Abstract

The embodiment of the invention provides an image recognition method and system. According to the method, an image recognition system can obtain a to-be-recognized product image, first product information of a target product in the product image is recognized through a character recognition model, and second product information of the target product in the product image is recognized through a multi-modal recognition model. Furthermore, the image recognition system determines target product information of the target product based on the first product information and the second product information.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to an image recognition method and system. Background Technology

[0002] With the rapid development of internet technology, image recognition technology is being used more and more widely in e-commerce, smart security, industrial inspection and other fields.

[0003] Currently, common image recognition technologies mainly rely on deep learning models, especially large models based on massive pre-training. These large models can learn general visual features through massive amounts of data, achieving zero-shot or small-shot recognition. However, when encountering niche or novel products not covered in the training data, existing large models are prone to misclassification due to the "illusion" phenomenon. For example, they may misidentify rare items as common internet products (e.g., in drug recognition scenarios, if the image contains uncommon characters or the angle of the image is unusual, the large model may easily misidentify it as a common drug or even a non-drug), leading to unreliable recognition results.

[0004] Therefore, effectively suppressing the illusion problem of large models and improving the recognition ability of their products has become an important challenge in the field of image recognition technology.

[0005] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it can be considered prior art in this disclosure. Summary of the Invention

[0006] This specification provides an image recognition method and system that can be applied to scenarios requiring accurate identification of target products in product images.

[0007] In a first aspect, this specification provides an image recognition method applied to an image recognition model, the image recognition model including a character recognition model and a multimodal recognition model, the method including: acquiring a product image to be recognized; recognizing first product information of a target product in the product image through the character recognition model; recognizing second product information of the target product in the product image through the multimodal recognition model; and determining target product information of the target product based on the first product information and the second product information.

[0008] In some embodiments, determining the target product information of the target product based on the first product information and the second product information includes: if the first product information and the second product information are consistent, then determining the first product information or the second product information as the target product information; if the first product information and the second product information are inconsistent, then matching the first product information and the second product information with product information in a preset product library to determine the target product information based on the matching result.

[0009] In some embodiments, matching the first product information and the second product information with product information in a preset product library to determine the target product information based on the matching result includes: determining the distance between the first product information and the second product information; if the distance is less than or equal to a first distance threshold, determining the target product information in the product library based on a first preset rule; if the distance is greater than the first distance threshold, determining the target product information in the product library based on a second preset rule.

[0010] In some embodiments, determining the target product information in the product library based on a first preset rule includes: matching the first product information and the second product information with product information in the product library respectively; if either the first product information or the second product information is consistent with product information in the product library, then the consistent product information in the library is determined as the target product information; if neither the first product information nor the second product information is consistent with product information in the product library, then the distance between each piece of product information in the library and the first product information or the second product information is determined; if the distance is less than a second distance threshold, then the product information in the library is determined as the target product information; if the distances corresponding to each piece of product information in the library are all greater than the second distance threshold, then the second product information is determined as the target product information.

[0011] In some embodiments, determining the target product information in the product library based on the second preset rule includes: when there is product information in the product library that is consistent with the first product information, determining the consistent product information in the product library as the target product information; when there is no product information in the product library that is consistent with the first product information, determining that the identification has failed.

[0012] In some embodiments, the method further includes: if the first product information and the second product information are inconsistent, then fine-tuning the image recognition model based on the target product information.

[0013] In some embodiments, the method further includes: if the first product information and the second product information are consistent, and the first product information does not exist in the product library, then the first product information is added to the product library.

[0014] In some embodiments, before recognizing the first product information of the target product in the product image through the character recognition model, the method further includes: performing target product recognition on the product image to obtain a main body frame corresponding to the target product in the product image; obtaining a first main body image within the product image based on the main body frame, and recognizing the first main body image through the character recognition model or the multimodal recognition model.

[0015] In some embodiments, recognizing the first subject image using the multimodal recognition model includes: enlarging the subject frame based on a preset ratio to obtain an enlarged subject frame, and obtaining a second subject image within the product image based on the enlarged subject frame; the multimodal recognition model then recognizes the second subject image to obtain second product information of the target product.

[0016] In some embodiments, the image recognition model further includes a preset product detection module. Before performing target product recognition on the product image, the method further includes: using the preset product detection module to detect whether the target product exists in the product image.

[0017] In some embodiments, the method further includes: when the product image does not contain a main frame corresponding to the target product, the character recognition model recognizes the product image to obtain first product information of the target product; and the multimodal recognition model recognizes the product image to obtain second product information of the target product.

[0018] In some embodiments, the multimodal recognition model identifies the product image to obtain second product information of the target product, including: determining recognition failure when the multimodal recognition model identifies that the product image does not contain the target product.

[0019] In some embodiments, the multimodal recognition model is a multimodal recognition model that has been iteratively trained in advance based on a product library in the target domain.

[0020] In some embodiments, the multimodal recognition model is a lightweight multimodal recognition model.

[0021] In some embodiments, the step of identifying the first product information of the target product in the product image through the character recognition model includes: identifying character information contained in the product image through the character recognition model, wherein the character information includes: character content and detection boxes, wherein the detection boxes are used to mark the position and range corresponding to the characters in the product image; and obtaining the first product information based on the character information through detection, fusion, and localization algorithms.

[0022] Secondly, this specification also provides an image recognition system, including at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for performing image recognition; the at least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes the image recognition method described in the first aspect according to the instructions of the at least one instruction set.

[0023] As can be seen from the above technical solutions, the image recognition method and system provided in this specification, after the image recognition system acquires the product image to be recognized, firstly extracts the first product information of the target product in the image through a character recognition model, and then analyzes the image content in the product image through a multimodal recognition model to generate second product information. Subsequently, the image recognition system combines the first and second product information to determine the target product information. In the above method, the image recognition system effectively suppresses the illusion problem of large models in related technologies by fusing a dual verification mechanism of character recognition and multimodal recognition. The image recognition system can use the structured information provided by the character recognition model as a strong prior constraint, reducing the tendency of the multimodal model to misjudge unknown products. Simultaneously, the image recognition system can correct potential missed detections or errors in character recognition based on the visual semantic features supplemented by the multimodal model. Therefore, through system collaboration between the character recognition model and the multimodal recognition model, the image recognition system improves the robustness of product recognition while avoiding the risk of a single model misclassifying unfamiliar products as common categories, thereby enhancing the overall accuracy and reliability of product recognition in complex scenarios.

[0024] Other functions of the image recognition methods and systems provided in this specification will be partially listed in the following description. The inventive aspects of the image recognition methods and systems provided in this specification can be fully explained through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating an application scenario of an image recognition system provided according to an embodiment of this specification is shown.

[0027] Figure 2 A schematic diagram of the hardware structure of a computing device provided according to some embodiments of this specification is shown;

[0028] Figure 3 A schematic flowchart of an image recognition method according to an embodiment of this specification is shown;

[0029] Figure 4 A flowchart illustrating an image recognition method according to another embodiment of this specification is shown; and

[0030] Figure 5 A schematic flowchart of an image recognition method according to another embodiment of this specification is shown. Detailed Implementation

[0031] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0032] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0033] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0034] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0035] In this specification, "X includes at least one of A, B, or C" means that X includes at least A, or X includes at least B, or X includes at least C. That is, X can include only one of A, B, and C, or any combination of A, B, and C, as well as other possible content / elements. Any combination of A, B, and C can be A, B, C, AB, AC, BC, or ABC.

[0036] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0037] It should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0038] For ease of description, the terms that will appear later in this manual will be explained first.

[0039] Multimodal recognition models are commonly used in the field of artificial intelligence in natural language processing (NLP), especially referring to large machine learning models with a large number of parameters and computational resources. Multimodal recognition models are named for their huge number of parameters and complex network structures. They have powerful feature representation and feature understanding capabilities, and can better capture patterns and regularities in data when dealing with complex tasks. They are designed and trained to better understand and generate natural language.

[0040] Optical Character Recognition (OCR) refers to the process of analyzing and recognizing textual information from image files to obtain text and layout information. In this process, various types of images, such as handwritten and printed text, are converted into machine-encoded text. The core of OCR technology is recognizing textual information in images and converting it into a computer-editable and processable text format.

[0041] In image recognition scenarios, after receiving an image uploaded by a user, a large model typically identifies the objects contained within the image to obtain their names or other relevant information, and then returns the recognition results to the user. Therefore, in image recognition scenarios, it is crucial to ensure the accuracy of the recognition results to improve the user experience.

[0042] For example, image recognition scenarios can include image recognition in vertical fields such as drug recognition, department store recognition, and book recognition. Taking drug recognition as an example, users can obtain relevant information about the drug, such as its name, brand, dosage, precautions, and ingredients, by uploading a drug image to a large model or by taking a photo of the drug and uploading the photo. It should be understood that the above embodiments are merely illustrative, and the specific applicable scenarios of the image recognition methods provided in this specification can be flexibly adjusted according to user needs and are not limited to those given in the above embodiments.

[0043] The image recognition method provided in this specification, after acquiring an image of the product to be recognized by the image recognition system, first extracts the first product information of the target product from the image using a character recognition model, and then analyzes the image content in the product image using a multimodal recognition model to generate second product information. Subsequently, the image recognition system combines the first and second product information to determine the target product information. In this method, the image recognition system, through a dual verification mechanism that integrates the character recognition model and the multimodal recognition model, improves the accuracy of the recognition results while effectively suppressing the illusion problem that may occur with large models in related technologies. Through system collaboration between the character recognition model and the multimodal recognition model, the image recognition system enhances the robustness of product recognition while avoiding the risk of a single model misclassifying unfamiliar products as common categories, thereby enhancing the overall accuracy and reliability of product recognition in complex scenarios.

[0044] It should be noted that the above description of application scenarios is only one of the many usage scenarios provided in this specification. Those skilled in the art should understand that when the image recognition method and system provided in this specification are applied to other usage scenarios, their implementation methods and technical effects are similar.

[0045] Figure 1 A schematic diagram of an application scenario 100 of an image recognition system 130 provided according to an embodiment of this specification is shown.

[0046] like Figure 1 As shown, the image recognition system includes an image recognition model. After acquiring the product image to be recognized input by the user, the image recognition system uses the character recognition model and the multimodal recognition model within the image recognition model to identify the target product in the product image, thereby obtaining first product information and second product information corresponding to the target product. Furthermore, based on the first and second product information, the image recognition system determines and outputs the target product information.

[0047] Image recognition system 130 may be a computing system with a certain computing capability. Image recognition system 130 can execute the training method of the image recognition model provided in this specification, thereby training the image recognition model based on a training sample data set to obtain a trained image recognition model. Image recognition system 130 may store data or instructions for executing the training method of the image recognition model described in this specification, and may execute or be used to execute the data or instructions. Image recognition system 130 may include hardware devices with data information processing capabilities and the necessary programs required to drive the hardware devices.

[0048] The image recognition system 130 can be a single computing device or a cluster system composed of multiple computing devices. The data or instructions stored in the image recognition system 130 for executing the training method of the image recognition model described in this specification can adopt any form of system architecture. For example, layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This specification does not limit this.

[0049] It should be noted that all user data obtained in this manual has been authorized by the user and does not involve user privacy.

[0050] Figure 2 A schematic diagram of the hardware structure of a computing device 200 according to some embodiments of this specification is shown. This computing device 200 can be used as... Figure 1 The image recognition system 130 is described above. In some embodiments, when the image recognition system 130 employs a device cluster, the computing device 200 can be any one of the devices in the image recognition system 130.

[0051] like Figure 2As shown, the computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, the computing device 200 may further include an internal communication bus 210. In some embodiments, the computing device 200 may further include a communication port 250. In some embodiments, the computing device 200 may further include I / O components 260.

[0052] The internal communication bus 210 can connect different system components, including storage medium 230 and processor 220. I / O component 260 supports input / output between computing device 200 and other components.

[0053] Communication port 250 is used for data communication between computing device 200 and the outside world. For example, computing device 200 can connect to a network through communication port 250.

[0054] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the image recognition methods provided in this specification.

[0055] At least one processor 220 is communicatively connected to at least one storage medium 230 via an internal communication bus 210. The at least one processor 220 is used to execute at least one instruction set. When the system 130 is running, the at least one processor 220 reads at least one instruction set and executes the image recognition method provided in this specification according to the instructions of the at least one instruction set.

[0056] Processor 220 can execute all the steps included in the image recognition method. Processor 220 can be in the form of one or more processors. Processor 220 can issue execution instructions. Processor 220 may include one or more hardware processors, such as microcontrollers, microprocessors, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), central processing units (CPUs), graphics processing units (GPUs), physical processing units (PPUs), microcontroller units, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), advanced RISC machines (ARMs), programmable logic devices (PLDs), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0057] For illustrative purposes only, only one processor 220 is shown in the accompanying drawings of the computing device 200. However, it should be noted that the computing device 200 may also include multiple processors. Therefore, the operation and / or method steps disclosed herein may be executed by a single processor or by multiple processors in combination, as described herein. For example, if processor 220 of the computing device 200 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0058] Figure 3 A flowchart illustrating an image recognition method according to an embodiment of this specification is shown; the image recognition method P300 can be executed by an image recognition system 130, and the image recognition model includes: a character recognition model and a multimodal recognition model. For example... Figure 3 As shown, the method P300 provided in this specification may include S310-S370, wherein:

[0059] S310: Acquire the product image to be identified.

[0060] In some embodiments, the product image to be identified is uploaded by the user. It can be selected and uploaded by the user from the phone's photo album, or obtained and uploaded by the user after taking a picture, or selected and uploaded by the user from a preset image library. The specific method of uploading the product image can be flexibly adjusted according to the user's needs and is not limited to those included in the above embodiments.

[0061] S330: Identifies the first product information of the target product in a product image using a character recognition model.

[0062] In some possible embodiments, the character recognition model includes an optical character recognition model. Specifically, the character recognition model can be based on a preset recognition algorithm to determine the text detection box corresponding to each text in at least one text present in the product image, and determine the size of each text detection box, as well as the distance, confidence, and other indicators between each text detection box, in order to fuse the text detection boxes. Furthermore, based on the fused detection boxes (a fused detection box may contain multiple character information), the character recognition model obtains a preset number of first product information items ranked first by recording the relationships such as the size, position, and confidence of the fused detection boxes. Alternatively, the character recognition model may only record the first product information item ranked first. It should be understood that the above embodiments are merely illustrative examples, and the specific number and acquisition method of the first product information can be flexibly adjusted according to user needs and are not limited to the embodiments given above.

[0063] For example, the first product information may include: product model, product specifications, product parameters, or product brand, etc. The image recognition system identifies the first product information of the target product through a character recognition model, directly extracting accurate text information from the product image, providing reliable prior knowledge for the subsequent determination of target product information. The structured data extracted by the image recognition system through the character recognition model has clear semantic meaning and a fixed expression format, which can significantly improve the accuracy and interpretability of the subsequent image recognition system in determining the target product information.

[0064] In some embodiments, the image recognition system can perform target product recognition on a product image to obtain the subject bounding box corresponding to the target product in the product image. Based on the subject bounding box, a first subject image within the product image is obtained, and the first subject image is recognized using a character recognition model or a multimodal recognition model.

[0065] In this model, the character recognition model may identify one or more first subject images in the product image. The specific number of first subject images is related to the number of products contained in the product image, with each product having its corresponding first subject image. Taking a medicine as an example, if the product image contains only one medicine 1, the character recognition model will identify the first subject image corresponding to medicine 1. If the product image contains medicine 1, medicine 2, and medicine 3, the character recognition model will identify first subject image 1 corresponding to medicine 1, first subject image 2 corresponding to medicine 2, and first subject image 3 corresponding to medicine 3, respectively. The specific number of first subject images identified is related to the number of target products contained in the product image and is not limited to the examples given above.

[0066] Image recognition systems analyze product images, locating and selecting regions within the image that may contain the target product to generate precise subject bounding boxes. Subsequently, the system crops the product image using these subject bounding boxes to extract a primary subject image containing only the target product, thus eliminating background interference unrelated to the target product. This processing method provides high-quality input data for subsequent character recognition models, enabling them to more accurately locate and read text information on the target product's surface, avoiding interference from background noise and significantly improving the accuracy of character recognition results. Furthermore, this processing method helps multimodal recognition models focus on the product's inherent visual features, reducing the misleading influence of irrelevant environmental characteristics on the recognition results.

[0067] The aforementioned method of first-object image recognition based on the subject bounding box not only improves the accuracy and efficiency of subsequent recognition steps, but also enables the image recognition system to correctly handle product images containing multiple target products or complex backgrounds. Especially for small-sized products, partially occluded products, or products with colors similar to the background, the precise extraction of the first-object image significantly improves product recognition performance.

[0068] For example, a product image can contain one or more subject bounding boxes. When a product image includes multiple subject bounding boxes, first subject images corresponding to multiple products can be extracted based on each of the multiple subject bounding boxes. Subsequently, a character recognition model is used to recognize each of the multiple first subject images. Thus, by using different subject bounding boxes, multiple products can be recognized, enabling the image recognition system to have the capability of multi-product recognition.

[0069] In some embodiments, the image recognition system may employ a preset product detection module to detect whether a target product exists in a product image.

[0070] For example, the preset product detection module can be a lightweight detection model used to identify target products in product images. The preset product detection module first identifies the product image to determine whether it includes the target product. If the product image includes the target product, the preset product detection module then determines how many target products are included, and identifies the first bounding box corresponding to each target product. This detection process can be considered a pre-filtering step in product recognition, avoiding unnecessary depth calculations for images without products or low-quality images. Furthermore, using a lightweight detection model can significantly reduce the resource consumption of the image recognition system while ensuring detection accuracy, further improving the efficiency of image recognition. The preset product detection module can be a dedicated detection module corresponding to the target domain. It should be understood that the above embodiments are merely illustrative, and the specific selection of the detection module can be flexibly adjusted according to user needs or scenario requirements, and is not limited to the embodiments described above.

[0071] S350: Secondary product information for identifying a target product in a product image using a multimodal recognition model.

[0072] For example, the information categories contained in the first and second product information may be the same, but the content of the information within each category may be the same or different. During the process of obtaining the second product information through a multimodal recognition model in an image recognition system, the generalization and contextual understanding capabilities of the multimodal model allow it to handle unstructured product features that character recognition models cannot cover, especially for products lacking clear textual markings or with complex appearances. The multimodal recognition model effectively identifies the essential attributes of the target product through multidimensional feature fusion, significantly improving the image recognition system's ability to distinguish between novel products, variant products, or products with similar appearances.

[0073] In some embodiments, the multimodal recognition model can be a multimodal recognition model that has been pre-trained iteratively based on a product database in the target domain, such as a large multimodal model. For example, the multimodal recognition model can be pre-trained iteratively based on a drug database in the pharmaceutical field to enable it to recognize certain drugs. Alternatively, the multimodal recognition model can be pre-trained iteratively based on a book database in the book field to enable it to recognize certain books. It should be understood that the above embodiments are merely illustrative examples, and the specific method of pre-training the multimodal recognition model can be flexibly adjusted according to the current application scenario and is not limited to the embodiments described above.

[0074] For example, the multimodal recognition model can be a lightweight multimodal recognition model. Since image recognition systems determine target product information based on a dual-link (multimodal recognition model and character recognition model) approach, this method offers higher fault tolerance for the multimodal recognition model. Therefore, a lightweight multimodal recognition model can be used, significantly reducing computational resource requirements while maintaining multimodal recognition capabilities. In the embodiments of this specification, by selecting a lightweight multimodal recognition model, the high deployment cost of traditional large models is solved while retaining the core advantages of multimodal fusion. This achieves low latency and high concurrency requirements while maintaining the ability to recognize complex products.

[0075] In some embodiments, the image recognition system can enlarge the subject frame according to a preset ratio to obtain an enlarged subject frame, and then obtain a second subject image within the product image based on the enlarged subject frame. Subsequently, a multimodal recognition model recognizes the second subject image to obtain second product information of the target product.

[0076] In the above embodiments, the image recognition system first appropriately expands the subject bounding box according to a preset ratio to ensure that background information around the target product that may contain auxiliary recognition information is included in the processing range. Then, based on the expanded subject bounding box, a second subject image containing richer contextual information is obtained within the product image. This processing method allows the multimodal recognition model to obtain more comprehensive visual cues, especially for product images that are partially occluded, have poor shooting angles, or have important surrounding information. By analyzing the subject information and supplementary information in the second subject image, the accuracy of the recognition results can be significantly improved. At the same time, since the preset ratio is set in advance, it achieves the goal of increasing the amount of effective information while avoiding the introduction of too much irrelevant background noise, effectively improving the recognition capability of the multimodal recognition model and achieving a good balance between recognition accuracy and computational efficiency. For example, the preset ratio can be set to 1.2, but it can also be set to 1.1, 1.25, or any other ratio greater than 1. The specific preset ratio setting can be flexibly adjusted according to user needs and scenario requirements, and is not limited to the settings given in the above embodiments.

[0077] For example, when the product image does not contain the main bounding box corresponding to the target product, the character recognition model directly recognizes the product image to obtain the first product information of the target product. The multimodal recognition model directly recognizes the product image to obtain the second product information of the target product. It should be understood that when the multimodal recognition model obtains a product image, no enlargement processing is required; the multimodal recognition model can directly recognize the product image.

[0078] Specifically, when the product image does not contain the subject bounding box corresponding to the target product, the character recognition model will comprehensively recognize the complete product image to detect any text information that may be contained within it. Simultaneously, the multimodal recognition model extracts global features from the complete product image in parallel, generating recognition results by comprehensively analyzing the visual content and local features of the product image. This recognition method captures any product-related information that may exist in the product image through a comprehensive scan, improving the image recognition system's adaptability to unconventional shooting angles or complex background images. Secondly, the parallel processing of the two models (character recognition model and multimodal recognition model) complements each other: the character recognition model focuses on precise text extraction, while the multimodal recognition model maintains an understanding of the overall visual semantics. Furthermore, as a backup solution for the image recognition system, this method effectively avoids the problem of complete recognition interruption due to the failure to determine the subject bounding box, preventing false recognition and significantly improving the robustness and practicality of the image recognition system to enhance the accuracy of rejection.

[0079] If the product image does not contain the main bounding box corresponding to the target product, and the multimodal recognition model identifies that the product image does not contain the target product, then the image recognition model determines that the recognition has failed.

[0080] The above steps establish a dual-verification failure determination mechanism. When the preset product detection module fails to detect the main bounding box in the product image, the image recognition system scans and recognizes the entire product image using a character recognition model, while a multimodal recognition model performs a global analysis of the image content. If the multimodal model determines that there is no valid second product information in the product image, the recognition is considered a failure. This mechanism first determines whether the product image contains the target product through the preset product detection module, and then verifies the actual existence of the target product from a visual semantic perspective through the multimodal model. When the preset product detection module fails to detect the target product in the product image, and the multimodal model also identifies that the product image does not contain the target product, the image recognition system can be highly confident that the current product image does not contain an identifiable target product. This dual-verification mechanism effectively avoids misjudgments caused by a single recognition result from the preset product detection module, significantly improving the reliability of failure determination.

[0081] S370: Based on the first product information and the second product information, determine the target product information of the target product.

[0082] The aforementioned image recognition system determines the target product information based on the combined use of first and second product information. This fully leverages the complementary advantages of structured character information and unstructured semantic information. By establishing a dual verification mechanism, it effectively addresses the limitations of single-model recognition results and suppresses the illusionary product recognition problem caused by large models in related technologies. This collaborative recognition method not only improves the accuracy of product recognition but also significantly enhances the robustness of the image recognition system, increasing the accuracy of the recognition results and greatly reducing the risk of misjudgment.

[0083] In some embodiments, if the first product information and the second product information are consistent, it indicates that the recognition results of the character recognition model and the multimodal recognition model can be mutually verified, demonstrating high reliability. In this case, the image recognition system can directly determine either the first product information or the second product information as the target product information. If the first product information and the second product information are inconsistent, the image recognition system matches the first product information and the second product information with product information in a preset product database to determine the target product information based on the matching results.

[0084] The above processing method enables the image recognition system to quickly output the target product information when the first and second product information are consistent, ensuring the system's response efficiency. When discrepancies arise between the first and second product information, the image recognition system uses a product database for decision-making, effectively resolving recognition conflicts between the character recognition model and the multimodal recognition model. The product database provides an additional knowledge verification layer for the image recognition system, enabling precise database queries using character information and similar product matching through the generalization capabilities of multimodal features.

[0085] In some embodiments, the image recognition system can determine the distance between first product information and second product information. If the distance is less than or equal to a first distance threshold, the image recognition system determines the target product information in the product database based on a first preset rule. If the distance is greater than the first distance threshold, the image recognition system determines the target product information in the product database based on a second preset rule.

[0086] In the above embodiments, the image recognition system establishes an intelligent decision-making mechanism based on the distance between first product information and second product information. The distance between the first and second product information integrates a multi-dimensional evaluation of textual semantic similarity and visual feature similarity. When the distance is less than or equal to a first distance threshold, it indicates a high degree of consistency between the two recognition results, and the image recognition system uses a first preset rule for precise matching. When the distance is greater than the first distance threshold, it indicates a low degree of consistency between the two recognition results, and the target product information is determined based on a second preset rule. This dynamic decision-making mechanism achieves an objective measurement of recognition credibility by quantitatively evaluating the degree of consistency between the two recognition results. Furthermore, the use of differentiated preset rule matching strategies for different confidence levels ensures recognition efficiency when the two recognition results are highly consistent, while also providing a more complex solution for situations where there are discrepancies. This allows the image recognition system to intelligently adjust its recognition strategy according to the confidence level of the specific recognition scenario, thereby improving the overall accuracy and robustness of the image recognition results.

[0087] For example, the image recognition system can determine the target product information in the product database based on a first preset rule as follows: The image recognition system matches the first product information and the second product information with the product information in the product database, respectively. If either the first or second product information matches a product in the database, the image recognition system determines the matching product in the database as the target product information. If neither the first nor the second product information matches a product in the database, the image recognition system determines the distance between each product in the database and the first or second product information. If the distance is less than a second distance threshold, the image recognition system determines the product in the database as the target product information. If the distances corresponding to each product in the database are greater than the second distance threshold, the image recognition system determines the second product information as the target product information.

[0088] In the above embodiments, the image recognition system first queries the product database based on the first product information and the second product information. When either the first or second product information can be matched with a completely corresponding product in the product database, the matching product is identified as the target product, ensuring a rapid response when the target product exists in the product database. When neither the first nor the second product information has a completely matching product in the product database, a second distance threshold filtering strategy is used to identify similar but not identical product information within the product database. This indicates that the product database contains product information that is similar but not identical to the first or second product information. If at least one candidate product in the product database exists that is less than the second distance threshold from the first or second product information, the candidate product with the smallest distance (i.e., the closest) is identified as the target product. When the distance between all product information in the product database and either the first or second product information is greater than a second distance threshold, the recognition result of the multimodal recognition model (the second product information) is adopted by default, and the second product information is identified as the target product information. This hierarchical decision-making approach of the image recognition system significantly improves the efficiency and accuracy of target product recognition, effectively balancing the requirements of processing efficiency and generalization ability of the image recognition system while ensuring recognition accuracy.

[0089] For example, the image recognition system can determine the target product information in the product database based on a second preset rule as follows: if a product in the product database that matches the first product information exists, the image recognition system determines the matching product in the database as the target product information. If no product in the product database that matches the first product information exists, the image recognition system determines that the recognition has failed.

[0090] In the above embodiments, when the image recognition system determines the target product information based on the second preset rule, it uses the first product information recognized by the character recognition model as a benchmark and performs a matching query in the product database. When there is product information in the product database that is completely consistent with the first product information, the product information in the database is directly used as the target product information. When there is no corresponding product information in the product database, the image recognition system determines that the recognition has failed.

[0091] In the above processing, the character recognition model achieves high accuracy and reliability. Furthermore, by rigorously matching the first product information with product information in the product database, mismatches of similar but inaccurate products are effectively avoided. In addition, the explicit recognition failure mechanism can promptly terminate erroneous recognition processes, preventing the image recognition system from making incorrect judgments based on incomplete or uncertain information. This ensures accuracy while effectively controlling the risks associated with incorrect recognition. Especially in specialized fields, the recognition of target products is crucial; errors can have a significant negative impact on users. For example, in pharmaceutical recognition scenarios, incorrect drug identification would drastically reduce the user experience.

[0092] In some embodiments, if the first product information and the second product information are inconsistent, the image recognition system fine-tunes the image recognition model based on the target product information. This process of fine-tuning the image recognition model based on the recognition results can efficiently collect samples that are prone to errors in the multimodal model, thereby promoting the training and updating of the large multimodal model in the target domain, realizing efficient iterative upgrades of the large multimodal model, and significantly improving the recognition capability and professionalism of the image recognition model in the target domain.

[0093] For example, the fine-tuning training process of an image recognition model can be implemented based on offline algorithms, thereby isolating the training process, ensuring that the normal image recognition algorithm is not affected by the training process, avoiding model oscillations caused by online learning, and guaranteeing the stability of the image recognition method application. The fine-tuning training process of the image recognition model can be performed after collecting a batch of error samples. Processing the accumulated error samples in batches allows for more systematic feature learning. It should be understood that the above embodiments are merely illustrative, and the specific fine-tuning training method can be flexibly adjusted according to user needs and is not limited to the embodiments described above.

[0094] In some embodiments, if the first product information and the second product information are the same, and the first product information does not exist in the product library, then the first product information is added to the product library.

[0095] Specifically, the image recognition system establishes an intelligent self-expanding mechanism for its product database. When the outputs of the character recognition model and the multimodal recognition model are highly consistent, it indicates that the reliability of the current recognition result is high. However, if the corresponding product information does not exist in the current product database, the image recognition system automatically adds the first recognized product information as a new entry to the product database. This automatic expansion of the product database first verifies the confidence level of the recognition results, ensuring that the confidence level is met only when the first and second product information are consistent. Only then is the first product information stored in the product database, ensuring the accuracy of newly added data and preventing erroneous data from contaminating the database, thus achieving automatic expansion of the product database.

[0096] In summary, the image recognition method and system provided in this specification, after acquiring the product image to be recognized, first extracts the first product information of the target product from the image using a character recognition model, and then analyzes the image content in the product image using a multimodal recognition model to generate second product information. Subsequently, the image recognition system combines the first and second product information to determine the target product information. In the above method, the image recognition system effectively suppresses the illusion problem of large models in related technologies by fusing character recognition and multimodal recognition dual verification mechanisms, while providing an accurate basis for the classification of fuzzy visual features. The image recognition system can use the structured information provided by the character recognition model as a strong prior constraint, reducing the tendency of the multimodal model to misjudge unknown products. Simultaneously, the image recognition system can correct potential missed detections or errors in character recognition based on the visual semantic features supplemented by the multimodal model. Therefore, through system collaboration between the character recognition model and the multimodal recognition model, the image recognition system improves the robustness of product recognition while avoiding the risk of a single model misclassifying unfamiliar products as common categories, thereby enhancing the overall accuracy and reliability of product recognition in complex scenarios.

[0097] Figure 4 A schematic flowchart of an image recognition method according to another embodiment of this specification is shown, such as... Figure 4As shown, the image recognition method includes a preprocessing stage and a post-processing stage. In the preprocessing stage, the character recognition model and the multimodal recognition model respectively recognize the product image to obtain first product information and second product information. In the post-processing stage, the image recognition system performs recognition based on the first product information, the second product information, and the product database to obtain the target product information, or returns a recognition failure (rejection). In the case of recognition failure, the image recognition system will also automatically collect the failed product images as misclassified / difficult example data and return it to the image recognition system. Based on the misclassified / difficult example data, the character recognition model and the multimodal recognition model can be fine-tuned and trained. This feedback mechanism allows for iterative upgrades of the image recognition system, effectively addressing its shortcomings.

[0098] In some embodiments, fine-tuning the character recognition model and the multimodal recognition model based on faulty / difficult example data can be achieved by manually labeling the faulty / difficult example data, and then fine-tuning the character recognition model and the multimodal recognition model based on the labeled faulty / difficult example data. It should be understood that the above embodiments are merely illustrative, and the specific fine-tuning training method can be flexibly adjusted according to user needs or the needs of the current application scenario, and is not limited to the methods given in the above embodiments.

[0099] Figure 5 A schematic flowchart of an image recognition method according to another embodiment of this specification is shown, such as... Figure 5 As shown, after the image recognition system acquires the product image to be recognized, it detects whether a subject frame corresponding to the target product exists in the product image based on a preset product detection module. When a subject frame corresponding to the target product exists in the product image, the image recognition system acquires a first subject image within the product image based on the subject frame and sends the first subject image to the character recognition module, so that the character recognition module can recognize the first subject image to obtain the first product information. In addition, the image recognition system enlarges the subject frame according to a preset ratio to obtain an enlarged subject frame, and acquires a second subject image within the product image based on the enlarged subject frame, so that the second subject image is input into the multimodal recognition model, so that the multimodal recognition model can recognize the second subject image to obtain the second product information of the target product.

[0100] Continue as Figure 5As shown, when the target product's bounding box is absent from the product image, the system does not immediately determine recognition failure. Instead, the original product image is sent to both the character recognition model and the multimodal recognition model (product images and bounding boxes are mutually exclusive). Recognition failure is confirmed when the multimodal recognition model detects that the product image does not contain the target product. In other words, when the preset product detection module detects the absence of the target product's bounding box in the product image, the image recognition system does not immediately determine recognition failure. Instead, it first records the recognition result of the product detection model. The product image is then input into the subsequent recognition process for a secondary verification step, where the multimodal recognition model performs a second recognition. Recognition failure is only confirmed when both the product detection module and the multimodal recognition model detect the absence of the target product's bounding box in the product image. This improves the accuracy of image recognition model rejection and avoids false rejections.

[0101] Even if the target product's bounding box is not present in the product image, when the multimodal recognition model identifies the target product, the image recognition system will acquire the second product information identified by the multimodal recognition model and the first product information identified by the character recognition model.

[0102] Continue as Figure 5 As shown, after acquiring the first product information and the second product information, the image recognition system determines the target product information based on the first product information, the second product information, and the product library. When the distance between the first product information and the second product information is less than a first threshold, the target product information is determined and output from the first product information, the second product information, and the product information within the product library. When the distance between the first product information and the second product information is greater than a second preset distance, and there is product information within the product library that matches the first product information, the matching product information within the product library is determined to be the target product information. When the distance between the first product information and the second product information is greater than the second preset distance, and there is no product information within the product library that matches the first product information, the recognition fails.

[0103] Furthermore, when the first product information and the second product information are inconsistent, the image recognition system fine-tunes and trains the character recognition model and the image recognition model based on the target product information to obtain a trained image recognition system. When recognition fails, the image recognition system acquires the product image that failed to be recognized and fine-tunes and trains the image recognition system based on the product image that failed to be recognized to obtain a trained image recognition system.

[0104] This specification, in another aspect, provides a computer-readable non-transitory storage medium storing at least one set of instructions executable for image recognition. When the at least one set of instructions is executed by a processor, it instructs the processor to implement the steps of the image recognition method P300 described herein. In some possible embodiments, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is run on the image recognition system 130, the program code causes the image recognition system 130 to perform the steps of the image recognition method P300 described herein. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the image recognition system 130. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the foregoing. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar programming languages.

[0105] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0106] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure may be presented by way of example only and may not be restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0107] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0108] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

[0109] Every patent, patent application, publication of a patent application, and other material, such as articles, books, specifications, publications, documents, and literature (excluding any related historical examination documents), cited in this disclosure is incorporated herein for all purposes, including, for example, in the specification and claims of this disclosure. However, in the event of any inconsistency or conflict between the descriptions, definitions, and / or terms used in the foregoing and those used in this disclosure, the descriptions, definitions, and / or terms used in this disclosure shall prevail.

[0110] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. An image recognition method, wherein, The method is applied to an image recognition model, which includes a character recognition model and a multimodal recognition model, and includes: Acquire the product image to be identified; The character recognition model is used to identify the first product information of the target product in the product image; The second product information of the target product in the product image is identified using the multimodal recognition model; and Based on the first product information and the second product information, the target product information of the target product is determined.

2. The method according to claim 1, wherein, The step of determining the target product information of the target product based on the first product information and the second product information includes: If the first product information and the second product information are consistent, then the first product information or the second product information is determined to be the target product information; If the first product information and the second product information are inconsistent, the first product information and the second product information are matched with product information in a preset product library to determine the target product information based on the matching result.

3. The method according to claim 2, wherein, The step of matching the first product information and the second product information with product information in a preset product database to determine the target product information based on the matching result includes: Determine the distance between the first product information and the second product information; If the distance is less than or equal to the first distance threshold, the target product information is determined in the product database based on the first preset rule; If the distance is greater than the first distance threshold, the target product information is determined in the product database based on the second preset rule.

4. The method according to claim 3, wherein, The step of determining the target product information in the product database based on the first preset rule includes: The first product information and the second product information are respectively matched with the product information in the product database; If either the first product information or the second product information is consistent with the product information in the product library, then the consistent product information in the library is determined to be the target product information. If both the first product information and the second product information are inconsistent with the product information in the product library, then the distance between each piece of product information in the library and the first product information or the second product information is determined. If the distance is less than the second distance threshold, then the product information in the database is determined to be the target product information; If the distances corresponding to each product information in the database are all greater than the second distance threshold, then the second product information is determined to be the target product information.

5. The method according to claim 3, wherein, The step of determining the target product information in the product database based on the second preset rule includes: When there is product information in the product library that is consistent with the first product information, the consistent product information in the product library is determined to be the target product information. If no product information matching the first product information exists in the product database, the identification is deemed to have failed.

6. The method according to claim 2, wherein, The method further includes: If the first product information and the second product information are inconsistent, the image recognition model is fine-tuned and trained based on the target product information.

7. The method according to claim 2, wherein, The method further includes: If the first product information and the second product information are the same, and the first product information does not exist in the product database, then the first product information is added to the product database.

8. The method of claim 1, wherein, Before recognizing the first product information of the target product in the product image through the character recognition model, the method further includes: Target product recognition is performed on the product image to obtain the main bounding box corresponding to the target product in the product image; The first subject image within the product image is obtained based on the subject bounding box, and the first subject image is recognized by the character recognition model or the multimodal recognition model.

9. The method according to claim 8, wherein, The step of recognizing the first subject image using the multimodal recognition model includes: The main frame is enlarged according to a preset ratio to obtain an enlarged main frame, and a second main image within the product image is obtained based on the enlarged main frame; and The multimodal recognition model identifies the second subject image to obtain the second product information of the target product.

10. The method according to claim 8, wherein, The image recognition model also includes a preset product detection module. Before performing target product recognition on the product image, the method further includes: The preset product detection module is used to detect whether the target product exists in the product image.

11. The method according to claim 8, wherein, The method further includes: When the product image does not contain the main frame corresponding to the target product, the character recognition model recognizes the product image to obtain the first product information of the target product; and The multimodal recognition model identifies the product image to obtain second product information of the target product.

12. The method according to claim 11, wherein, The multimodal recognition model identifies the product image to obtain second product information of the target product, including: When the multimodal recognition model identifies that the target product is not included in the product image, the recognition is determined to have failed.

13. The method according to claim 1, wherein, The multimodal recognition model is a multimodal recognition model that has been iteratively trained in advance based on a product database in the target domain.

14. The method according to claim 1, wherein, The multimodal recognition model is a lightweight multimodal recognition model.

15. The method according to claim 1, wherein, The first product information for identifying the target product in the product image using the character recognition model includes: The character recognition model is used to identify the character information contained in the product image. The character information includes: character content and detection box. The detection box is used to mark the position and range of the character in the product image. The first product information is obtained by processing the character information through detection, fusion, and positioning algorithms.

16. An image recognition system, comprising: At least one storage medium storing at least one instruction set for image recognition; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes the method of any one of claims 1-15 according to the instructions of the at least one instruction set.

Images