Open category item identification method, device, equipment, storage medium and product

Abstract

The application discloses an open category article identification method and device, equipment, a storage medium and a product, and relates to the technical field of image recognition. The identification result of the to-be-identified article is obtained by matching the local feature vector in the reference article vector library. Since the segmentation processing is only responsible for outputting the segmentation mask and the boundary box of the to-be-identified article, when a new business category needs to be added, the reference fusion vector corresponding to the new business category is only written into the reference article vector library, so that the category expansion can be completed, the segmentation model does not need to be retrained, and the cost of model maintenance and online is reduced. According to the comprehensive score of the candidate category, the candidate difference, the modal consistency, the image quality and the confusion category group identifier, it is determined whether to trigger review, so that the visual small model and the vector retrieval module process the conventional samples, and the multi-modal review model only processes difficult samples with low confidence, high confusion or inconsistent modes, thereby reducing unnecessary multi-modal model calling.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to methods, apparatus, devices, storage media and products for open category item recognition. Background Technology

[0002] In scenarios such as warehousing and logistics, retail shelf inspection, industrial parts sorting, and fixed asset management, item recognition systems typically employ deep learning-based object detection models. A classic implementation of this type of model involves pre-defining a fixed set of categories during the training phase and learning the target location and category of items through labeled samples. After training, the model implicitly stores category knowledge through neuron connection weights, enabling it to simultaneously output location information and classification results for items in an input image.

[0003] However, in real-world business applications, item categories are not static. With product iterations, seasonal changes, new product launches, or business expansion, the set of categories for the items to be identified needs continuous expansion. For example, retail stores stock new products weekly, and industrial manufacturing companies dynamically change part models based on orders. In such open-category scenarios, using traditional solutions leads to long model maintenance cycles and high deployment costs. Summary of the Invention

[0004] The main purpose of this application is to provide an open category item recognition method, apparatus, device, storage medium and product, which aims to solve the technical problems of long maintenance cycle and high cost of existing item recognition methods in open category scenarios.

[0005] To achieve the above objectives, this application proposes an open-category item identification method, which includes: Acquire an image of a scene to be identified, wherein the image of the scene to be identified contains an item to be identified; The scene image to be identified is segmented to determine the segmentation mask and bounding box of the object to be identified; The object to be identified is cropped from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified; Feature extraction is performed on the local image to obtain a local feature vector; The identification result of the object to be identified is obtained by matching the local feature vector in the reference item vector library.

[0006] In one embodiment, the step of segmenting the scene image to be identified and determining the segmentation mask and bounding box of the object to be identified includes: The image of the scene to be identified is segmented at the category-independent pixel level using a visual small model to obtain the segmentation mask and bounding box of the object to be identified. The visual mini-model includes a backbone network, an efficient hybrid encoder, a pixel decoder, and a mask prediction head. Accordingly, the step of performing category-independent pixel-level segmentation of the scene image to be identified using a visual small model to obtain the segmentation mask and bounding box of the object to be identified includes: The backbone network is used to extract features from the scene image to be identified to obtain basic visual features; The high-efficiency hybrid encoder is used to perform cross-scale fusion enhancement on the basic visual features to obtain multi-scale features; The enhanced multi-scale features are decoded pixel by pixel by the pixel decoder to obtain a pixel feature map, and the enhanced multi-scale features are predicted by the mask prediction head to obtain the mask embedding of the object to be identified. The segmentation mask corresponding to the item to be identified is determined by matching the mask embedding and the pixel feature map. Based on the segmentation mask, the minimum bounding rectangle corresponding to the object to be identified in the scene image to be identified is determined, and a bounding box is generated based on the minimum bounding rectangle.

[0007] In one embodiment, the training process of the visual mini-model includes: Obtain a public instance segmentation dataset and merge multiple target category labels from the public item segmentation dataset into a foreground category label; The visual mini-model is trained using the foreground category label and the pixel-level mask annotation of each target item in the public instance segmentation dataset as supervision signals.

[0008] In one embodiment, the loss function for the training process of the visual mini-model is: ; in, Focal Loss is used to classify and characterize the ability to determine whether a candidate region contains a valid target object. The L1 regression loss is used to characterize the degree of penalty for the coordinate difference between the predicted and actual bounding boxes. For generalized intersection-union loss, it is used to characterize the degree of penalty between the predicted bounding box and the ground truth bounding box; Pixel-level masked binary cross-entropy loss is used to characterize the model's ability to classify whether each pixel belongs to the current object. The Dice loss is used to characterize the degree of overlap between the model's predicted mask and the real mask.

[0009] In one embodiment, the step of cropping the object to be identified from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified includes: Obtain the segmentation confidence score of the segmentation mask, and determine whether the segmentation mask is usable based on the segmentation confidence score; When the segmentation mask is available, the foreground pixel region corresponding to the object to be identified in the scene image to be identified is extracted according to the segmentation mask, and the local image of the object to be identified is determined according to the foreground pixel region; When the segmentation mask is unavailable, a local image of the object to be identified is acquired from the scene image to be identified based on the bounding box.

[0010] In one embodiment, the step of extracting features from the local image to obtain a local feature vector includes: Visual features are extracted from the local image to obtain the visual feature vector of the local image; Text features are extracted from the local image to obtain the text feature vector of the local image; Semantic description features are extracted from the local image to obtain the semantic description feature vector of the local image; The visual feature vector, the text feature vector, and the semantic description feature vector are fused to obtain a local feature vector.

[0011] In one embodiment, the step of matching the local feature vector in a reference item vector library to obtain the identification result of the item to be identified includes: The local feature vector is used as a query vector to perform a nearest neighbor search in the reference item vector database to obtain a preset number of candidate categories. The target candidate category is determined based on the comprehensive score of each candidate category; If the target comprehensive score corresponding to the target candidate category does not meet the review conditions, the target candidate category shall be used as the identification result of the item to be identified.

[0012] In one embodiment, after determining the target candidate category based on the comprehensive score of each of the candidate categories, the method further includes: When the target comprehensive score corresponding to the target candidate category meets the review conditions, the output space is limited to the candidate category and the unknown category; The partial image of the object to be identified, the category information corresponding to the candidate category, the comprehensive score of the candidate category, and the text information corresponding to the object to be identified are obtained as multimodal information. The multimodal information is input into the multimodal verification model for category selection to obtain the recognition result of the item to be identified.

[0013] Furthermore, to achieve the above objectives, this application also proposes an open-category item identification device, which includes: The image acquisition module is used to acquire an image of a scene to be identified, wherein the image of the scene to be identified contains an object to be identified; The image recognition module is used to segment the scene image to be recognized and determine the segmentation mask and bounding box of the object to be recognized. The image cropping module is used to crop the object to be identified from the scene image to be identified based on the segmentation mask and the bounding box, so as to obtain a local image corresponding to the object to be identified; The local feature extraction module is used to extract features from the local image to obtain a local feature vector; The category recognition module is used to match the local feature vector in a reference item vector library to obtain the recognition result of the item to be recognized.

[0014] In addition, to achieve the above objectives, this application also proposes an open category item recognition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the open category item recognition method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the open category item identification method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the open category item identification method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This application obtains an image of a scene to be identified, which contains an object to be identified; segments the scene image to determine the segmentation mask and bounding box of the object to be identified; crops the object to be identified from the scene image based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified; extracts features from the local image to obtain a local feature vector; and matches the local feature vector with a reference object vector library to obtain the identification result of the object to be identified.

[0018] Because the process first segments the scene image to determine the segmentation mask and bounding box of the item to be identified, and then matches it with a reference item vector library to obtain the recognition result, the identification of "item location" and "item category" is decoupled. The segmentation process is only responsible for outputting the segmentation mask and bounding box of the item to be identified, and does not output the business category of the item to be identified; category recognition is completed by matching in the reference item vector library. This architecture design ensures that the knowledge of business categories is stored in the reference item vector library rather than in the segmentation model. When a new business category needs to be added, only the reference fusion vector corresponding to the new business category needs to be written to the reference item vector library to complete the category expansion, without retraining the segmentation model. Compared with the traditional approach of re-collecting samples, labeling samples, and retraining the detection model when adding a new category, the above method significantly shortens the category expansion cycle, reducing the category deployment time from several days to several weeks to minutes, while greatly reducing the cost of model maintenance and deployment. By fusing visual features, textual features, and semantic description features of the object to be identified, and dynamically adjusting the fusion weights of different modalities based on image quality, text recognition confidence, effective text length, and semantic description confidence, the fusion vector can adaptively utilize effective modal information according to the actual input quality, improving the ability to distinguish fine-grained items with different brands, models, specifications, and local identifiers. The system determines whether to trigger a review by using the comprehensive score of candidate categories, candidate score difference, modal consistency, image quality, and confusion category group identifiers. This allows the visual small model and vector retrieval module to handle regular samples, while the multimodal review model only handles difficult samples with low confidence, high confusion, or modal inconsistency, thereby reducing unnecessary multimodal model calls. By recording candidate categories, comprehensive scores, whether a review is triggered, the source of identification, and the review results, and by writing low-confidence samples, large model correction samples, unknown category samples, and manually confirmed samples into a difficult example library, the identification process is traceable and provides a data foundation for subsequent updates to the reference fusion vector, confusion category group, and gating threshold. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the open category item identification method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the open category item identification method of this application; Figure 3 This is a schematic diagram of the feature fusion process in one implementation of the open category item recognition method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the open category item identification method of this application; Figure 5 This is a schematic diagram illustrating the process of constructing a reference item vector library in one implementation of the open category item recognition method of this application; Figure 6 This is a schematic diagram of the module structure of the open category item recognition device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the open category item identification method in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application embodiment is as follows: acquiring a scene image to be identified, which contains an object to be identified; performing segmentation processing on the scene image to be identified to determine the segmentation mask and bounding box of the object to be identified; cropping the object to be identified from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified; extracting features from the local image to obtain a local feature vector; and matching the local feature vector in a reference object vector library to obtain the identification result of the object to be identified.

[0026] In traditional approaches, object detection models learn both "object location" and "object category" simultaneously during the training phase, with both bound to the same model. This approach requires retraining the model when a new category is added, as it needs to learn the visual feature distribution of the new category while also preventing the forgetting of old categories. This results in long model deployment cycles and high change costs.

[0027] This application's embodiments employ category-independent segmentation processing, completely avoiding contact with business category information during the segmentation stage. It relies solely on visual capabilities to segment individual items from an image. By recognizing the segmented and cropped local images, the recognition results for each item are determined, achieving decoupling between "item location" and "item category" recognition. When the business category library changes, the input and output spaces of "item location" recognition remain unaffected, thus eliminating the need for retraining. Category knowledge is stored using a reference item vector library. When the system needs to add a new category, expansion is achieved simply by writing the reference fusion vector of the new category into the vector index. The "item category" is determined by the distance between vectors, effectively transferring category knowledge outside the model and avoiding the problems of long model maintenance cycles and high deployment costs at the architectural level.

[0028] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a computer or server, or an electronic device or virtual device capable of performing the above functions. The following description uses an open-category item identification device (hereinafter referred to as the identification device) as an example to illustrate this embodiment and the subsequent embodiments.

[0029] Based on this, embodiments of this application provide an open category item identification method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the open category item identification method of this application.

[0030] In this embodiment, the open category item identification method includes steps S10 to S50: Step S10: Obtain an image of the scene to be identified, wherein the image of the scene to be identified contains an item to be identified; Step S20: Perform segmentation processing on the scene image to be identified, and determine the segmentation mask and bounding box of the object to be identified.

[0031] It should be noted that the aforementioned scene image to be identified can be a video frame image captured in real time by a fixed camera, a still image taken and uploaded by a mobile terminal such as a mobile phone or handheld camera, or an offline image file imported in batches by a business system, etc., and this application embodiment does not impose any limitations on this. The scene image to be identified may contain an image corresponding to at least one item to be identified, such as a water cup, a washbasin, or a fan. Simultaneously, the scene image to be identified may also contain non-object areas such as shelves, desktops, and background walls.

[0032] It is understood that, in the embodiments of this application, when performing image acquisition, image information such as image source, acquisition time, business scenario, shooting device and area information can also be recorded.

[0033] It should be understood that the aforementioned items to be identified are items that need to be identified and classified, such as goods on shelves, goods in warehouses, parts in industrial settings, or equipment in fixed asset management, etc., and this application embodiment does not limit them.

[0034] It's important to note that segmentation processing enables the identification of each pixel in the scene image, determining which pixels belong to foreground objects and which belong to background regions. Furthermore, within foreground pixels, it distinguishes the boundaries between different objects, thus separating them at the pixel level. Compared to object detection, segmentation outputs a pixel-level segmentation mask. This mask is a binary matrix with the same spatial resolution as the scene image. Each element in this matrix represents whether a pixel at a corresponding location in the scene image belongs to an object. For example, for a given object, a pixel with a value of "1" in the segmentation mask indicates that it belongs to the object, while a pixel with a value of "0" indicates that it does not belong to the object.

[0035] Understandably, an image of a scene to be identified typically contains multiple objects to be identified. When multiple objects exist, there can also be segmentation masks and bounding boxes corresponding to each object.

[0036] It should be understood that the bounding box mentioned above can be a box used to represent the number of pixels occupied by the object to be identified in the scene image to be identified. The bounding box can be determined according to the segmentation mask. For example, the outer boundary of the graphic formed by the pixel position with a value of "1" in the segmentation mask can be used as the bounding box, or the smallest bounding rectangle corresponding to the pixel position with a value of "1" in the segmentation mask can be used as the bounding box. The embodiments of this application do not limit this.

[0037] It is understood that the shape of the bounding box can be a rectangle, a circle, an irregular shape, etc., and the embodiments of this application do not limit this.

[0038] In some embodiments of this application, to improve the accuracy of segmentation processing, the step of segmenting the scene image to be identified and determining the segmentation mask and bounding box of the object to be identified includes: performing category-independent pixel-level segmentation of the scene image to be identified using a visual mini-model to obtain the segmentation mask and bounding box of the object to be identified; wherein, the visual mini-model includes a backbone network, an efficient hybrid encoder, a pixel decoder, and a mask prediction head; correspondingly, the step of performing category-independent pixel-level segmentation of the scene image to be identified using the visual mini-model to obtain the segmentation mask and bounding box of the object to be identified includes: performing category-independent pixel-level segmentation of the scene image to be identified using the visual mini-model to obtain the segmentation mask and bounding box of the object to be identified. Feature extraction is performed on the scene image to obtain basic visual features; the basic visual features are enhanced by cross-scale fusion using the efficient hybrid encoder to obtain multi-scale features; the enhanced multi-scale features are pixel-decoded using the pixel decoder to obtain a pixel feature map, and the enhanced multi-scale features are predicted using the mask prediction head to obtain the mask embedding of the object to be identified; the mask embedding and the pixel feature map are matched to determine the segmentation mask corresponding to the object to be identified; the minimum bounding rectangle corresponding to the object to be identified in the scene image is determined based on the segmentation mask, and a bounding box is generated based on the minimum bounding rectangle.

[0039] It is understood that the visual mini-model in the embodiments of this application can be a model used for segmenting the object to be identified, such as a model based on the Mask2Former architecture, a model based on the OneFormer architecture, etc., and the embodiments of this application do not limit it.

[0040] It should be noted that the visual small model F_seg in this application embodiment can perform class-independent pixel-level instance segmentation on the scene image to be identified, so as to output the pixel binary mask of each target instance (item to be identified), the bounding box determined by the bounding rectangle of the mask, and the instance segmentation confidence, without outputting the specific business category.

[0041] In some embodiments of this application, the visual small model of this application can be implemented based on the Mask-RT-DETR-L architecture. In this application embodiment, the Mask-RT-DETR-L architecture is based on the end-to-end Transformer detection framework of RT-DETR-L, the backbone network adopts HGNetV2-L (Hybrid Gated Network V2-Large), the feature fusion part adopts an efficient hybrid encoder, and pixel decoders and instance mask prediction heads are set on multi-scale features; each Transformer query, in addition to predicting the object class score and bounding box, also generates a mask embedding, and matches it with the high-resolution pixel features output by the pixel decoder to obtain the pixel-level mask of the corresponding target instance.

[0042] It is understood that the backbone network mentioned above can be a convolutional neural network structure used for basic visual feature extraction in a visual small model. The backbone network in this embodiment can adopt multi-layer convolution and pooling operations to gradually map the scene image to be recognized from a high-resolution, high-detail pixel space to a low-resolution, high-semantic feature space, thereby obtaining multi-scale basic visual features.

[0043] It should be understood that the aforementioned efficient hybrid encoder can be a neural network model used in small visual models to perform cross-scale feature fusion and enhancement on the basic visual features output by the backbone network. This efficient hybrid encoder can fuse feature maps of different resolutions so that the final output features simultaneously possess both detailed and semantic information, resulting in multi-scale features, which are the enhanced basic visual features.

[0044] It is understood that the aforementioned pixel decoder is a neural network module capable of restoring multi-scale features output by an efficient hybrid encoder into a high-resolution pixel feature map, such as a Transformer-based pixel decoder or a regression-based pixel decoder. This application does not limit this approach. By upsampling and recombining the input multi-scale features using the pixel decoder, a low-resolution feature map can be restored to a higher spatial resolution, preserving pixel-level spatial details.

[0045] It should be understood that the aforementioned mask prediction head can be a neural network module used for mask embedding prediction, such as a convolution-based masking head, a Transformer-based masking head, etc., and this application embodiment does not limit this. Through the mask prediction head, a corresponding mask embedding can be generated for each item to be identified.

[0046] In this embodiment, a category-agnostic training strategy can be adopted when training the visual mini-model. Specifically, all category labels in a public instance segmentation database (such as the 80 target categories in the COCO dataset) can be merged into a single object category label, while retaining the pixel-level mask annotation of each target instance (the target instance corresponds to the item to be identified). This allows the model to learn foreground pixel attribution, instance separation, and spatial boundaries, without learning the semantics of specific business categories. Based on this, instance mask annotation data collected from business scenarios are mixed in for joint training or fine-tuning to adapt to the target scale, density, occlusion relationships, and background distribution of specific scenarios. That is, the training process of the visual mini-model is as follows: obtain a public instance segmentation dataset, merge multiple target category labels in the public item segmentation dataset into a foreground category label, and use the foreground category label and the pixel-level mask annotation of each target item in the public instance segmentation dataset as supervision signals for training to obtain the visual mini-model. Through this training method, the visual mini-model can learn foreground pixel attribution, item separation, and spatial boundaries of item instances, without learning the classification semantics of specific business categories.

[0047] In this embodiment, the training loss function can use Hungarian Matching Loss to match predicted instances with ground truth instances, and is composed of five weighted components: classification loss, bounding box L1 regression loss, generalized intersection-over-union loss, mask binary cross-entropy loss, and Dice loss. ; in, Focal Loss is used to classify and characterize the ability to determine whether a candidate region contains a valid target object. The L1 regression loss is used to characterize the degree of penalty for the coordinate difference between the predicted and actual bounding boxes. The Generalized Intersection over Union (GIoU) ​​loss is used to characterize the degree of penalty for the overlap between the predicted bounding box and the ground truth bounding box. Pixel-level masked binary cross-entropy loss is used to characterize the model's ability to classify whether each pixel belongs to the current object. The Dice loss is used to characterize the degree of overlap between the model's predicted mask and the real mask.

[0048] In one example embodiment of this application, the weighting coefficient... , , , , .

[0049] In some embodiments of this application, the training optimizer may be AdamW with an initial learning rate of 1×10⁻⁶. -4 The weight decay coefficient is 5×10 -2 The learning rate scheduling strategy can employ cosine annealing. Training data augmentation includes any one or more of the following: Mosaic four-image stitching, random horizontal flipping (probability 0.5), color dithering (brightness variation range ±0.3, contrast variation range ±0.3, saturation variation range ±0.3), and random scaling (scaling ratio 0.5 to 1.5); among which, geometric transformations are performed synchronously on the scene image, instance mask, and bounding boxes derived from the mask to maintain consistent pixel-level annotations.

[0050] In some embodiments of this application, for the k-th target instance, the class-independent instance segmentation output includes a bounding box defined by the mask bounding rectangle. Pixel-level binary segmentation mask and instance segmentation confidence : ; Where K represents the number of objects (target instances) segmented from the scene image to be identified. ∈{0,1} {H×W} , is a pixel-level binary mask (segmentation mask) that has been upsampled to restore the original image size; bounding box The pixel-level mask is calculated from the minimum bounding rectangle of the foreground pixels in the mask. It is a mandatory output for each target instance. Instance segmentation confidence is below a threshold. (like The results of (=0.3) are filtered.

[0051] In a specific implementation, this embodiment of the application can segment the scene to be recognized using a visual mini-model deployed in the recognition device. First, the recognition device can scale the scene image to be recognized to the input resolution required by the visual mini-model, for example, scaling the scene image to 640×640 pixels. Next, the recognition device can extract features from the scene image using the backbone network of the visual mini-model to obtain basic visual features. These basic visual features contain feature maps at multiple levels, with different levels having different resolutions. Shallow feature maps have higher resolution and contain more detailed information, while deep feature maps have lower resolution and contain stronger semantic information. Subsequently, the recognition device can perform cross-scale fusion enhancement on the basic visual features using an efficient hybrid encoder in the visual mini-model, allowing high-level semantic features and low-level detailed features to complement each other, resulting in enhanced multi-scale features that retain spatial details while possessing strong semantic expressive power. Afterward, processing can be performed through two parallel branches: a pixel decoder and a mask prediction head. In the pixel decoder branch, multi-scale features can be upsampled and reconstructed, gradually restoring the multi-scale features to a higher spatial resolution, resulting in pixel feature maps. The pixel feature map preserves the pixel-level spatial information of the scene image to be identified. Each position in the pixel feature map corresponds to a pixel region in the scene image to be identified. In the mask prediction head branch, multi-scale features can be predicted to generate corresponding mask embeddings. The mask embeddings can represent the mask feature information of the object to be identified in vector form.

[0052] Furthermore, the recognition device can embed the mask of each object to be recognized and calculate the similarity between it and each location in the pixel feature map. Locations with high similarity scores indicate that they belong to the foreground region of the object to be recognized, while locations with low similarity scores indicate that they do not belong to the foreground region. Through this matching method, a segmentation mask corresponding to each object to be recognized can be determined. Based on the segmentation mask, by traversing all pixels belonging to the foreground region, the minimum and maximum x-coordinates, minimum and maximum y-coordinates of these foreground pixels are determined. Then, a rectangle is constructed using these four coordinate values ​​as boundaries. This rectangle is the minimum bounding rectangle that completely encloses all foreground pixels in the segmentation mask, thus determining the bounding box.

[0053] Step S30: Based on the segmentation mask and bounding box, the object to be identified is cropped from the scene image to be identified to obtain a local image corresponding to the object to be identified; Step S40: Extract features from the local image to obtain a local feature vector; Step S50: Match the local feature vector in the reference item vector library to obtain the identification result of the item to be identified.

[0054] It should be noted that by using segmentation masks and bounding boxes, the image of the scene to be identified can be cropped, thus extracting the local image corresponding to the object to be identified from the scene image. This separation operation allows subsequent feature extraction to focus on the object to be identified itself, avoiding interference from the background and other objects.

[0055] It is understood that the aforementioned partial image is the image corresponding to the object to be identified. The partial image can be scaled according to a preset resolution, and this application embodiment does not limit this.

[0056] It should be noted that the aforementioned local feature vector is the feature vector obtained by performing feature extraction operations on local features. The extraction method can be single-modal feature extraction or multi-modal feature extraction, and the embodiments of this application do not limit this.

[0057] It should be explained that the aforementioned reference item vector library is a pre-built database that can store reference fusion vectors corresponding to multiple business categories. For each business category, there can be a corresponding reference fusion vector. By matching local feature vectors against the reference item vector library, it can be determined whether the item to be identified belongs to an existing business category in the library or to an unknown category, thus obtaining the identification result of the item.

[0058] This application embodiment acquires a scene image to be identified, which contains an object to be identified; it performs segmentation processing on the scene image to determine the segmentation mask and bounding box of the object to be identified; it crops the object to be identified from the scene image based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified; it extracts features from the local image to obtain a local feature vector; and it matches the local feature vector with a reference object vector library to obtain the identification result of the object to be identified. Because the scene image to be identified is segmented first to determine the segmentation mask and bounding box of the object to be identified, and then the identification result is obtained by matching with the reference object vector library, the identification of "object location" and "object category" is decoupled. The segmentation processing is only responsible for outputting the segmentation mask and bounding box of the object to be identified, and does not output the business category of the object to be identified; category identification is completed through matching in the reference object vector library. This architecture design allows the knowledge of business categories to be stored in the reference object vector library rather than in the segmentation model. When a new business category needs to be added, only the reference fusion vector corresponding to the new business category needs to be written to the reference object vector library to complete the category expansion, without retraining the segmentation model. Compared to traditional methods that require re-collecting, labeling, and retraining the detection model to add new categories, the above method significantly shortens the category expansion cycle, reducing the category deployment time from several days to weeks to minutes, while also greatly reducing the cost of model maintenance and deployment.

[0059] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating Embodiment 2 of the open category item identification method of this application.

[0060] like Figure 2 As shown in this embodiment, the step of cropping the object to be identified from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified includes: Step S31: Obtain the segmentation confidence of the segmentation mask, and determine whether the segmentation mask is usable based on the segmentation confidence. Step S32: When the segmentation mask is available, extract the foreground pixel region corresponding to the object to be identified in the scene image to be identified according to the segmentation mask, and determine the local image of the object to be identified according to the foreground pixel region; Step S33: When the segmentation mask is unavailable, a local image of the object to be identified is acquired from the scene image to be identified based on the bounding box.

[0061] It should be noted that the aforementioned segmentation confidence score can be a reliability score used by the visual small model to measure the accuracy of the segmentation mask when outputting it. In this embodiment, the segmentation confidence score can be represented by a value between 0 and 1. A higher value indicates that the visual small model's prediction result for the segmentation mask is more reliable, while a lower value indicates that the visual small model's prediction result for the segmentation mask is less reliable. This segmentation confidence score can be the confidence score obtained by the visual small model during the inference process, and this embodiment does not impose any limitations on it.

[0062] In this embodiment, a segmentation confidence threshold can be set to determine whether the segmentation mask is usable. This segmentation confidence threshold can be a fixed value or a dynamic value, and this embodiment does not limit it. For example, when the segmentation confidence threshold is a fixed value, if the segmentation confidence is greater than the segmentation confidence threshold, the segmentation mask can be considered usable; if the segmentation confidence is not greater than the segmentation confidence threshold, the segmentation mask can be considered unusable.

[0063] It is understandable that the foreground pixel region refers to the area in the scene image to be identified, which is composed of the pixels corresponding to the object to be identified. In the segmentation mask, the pixel position with the first value can be considered to belong to the foreground pixel region, and the pixel position with the second value can be considered to belong to the background pixel region. For example, the pixel position with a value of "1" belongs to the foreground pixel region, and the pixel position with a value of "0" belongs to the background pixel region.

[0064] In this embodiment, when a segmentation mask is available, the foreground pixel region marked by the segmentation mask can be considered to correspond relatively accurately to the true contour boundary of the object to be identified in the scene image. At this time, by extracting the foreground pixel region and filling the background pixel region with white or gray, only pixels belonging to the object itself can be retained, while background pixels are removed. This makes the input data for subsequent feature extraction purer, avoiding interference from background elements such as shelf shelves, adjacent goods, and light reflections on the feature representation. Simultaneously, when two or more items are adjacent in the scene image, extraction using the segmentation mask according to pixel-level boundaries can accurately separate one item from another, avoiding the erroneous inclusion of pixels from adjacent items in the feature representation of the current item.

[0065] In this embodiment, when the segmentation mask is unavailable, it can be assumed that the foreground pixel region marked by the segmentation mask cannot accurately correspond to the true contour boundary of the object to be identified in the scene image. In this case, the segmentation mask can be abandoned, and a rectangular region can be cropped from the scene image using a bounding box as a local image. In this way, the ability to identify the object can still be maintained even when the image quality is poor, the object is partially occluded, or the objects are stacked together.

[0066] In one example embodiment of this application, when a partial image is obtained through cropping, an image quality score can also be applied to that partial image. Specifically, this application embodiment can obtain the image quality score of the partial image by weighted summation of five components: sharpness, brightness appropriateness, occlusion degree, target area proportion, and segmentation confidence. The specific method can be as follows: ; ; ; in, Indicates sharpness, Represents the Laplacian operator. This represents variance calculation. This represents the normalization constant (which can take a value of 500.0). This represents a partial image of the k-th item to be identified. Indicates moderate brightness. Represents the pixel value of the brightness channel in a local image. Indicates image quality score. Indicates the degree of occlusion (value range [0, 1], 0 indicates no occlusion). This indicates the proportion of the foreground pixel region to the scene image to be identified. The minimum area threshold (e.g., a value of 0.01). This represents the split confidence level. , , , , As the weights corresponding to each component, one implementation method can take the following values: =0.25, =0.15, =0.20, =0.15, =0.25.

[0067] In one example embodiment of this application, the step of extracting features from the local image to obtain a local feature vector includes: extracting visual features from the local image to obtain a visual feature vector; extracting text features from the local image to obtain a text feature vector; extracting semantic description features from the local image to obtain a semantic description feature vector; and fusing the visual feature vector, the text feature vector, and the semantic description feature vector to obtain a local feature vector.

[0068] It should be noted that visual feature vectors can be obtained by extracting visual features from local images using a visual encoder. After the local image is input into the visual encoder, the encoder can encode the local image to output the corresponding visual feature vector. This visual feature vector can characterize the appearance information of the object to be identified, such as its shape, color, texture, and pattern.

[0069] In some embodiments of this application, the visual encoder f v The visual branch of the CLIP ViT-B / 16 (Contrastive Language-Image Pre-training, Vision Transformer, patch size 16×16) model can be used. This model contains a 12-layer Transformer encoder with 768 hidden dimensions, 12 attention heads, and an input resolution of 224×224 pixels. CLIP ViT-B / 16 is pre-trained on large-scale image-text pair datasets (such as LAION-5B or WIT-400M) through contrastive learning, possessing strong visual semantic representation capabilities. The visual encoder outputs a 768-dimensional feature vector. In Chinese business scenarios, the Chinese-CLIP ViT-B / 16 version can be used, or other visual feature extraction models can be selected.

[0070] It should be noted that the Optical Character Recognition (OCR) engine and text encoder can extract text sequences and text feature vectors from local images. These text feature vectors can be used to characterize the text content information of the object to be identified.

[0071] In one example embodiment of this application, the OCR engine can employ PaddleOCR v4, which includes two stages: a text detection module and a text recognition module. The text detection module uses DBNet++ (Differentiable Binarization with Adaptive Scale Fusion), based on a ResNet-50 backbone network, and outputs the boundary polygons of the text regions. The text recognition module uses SVTR (Scene Visual Text Recognition with Transformer), based on a Transformer encoder-CTC decoder architecture, and outputs the text sequence and character-by-character recognition confidence. The OCR engine outputs the recognized text sequence `text` to the target instance image `xk`. k and overall OCR confidence level C ocr.k Text encoder f tThe model employs BGE-Base-zh-v1.5 (BAAI General Embedding), which is based on the BERT architecture and includes a 12-layer Transformer encoder with a hidden dimension of 768. It was pre-trained on a large-scale Chinese corpus through contrastive learning and knowledge distillation. The text encoder encodes the OCR-recognized text sequence `text_k` into a 768-dimensional text vector.

[0072] It should be noted that semantic features can be extracted from the semantic description information of the item to be identified through the semantic description encoder. This semantic description information may include at least one attribute of the item, such as color, shape, packaging type, specifications, and brand. The semantic description information may originate from structured attribute text pre-maintained by the administrator for each reference category, or from attribute descriptions automatically generated from local images by a lightweight visual language model; this embodiment does not impose any limitations on this.

[0073] In one example embodiment of this application, the semantic description encoder f d It can be used with a text encoder f t Using the same BGE-Base-zh-v1.5 model, the input for semantic description is structured attribute description text, formatted as follows: color: {color}; shape: {shape}; packaging type: {packaging}; specification: {spec}; local identifier: {logo}; business attributes: {attributes}. Semantic descriptions are generated through one of three methods: (i) pre-maintaining structured attribute text for each reference category by the administrator during the offline phase; (ii) automatically generating attribute descriptions for the target instance image using a lightweight visual language model (such as MiniCPM-V-2B); (iii) automatically filling in the description based on predefined attribute templates and OCR text extraction rules. The semantic description encoder outputs a 768-dimensional semantic description vector.

[0074] In this embodiment of the application, in order to achieve feature fusion of visual feature vectors, text feature vectors, and semantic description vectors, these vectors can first be projected and aligned with the dimensions. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the feature fusion process in one implementation of the open category item recognition method of this application. Figure 3 As shown in this embodiment, the original output dimensions of the three encoders are all 768-dimensional, and they are projected onto a unified 512-dimensional common embedding space using trainable linear projection matrices. ; ; ; in, This represents the projected 512-dimensional visual vector. Represents visual feature vectors; This represents the projected 512-dimensional text feature vector. Represents the text feature vector; This represents the projected 512-dimensional semantic description feature vector. This represents a semantic description feature vector. , , This is the projection matrix corresponding to the visual feature vector, text feature vector, and semantic description feature vector.

[0075] In the embodiments of this application, , , It can be obtained through joint training using metric learning, and the corresponding training data can be obtained from the business-labeled triples (x anchor x positive x negative Composed of ) where x anchor x represents the target instance to be identified. positive As a reference sample of the same category, x negative Reference samples for different categories.

[0076] The training loss function uses triplet loss with hard negative sample mining: ; Among them, z a z p z n These are the fused feature vectors corresponding to the anchor sample, positive sample, and negative sample, respectively. The anchor sample is the target instance selected as the baseline in the training dataset. m is the margin parameter (preferably m=0.3), and ||·|| represents the L2 norm. The training optimizer uses AdamW with a learning rate of 5×10⁻⁶. -5 The batch size is 128, and the training rounds are 50. Within each batch, the nearest negative sample pair can be selected through online hard negative sample mining.

[0077] In some embodiments of this application, the projection matrix training can use InfoNCE contrastive loss instead of triplet loss, and the formula can be as follows: ; Where τ is a temperature parameter (its value can be 0.07), the summation iterates through all samples in the batch.

[0078] In some embodiments of this application, in order to adapt to different image quality and text visibility conditions, the system dynamically calculates the three-way fusion weights based on image quality score, OCR confidence, effective text length, and semantic description confidence: ; ; ; ; in, Indicates the overall confidence level of OCR recognition. Normalized effective text length L max This is the maximum effective text length (the value can be 20). Indicates the confidence level of the semantic description. Indicates the degree of occlusion. to This is an adjustable parameter, and its initial value can be: a v0 =1.0, a v1 =0.5, a v2 =0.3, a t0 =0.5, a t1 =1.0, a t2 =0.5, a d0 =0.3, a d1 =0.8, a d2 =0.3. These adjustable parameters can be further tuned on the validation set using grid search or gradient-based optimization methods. When the OCR does not detect valid text (i.e., len(text)... k When )=0), l t Approaching a t0 Text weight β after Softmax k As natural decreases, the weight of visual and semantic descriptions increases accordingly.

[0079] Understandably, after aligning these vector projections with dimensions, a fused vector can be generated to obtain local feature vectors. The fusion formula can be shown below: ; Where Norm(·) represents performing L2 normalization operation path by path, z k The fusion vector is a 512-dimensional local feature vector. Normalizing each projection vector first can eliminate the influence of scale differences of single features on the fusion result. Normalizing again after fusion can ensure that the fusion vector is on the unit hypersphere, which is convenient for subsequent cosine similarity calculation.

[0080] This application embodiment obtains the segmentation confidence score of the segmentation mask and determines whether the segmentation mask is usable based on the segmentation confidence score. When the segmentation mask is usable, the foreground pixel region corresponding to the object to be identified in the scene image to be identified is extracted based on the segmentation mask, and a local image of the object to be identified is determined based on the foreground pixel region. When the segmentation mask is unavailable, a local image of the object to be identified is acquired from the scene image to be identified based on the bounding box. By cropping the segmentation mask at the pixel level, high-quality input data is provided for subsequent feature extraction, ensuring that the feature vector can accurately represent the target object, thereby improving the matching accuracy of vector retrieval and ensuring that newly added categories can be correctly identified.

[0081] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and / or second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating Embodiment 3 of the open category item identification method of this application.

[0082] The step of matching the local feature vector in the reference item vector library to obtain the identification result of the item to be identified includes: Step S51: Use the local feature vector as a query vector to perform a nearest neighbor search in the reference item vector database to obtain a preset number of candidate categories; Step S52: Determine the target candidate category based on the comprehensive score of each candidate category; Step S53: When the target comprehensive score corresponding to the target candidate category does not meet the review conditions, the target candidate category is taken as the identification result of the item to be identified.

[0083] It should be noted that nearest neighbor retrieval refers to the operation where a recognition device uses a local feature vector as the query vector and searches for several reference fused vectors in a reference item vector library that are closest to the query vector in the feature space. Specifically, this can be achieved by calculating the cosine similarity between the query vector and each reference fused vector in the reference item vector library, sorting the search results in descending order of cosine similarity, and returning a preset number of candidate categories that rank highly.

[0084] It is understood that the aforementioned preset number can be selected according to the needs of actual applications, such as 4, 5, etc., and this application embodiment does not impose any limitation on this. In this application embodiment, taking a preset number of 5 candidate categories as an example, the candidate category set TopN can be represented as C. N ={c(1),c(2),...,c(5)}.

[0085] In some embodiments of this application, the pre-built reference item vector library can be as follows: Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the process of constructing a reference item vector library in one implementation of the open category item recognition method of this application.

[0086] Reference Figure 5 The reference item vector library in this application embodiment can store structured metadata and multimodal fusion vectors for business categories. For each business category... The reference item vector library can store the following fields: category code, one or more reference images, text description (including brand, model, specifications, purpose, etc.), and visual vector v. ref,i Text vector t ref,i Semantic description vector d ref,i , fusion vector r i The system includes the following components: the category group identifier, the version number, and the category code. The category code is a unique identifier within the business system, used for final output and integration with the business system. The reference image is one or more sample images corresponding to the business category, used for generating visual vectors and for verification references. The text description includes brand, model, specifications, and usage information, used for generating text vectors and candidate constraint inputs. The fused vector is the vector resulting from the fusion of multimodal features, used for similarity retrieval and candidate recall. The confusion category group identifier is the identifier of a set of categories that are easily misidentified, used to trigger verification and adjust thresholds. The version number is the version identifier for the category information and vector index, used for grayscale updates, rollbacks, and result traceability.

[0087] In this embodiment, the vector index can be constructed using the FAISS (Facebook AI Similarity Search) library to create an HNSW (Hierarchical Navigable Small World) index. The preferred HNSW index parameters are: maximum number of connections per node M = 32, search width efConstruction = 200 during construction, and search width efSearch = 128 during retrieval. The fused vector dimension is d = 512, and it is stored as an L2-normalized floating-point vector. In a distributed deployment scenario, it can be replaced with a Milvus vector database.

[0088] In this embodiment, the administrator or business system uploads one or more reference images of the category through an interface, and fills in the category name, category code, brand, model, specifications, and text description. The offline feature generation module extracts visual feature vectors, text feature vectors, and semantic description feature vectors respectively, and then performs fusion vector generation and fusion vector storage.

[0089] In some embodiments of this application, when a category has multiple reference images, the fusion vector of each reference image can be generated separately, the average value can be taken, and then the vector can be re-normalized. ; Where, n i For category c i The number of reference images, z ref,ij Given the fusion vector for the j-th reference image, the system saves the category metadata to a relational database, writes the fusion vector to the HNSW index, and records the version number. Since the visual mini-model does not perform specific category classification tasks, it is not necessary to retrain the visual mini-model when adding new categories.

[0090] It should be noted that the aforementioned comprehensive score can be a score obtained after a comprehensive evaluation of the candidate categories, and can be used to measure the degree of matching between the item to be identified and the candidate categories. In some embodiments of this application, the comprehensive score can be composed of six weighted components: visual similarity, text matching degree, semantic description similarity, business prior, key text hit items, and confusion risk penalty items. The calculation method can be as follows: ; in, It represents the cosine similarity between the local feature vector and the reference vector of the candidate class. This represents the normalized edit distance score between the text to be identified on the object and the reference text for the candidate categories. It can be set to 0 when no valid text is detected. This represents the cosine similarity between the semantic description feature vector of the item to be identified and the semantic description feature vector of the candidate category reference. This represents the prior score of the business, which can be determined based on the historical frequency of the category in the current business scenario and region. This indicates a key text hit, which can be used to identify text using OCR. k With candidate category c i The brand name, model number, and specification text are matched precisely as substrings, but this application does not limit this. This indicates a penalty for confusion risk. When candidate category c... i It belongs to the same confusion category group as other candidate categories in the Top N, and the difference between their combined scores is less than the threshold T. confuse (The value can be T) confuse When R = 0.10), i Take a positive value (preferably R) i =0.05); otherwise R i =0. to This represents a non-negative real number, with values ​​of λ1=0.35, λ2=0.25, λ3=0.15, λ4=0.05, λ5=0.15, and λ6=0.05. This weight can be configured differently based on business scenarios and category groups.

[0091] In some embodiments of this application, the recognition device may include a confidence gating module, which can be used to determine whether a multimodal verification model needs to be invoked. In this application embodiment, the confidence gating module can comprehensively consider factors such as Top1 comprehensive score, Top1 and Top2 score difference, modal consistency, image quality score, and confusion category group identifier to determine the verification conditions.

[0092] It should be noted that modal consistency can be calculated as follows: ; Where I(·) is an indicator function, taking the value 1 when the condition within the parentheses is true, and 0 otherwise. C m =1 indicates that the visual, textual, and semantic features point to the same candidate category (i.e., modality consistency), C m =0 indicates that at least one feature path points to a different candidate category (i.e., modal inconsistency). When the OCR does not detect valid text (len(textk)=0), C m Judgment is based solely on the consistency between visual features and semantic descriptive features, i.e. .

[0093] It should be explained that the confidence gating module in this application embodiment can use hard gating rules, soft gating rules, or a combination of hard and soft gating rules to determine confidence. Specifically, when using hard gating rules, if any one of the following criteria—Top1 comprehensive score, Top1 vs. Top2 score difference, modal consistency, image quality score, or confusion category group identifier—does not meet the judgment conditions, the target comprehensive score is considered to meet the review conditions. In this application embodiment, these judgment conditions can be set according to the needs of actual applications, and this application embodiment does not impose any limitations on them.

[0094] It should be noted that for soft gating rules, the following continuous review risk score expression can be used: ; Where σ(·) represents the Sigmoid function, and μ0 to μ5 are weight parameters, for example, weight parameters μ0=-2.0, μ1=3.0, μ2=2.5, μ3=2.0, μ4=1.5, μ5=1.8. A review is triggered when ReviewRisk exceeds the threshold τ (e.g., τ=0.5). The optimal value of this weight parameter can be obtained by fitting a logistic regression model on historical labeled samples.

[0095] In some embodiments of this application, hard gating rules may be used by default for review and determination. When it is necessary to finely adjust the review ratio by category group, soft gating is also enabled. When both are enabled, the review process begins upon triggering any gating rule.

[0096] In some embodiments of this application, after determining the target candidate category based on the comprehensive score of each candidate category, the method further includes: when the target comprehensive score corresponding to the target candidate category meets the review conditions, limiting the output space to the candidate category and the unknown category; acquiring a local image of the item to be identified, category information corresponding to the candidate category, the comprehensive score of the candidate category, and text information corresponding to the item to be identified as multimodal information; inputting the multimodal information into a multimodal review model for category selection to obtain the identification result of the item to be identified.

[0097] It should be noted that, upon entering the review process, this embodiment of the application can construct candidate constraint inputs and invoke a multimodal review model. This multimodal review model can be a visual large language model with joint image and text understanding capabilities. This model can be deployed via a cloud interface, a local server, or a lightweight edge model, as long as it can receive the target instance image, candidate category reference information, and structured prompts, and output the category number, confidence level, and judgment criteria according to the candidate set constraints.

[0098] In some embodiments of this application, the model inference parameters are preferably set as follows: temperature coefficient = 0 to improve output stability; maximum number of output tokens = 512 to ensure that the review reasons can be returned completely.

[0099] It should be noted that the candidate constraint input in this application embodiment may include a local image, candidate categories and their reference images and category attributes, and OCR-recognized text. k The overall scores of each candidate and the fine-grained attributes that need to be compared are also considered.

[0100] In some embodiments of this application, the output space of the multimodal verification model can be restricted to selecting between candidate categories and unknown categories, and the generation of category names not in the candidate set is not allowed.

[0101] In some embodiments of this application, candidate constraints can be implemented using structured prompt word templates. In this application, the specific content of the prompt word templates is not limited, but can be limited according to the needs of actual applications.

[0102] In some embodiments of this application, when the review result is unknown, the system outputs an unknown category label.

[0103] In some embodiments of this application, four types of samples can be written into the difficult example library H: (i) low-confidence samples: samples whose Confk is lower than the business threshold Tlow (preferably Tlow=0.65); (ii) large model correction samples: samples whose category output by the multimodal verification model is different from the Top1 candidate category; (iii) unknown category samples: samples whose final category is unknown; (iv) manually confirmed samples: samples that have been manually verified or corrected. The difficult example library enables continuous optimization tasks, such as reference vector updates, automatic discovery and maintenance of confused category groups, dynamic adjustment of gating thresholds, and optimization of prompt templates. This application does not impose any limitations on these aspects.

[0104] In this embodiment, the local feature vector is used as the query vector to perform a nearest neighbor search in the reference item vector library to obtain a preset number of candidate categories; the target candidate category is determined based on the comprehensive score of each candidate category; when the target comprehensive score corresponding to the target candidate category does not meet the review conditions, the target candidate category is used as the identification result of the item to be identified.

[0105] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the open category item identification method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0106] This application also provides an open-category item recognition device, please refer to... Figure 6 , Figure 6 This is a schematic diagram of the module structure of the open category item recognition device according to an embodiment of this application. The open category item recognition device includes: Image acquisition module 10 is used to acquire an image of a scene to be identified, wherein the image of the scene to be identified contains an object to be identified; Image recognition module 20 is used to segment the scene image to be recognized and determine the segmentation mask and bounding box of the object to be recognized; The image cropping module 30 is used to crop the object to be identified from the scene image to be identified according to the segmentation mask and the bounding box, so as to obtain a local image corresponding to the object to be identified; The local feature extraction module 40 is used to extract features from the local image to obtain a local feature vector; The category recognition module 50 is used to match the local feature vector in the reference item vector library to obtain the recognition result of the item to be recognized.

[0107] The open-category item recognition device provided in this application, employing the open-category item recognition method in the above embodiments, can solve the technical problems of long maintenance cycles and high costs in existing item recognition methods under open-category scenarios. Compared with the prior art, the beneficial effects of the open-category item recognition device provided in this application are the same as those of the open-category item recognition method provided in the above embodiments, and other technical features in the open-category item recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0108] This application provides an open category item recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the open category item recognition method in Embodiment 1 above.

[0109] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing an open-category item recognition device according to embodiments of this application. The open-category item recognition device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), tablet computers, PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The open category item recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0110] like Figure 7As shown, the open category item recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the open category item recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the open category item identification device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show open category item identification devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0111] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0112] The open-category item recognition device provided in this application, employing the open-category item recognition method in the above embodiments, can solve the technical problems of long maintenance cycles and high costs in existing item recognition methods under open-category scenarios. Compared with the prior art, the beneficial effects of the open-category item recognition device provided in this application are the same as those of the open-category item recognition method provided in the above embodiments, and other technical features in this open-category item recognition device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0113] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0115] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the open category item identification method in the above embodiments.

[0116] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0117] The aforementioned computer-readable storage medium may be included in the open category item identification device; or it may exist independently and not be assembled into the open category item identification device.

[0118] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the open category item recognition device, cause the open category item recognition device to: Acquire an image of a scene to be identified, wherein the image of the scene to be identified contains an item to be identified; The scene image to be identified is segmented to determine the segmentation mask and bounding box of the object to be identified; The object to be identified is cropped from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified; Feature extraction is performed on the local image to obtain a local feature vector; The identification result of the object to be identified is obtained by matching the local feature vector in the reference item vector library.

[0119] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0122] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described open-category item recognition method. This solves the technical problems of long maintenance cycles and high costs associated with existing item recognition methods in open-category scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the open-category item recognition method provided in the above embodiments, and will not be elaborated upon here.

[0123] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the open category item identification method as described above.

[0124] The computer program product provided in this application can solve the technical problems of long maintenance cycles and high costs in existing item recognition methods under open category scenarios. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the open category item recognition method provided in the above embodiments, and will not be repeated here.

[0125] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for identifying open-category items, characterized in that, The method includes: Acquire an image of a scene to be identified, wherein the image of the scene to be identified contains an item to be identified; The scene image to be identified is segmented to determine the segmentation mask and bounding box of the object to be identified; The object to be identified is cropped from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified; Feature extraction is performed on the local image to obtain a local feature vector; The identification result of the object to be identified is obtained by matching the local feature vector in the reference item vector library. The step of extracting features from the local image to obtain local feature vectors includes: Visual features are extracted from the local image to obtain the visual feature vector of the local image; Text features are extracted from the local image to obtain the text feature vector of the local image; Semantic description features are extracted from the local image to obtain the semantic description feature vector of the local image; The visual feature vector, the text feature vector, and the semantic description feature vector are fused to obtain a local feature vector. The step of matching the local feature vector in the reference item vector library to obtain the identification result of the item to be identified includes: The local feature vector is used as a query vector to perform a nearest neighbor search in the reference item vector database to obtain a preset number of candidate categories. The target candidate category is determined based on the comprehensive score of each candidate category; When the target comprehensive score corresponding to the target candidate category does not meet the review conditions, the target candidate category shall be used as the identification result of the item to be identified. After the step of determining the target candidate category based on the comprehensive score of each candidate category, the method further includes: When the target comprehensive score corresponding to the target candidate category meets the review conditions, the output space is limited to the candidate category and the unknown category; The partial image of the object to be identified, the category information corresponding to the candidate category, the comprehensive score of the candidate category, and the text information corresponding to the object to be identified are obtained as multimodal information. The multimodal information is input into the multimodal verification model for category selection to obtain the recognition result of the item to be identified.

2. The open category item recognition method as described in claim 1, characterized in that, The step of segmenting the scene image to be identified and determining the segmentation mask and bounding box of the object to be identified includes: The image of the scene to be identified is segmented at the category-independent pixel level using a visual small model to obtain the segmentation mask and bounding box of the object to be identified. The visual mini-model includes a backbone network, an efficient hybrid encoder, a pixel decoder, and a mask prediction head. Accordingly, the step of performing category-independent pixel-level segmentation of the scene image to be identified using a visual small model to obtain the segmentation mask and bounding box of the object to be identified includes: The backbone network is used to extract features from the scene image to be identified to obtain basic visual features; The high-efficiency hybrid encoder is used to perform cross-scale fusion enhancement on the basic visual features to obtain multi-scale features; The enhanced multi-scale features are decoded pixel by pixel by the pixel decoder to obtain a pixel feature map, and the enhanced multi-scale features are predicted by the mask prediction head to obtain the mask embedding of the object to be identified. The segmentation mask corresponding to the item to be identified is determined by matching the mask embedding and the pixel feature map. Based on the segmentation mask, the minimum bounding rectangle corresponding to the object to be identified in the scene image to be identified is determined, and a bounding box is generated based on the minimum bounding rectangle.

3. The open category item recognition method as described in claim 2, characterized in that, The training process of the visual mini-model includes: Obtain a public instance segmentation dataset and merge multiple target category labels from the public item segmentation dataset into a foreground category label; The visual mini-model is trained using the foreground category label and the pixel-level mask annotation of each target item in the public instance segmentation dataset as supervision signals.

4. The open category item recognition method as described in claim 3, characterized in that, The loss function for training the visual mini-model is: ； in, Focal Loss is used to classify and characterize the ability to determine whether a candidate region contains a valid target object. The L1 regression loss is used to characterize the degree of penalty for the coordinate difference between the predicted and actual bounding boxes. For generalized intersection-union loss, it is used to characterize the degree of penalty between the predicted bounding box and the ground truth bounding box; Pixel-level masked binary cross-entropy loss is used to characterize the model's ability to classify whether each pixel belongs to the current object. The Dice loss is used to characterize the degree of overlap between the model's predicted mask and the real mask.

5. The open category item recognition method as described in claim 1, characterized in that, The step of cropping the object to be identified from the scene image to be identified based on the segmentation mask and bounding box to obtain a local image corresponding to the object to be identified includes: Obtain the segmentation confidence score of the segmentation mask, and determine whether the segmentation mask is usable based on the segmentation confidence score; When the segmentation mask is available, the foreground pixel region corresponding to the object to be identified in the scene image to be identified is extracted according to the segmentation mask, and the local image of the object to be identified is determined according to the foreground pixel region.

6. The open category item recognition method as described in claim 5, characterized in that, After the steps of obtaining the segmentation confidence score of the segmentation mask and determining whether the segmentation mask is usable based on the segmentation confidence score, the method further includes: When the segmentation mask is unavailable, a local image of the object to be identified is acquired from the scene image to be identified based on the bounding box.

7. An open-category item recognition device, characterized in that, The open category item recognition device includes: The image acquisition module is used to acquire an image of a scene to be identified, wherein the image of the scene to be identified contains an object to be identified; The image recognition module is used to segment the scene image to be recognized and determine the segmentation mask and bounding box of the object to be recognized. The image cropping module is used to crop the object to be identified from the scene image to be identified based on the segmentation mask and the bounding box, so as to obtain a local image corresponding to the object to be identified; The local feature extraction module is used to extract features from the local image to obtain a local feature vector; The category recognition module is used to match the local feature vector in a reference item vector library to obtain the recognition result of the item to be recognized; The local feature extraction module is further configured to perform visual feature extraction on the local image to obtain a visual feature vector of the local image; perform text feature extraction on the local image to obtain a text feature vector of the local image; perform semantic description feature extraction on the local image to obtain a semantic description feature vector of the local image; and perform feature fusion on the visual feature vector, the text feature vector, and the semantic description feature vector to obtain a local feature vector. The category recognition module is further configured to use the local feature vector as a query vector to perform nearest neighbor retrieval in the reference item vector library to obtain a preset number of candidate categories; determine the target candidate category based on the comprehensive score of each candidate category; and when the target comprehensive score corresponding to the target candidate category does not meet the review conditions, use the target candidate category as the recognition result of the item to be identified. The category recognition module is further configured to acquire a partial image of the object to be recognized, category information corresponding to the candidate category, comprehensive score of the candidate category, and text information corresponding to the object to be recognized as multimodal information; input the multimodal information into the multimodal verification model for category selection, and obtain the recognition result of the object to be recognized.

8. An open-category item recognition device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the open category item identification method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the open category item identification method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the open category item identification method as described in any one of claims 1 to 6.

Images