Object recognition method, apparatus, device, and storage medium

Abstract

This application discloses an object recognition method, apparatus, device, and storage medium. The method involves obtaining multiple candidate categories from the text modality corresponding to an image to be recognized, including the true categories of objects in the image. A multimodal large-scale model is used to extract text features for each candidate category, and visual features for the image to be recognized are also extracted. The similarity between the visual features of the image and the text features of each candidate category is calculated, and the candidate category with the highest similarity is selected as the target category of the object in the image. This application leverages the powerful general knowledge representation capabilities of the multimodal large-scale model to extract features more accurately, thereby improving recognition accuracy. This avoids the problems of low accuracy in traditional object recognition models when training data is insufficient, as well as the low accuracy in recognizing images taken from special perspectives.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an object recognition method, apparatus, device, and storage medium. Background Technology

[0002] Object recognition refers to the identification of the types of objects contained in a provided image. Object recognition is widely used in various scenarios, such as identifying objects in environmental images captured by a robot. This helps the robot determine the categories of objects in its work area, thereby enabling it to better construct a floor plan of the work area. Figure 7 As shown, the floor plan of the work area displays various types of object textures, such as dining tables, sofas, and cabinets.

[0003] Traditional object recognition methods typically train an object recognition model using training images labeled with object categories, and then use this model to identify objects in the image to be recognized. However, the accuracy of the object recognition model is highly dependent on the size of the training data. When the training data is insufficient, the accuracy of the trained object recognition model is prone to being inadequate. This is especially true for images taken from abnormal perspectives, such as images taken by a robot vacuum cleaner from a low angle, where objects are not fully visible. Using traditional object recognition models to identify such images further reduces accuracy. For example... Figure 1 As shown, this is a picture taken by a robot vacuum cleaner. Because the object is not fully captured in the picture, it cannot be accurately distinguished as a sofa or a bed when using a traditional object recognition model, which may result in an incorrect recognition result. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide an object recognition method, apparatus, device, and storage medium to solve the problem that traditional object recognition models are limited by the scale of training data and the shooting angle of the image to be recognized, which easily leads to low recognition accuracy. The specific solution is as follows:

[0005] Firstly, an object recognition method is provided, including:

[0006] Obtain multiple candidate categories for the text modality corresponding to the image to be identified, wherein the multiple candidate categories include the true categories of objects in the image to be identified;

[0007] The text features of each candidate category are extracted using the configured multimodal large model, and the visual features of the image to be identified are extracted using the multimodal large model.

[0008] Calculate the similarity between the visual features of the image to be identified and the text features of each candidate category, and take the candidate category corresponding to the highest similarity as the target category of the object in the image to be identified.

[0009] Preferably, the text features of each candidate category are extracted using the configured multimodal large model, and the visual features of the image to be identified are extracted using the multimodal large model, including:

[0010] Using the text encoder of the multimodal large model, text encoding is performed on each of the candidate categories to obtain the text features of each candidate category;

[0011] The visual encoder of the multimodal large model is used to visually encode the image to be identified to obtain visual features.

[0012] Preferably, the image to be identified is a number of frames from a video stream captured by the robot of the work area;

[0013] Before calculating the similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified, the method further includes:

[0014] The visual features of each frame of the image to be identified are fed into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model, and the hidden state features are used as the latest visual features of each frame of the image to be identified. The sequence model is configured to extract the hidden state features of each frame of the image to be identified in the input image sequence.

[0015] Preferably, the image to be identified is a number of frames from a video stream captured by the robot of the work area;

[0016] The process of calculating the similarity between the visual features of the image to be identified and the text features of each candidate category includes:

[0017] Calculate the first similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified;

[0018] The method also includes:

[0019] The visual features of each frame of the image to be identified are fed into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model. The sequence model is configured to extract the hidden state features of each frame of the image to be identified in the input image sequence.

[0020] Calculate the second similarity between the hidden state features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified;

[0021] The process of selecting the candidate category corresponding to the highest similarity as the target category of the object contained in the image to be identified includes:

[0022] For each frame of the image to be identified:

[0023] Based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category, a third similarity is determined, and the candidate category corresponding to the highest third similarity is taken as the target category of the object contained in each frame of the image to be identified.

[0024] Preferably, the process of determining the third similarity based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category includes:

[0025] The first similarity and the second similarity of each frame of the image to be identified with each candidate category are averaged, and the average similarity is taken as the third similarity.

[0026] Preferably, the robot's work area includes at least one room;

[0027] The process of feeding the visual features of each frame of the image to be identified into a pre-configured sequence model includes:

[0028] If the image to be identified in the current frame is the first photo taken by the robot after entering a room in the work area, the hidden state features of the sequence model are reset to 0, and then the visual features of the image to be identified in the current frame are sent into the sequence model.

[0029] Preferably, the image to be identified is a number of frames from a video stream captured by the robot of the work area, and the method further includes:

[0030] Obtain a floor plan map of the work area corresponding to the robot;

[0031] Determine the location of the object in each captured frame of the image to be identified within the floor plan map;

[0032] According to the target category of the object in each frame of the image to be identified, the object texture of the target category is displayed at the corresponding location of the object in the floor plan map.

[0033] Preferably, obtaining multiple candidate categories for the text modality corresponding to the image to be identified includes:

[0034] The image to be identified is fed into a pre-configured object recognition model to obtain multiple candidate categories output by the object recognition model; the object recognition model is trained using training images labeled with object category tags.

[0035] Secondly, an object recognition device is provided, comprising:

[0036] The candidate category acquisition unit is used to acquire multiple candidate categories of the text modality corresponding to the image to be identified, wherein the multiple candidate categories include the true categories of objects in the image to be identified;

[0037] The feature extraction unit is used to extract text features of each candidate category using the configured multimodal large model, and to extract visual features of the image to be identified using the multimodal large model.

[0038] The first similarity calculation unit is used to calculate the similarity between the visual features of the image to be identified and the text features of each candidate category;

[0039] The target category determination unit is used to select the candidate category corresponding to the highest similarity as the target category of the object in the image to be identified.

[0040] Thirdly, an object recognition device is provided, including: a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement the various steps of the object recognition method as described above.

[0043] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the object recognition method as described above.

[0044] Using the above technical solution, for the image to be identified, this application first obtains multiple candidate categories corresponding to its text modality. These multiple candidate categories contain the true categories of objects in the image to be identified. In subsequent steps, the true categories of objects can be selected from these multiple subsequent categories. Specifically, this application uses a configured multimodal large model to extract the text features of each candidate category and to extract the visual features of the image to be identified. Since the multimodal large model has been pre-trained on a large-scale dataset (text, image, video, and other multimodal data) and has a larger number of network parameters, the multimodal large model itself has a powerful ability to represent general knowledge. The text features of each candidate category extracted by the multimodal large model can more accurately represent the candidate category, and the visual features of the image to be identified extracted by the multimodal large model can more accurately represent the image to be identified. Based on this, the similarity between the visual features and the text features of each candidate category can be calculated, and the candidate category corresponding to the highest similarity can be selected as the target category of the object in the image to be identified. Clearly, this application leverages the powerful knowledge representation capabilities of multimodal large models to more accurately extract textual features of candidate categories and visual features of images to be identified. It then selects the target category of objects in the images to be identified by comparing feature similarity, thereby improving recognition accuracy. This avoids the problem of low recognition accuracy in traditional object recognition models when training data is insufficient, as well as the problem of low recognition accuracy when faced with images taken from special perspectives. Attached Figure Description

[0045] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0046] Figure 1 An example image was taken by a robot vacuum cleaner;

[0047] Figure 2 This is a flowchart illustrating the object recognition method provided in an embodiment of this application;

[0048] Figure 3 A flowchart illustrating an object recognition method based on a multimodal large model is provided.

[0049] Figure 4 A flowchart illustrating an object recognition method based on a combination of multimodal large model and sequence model is provided.

[0050] Figure 5 A flowchart illustrating another object recognition method based on a combination of multimodal large models and sequence models is provided.

[0051] Figure 6 An example of an environment map for a robot constructed using a SLAM algorithm is provided.

[0052] Figure 7 Example of a floor plan map;

[0053] Figure 8 This is a schematic diagram of the structure of an object recognition device provided in an embodiment of this application;

[0054] Figure 9 This is a schematic diagram of the structure of the object recognition device provided in the embodiments of this application. Detailed Implementation

[0055] Before introducing the proposed solution, let's first explain the basic concepts involved in this paper:

[0056] Artificial intelligence models, also known as large-scale deep learning models, are artificial intelligence models based on deep learning technology. They consist of hundreds of millions of parameters and can perform complex tasks such as natural language processing, image recognition, and speech recognition through learning and training on massive amounts of data. Artificial intelligence models can include large models.

[0057] Large models, also known as large-scale pre-trained models, refer to models pre-trained on large-scale datasets. Depending on the dataset, large models can be categorized into different modalities, such as language large models, image large models, speech large models, and multimodal large models. Multimodal large models can process data from multiple modalities simultaneously. Their training datasets typically include large amounts of text, images, videos, and other multimodal data from the internet. Large models acquire general knowledge representations by learning patterns and features from the data. The goal of these large models is to learn a universal representation capability for fine-tuning or transfer learning in various downstream tasks.

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] This application provides an object recognition scheme applicable to scenarios requiring object recognition in images. For example, it can be applied to object recognition in images of a work area captured by a robot (such as a robotic vacuum cleaner or a transport robot). The device executing the object recognition method of this application can be a robot. For instance, after capturing a video stream using a camera module, the robot extracts several frames from the stream that require object recognition, and then executes the object recognition method of this application on each frame.

[0060] In addition, the object recognition method of this application can also be implemented based on terminal devices such as base stations, servers, and smart speakers that communicate with the robot. For example, after the robot captures a video stream, it transmits it to the terminal device, which then extracts several frames of images that need to be recognized as objects, and then performs the object recognition method of this application on each frame of image.

[0061] Alternatively, the object recognition method of this application can also be implemented by the cooperation of a robot and a terminal device.

[0062] Next, combined Figure 2 The object recognition method of this application may include the following steps:

[0063] Step S100: Obtain multiple candidate categories for the text modality corresponding to the image to be recognized.

[0064] The multiple candidate categories include the true categories of the objects in the image to be identified.

[0065] The image to be identified is the image that needs to be identified as an object. It can be provided by the user or taken by a device, such as an image taken by a camera module on a robot.

[0066] In this step, obtaining multiple candidate categories corresponding to the image to be identified refers to obtaining multiple candidate results after performing object category identification on the image to be identified. The multiple candidate categories contain all possible categories of objects in the image to be identified, and therefore also contain the true category of the object. The purpose of this application is to determine the true category of the object from the multiple candidate categories, which can also be called the target category of the object.

[0067] Taking the image to be identified as a picture taken by a robot vacuum cleaner as an example, the multiple candidate categories can be various types of furniture, objects, etc. in the working area where the robot vacuum cleaner is located.

[0068] In this step, multiple candidate categories corresponding to the image to be identified can be obtained in various ways. For example, a pre-trained object recognition model can be used to perform object recognition on the image to be identified, obtaining multiple candidate categories output by the model. This object recognition model can be trained using training images labeled with object category labels. Various types of neural network models can be used for object recognition. Although the accuracy of the object recognition model may not be particularly high, it can output multiple candidate categories (generally 10 or other numbers can be selected), and these multiple candidate categories usually include the correct category of the object.

[0069] For example, users can input all possible categories of objects in the image to be recognized, which can then be used as multiple candidate categories. Alternatively, candidate categories of objects in images to be recognized can be obtained from other terminals, and so on.

[0070] Step S110: Extract text features of each candidate category using the configured multimodal large model, and extract visual features of the image to be identified using the multimodal large model.

[0071] Specifically, this application pre-configures a multimodal large model, which can process input data of different modalities, such as text data and image data. By leveraging the powerful knowledge representation capabilities of the multimodal large model, the text features of each candidate category and the visual features of the image to be identified can be extracted more accurately.

[0072] Reference Figure 3 As shown, a multimodal large model can include a text encoder and a visual encoder. The text encoder can encode each candidate category separately to obtain the text features of each candidate category. The visual encoder performs visual encoding on the image to be recognized to obtain visual features.

[0073] Alternatively, before using the multimodal large model to extract text features of candidate categories and visual features of the image to be recognized, a process of fine-tuning the multimodal large model using training data can be added, namely:

[0074] Training data consisting of training images and labeled text is obtained. Training images and their corresponding labeled text form positive example pairs, while training images and their labeled texts form negative example pairs. The labeled text for each training image describes the objects contained within that image. A multimodal large model is used to extract visual features from the training images and textual features from the labeled text. The cosine similarity between the visual and textual features is calculated. The goal is to maximize the similarity between positive example pairs and minimize the similarity between negative example pairs. The parameters of the multimodal large model are then updated until the set convergence condition is met.

[0075] Through the above training process, the labeled text and its corresponding image can be further brought closer to the same vector representation space.

[0076] Step S120: Calculate the similarity between the visual features of the image to be identified and the text features of each candidate category, and take the candidate category corresponding to the highest similarity as the target category of the object in the image to be identified.

[0077] Specifically, each candidate category contains the true category of the object in the image to be identified. To filter out the true category, the similarity between the visual features of the image to be identified and the text features of each candidate category can be calculated, such as... Figure 3 As shown. It is understandable that, theoretically, the textual features of the true category of an object have the highest similarity to the visual features of the image to be identified. Therefore, in this step, the candidate category corresponding to the highest similarity can be selected as the target category of the object in the image to be identified (that is, the true category of the object determined by this application).

[0078] In this step, when calculating the similarity between visual features and text features, cosine distance or other distance calculation formulas can be used to calculate the similarity.

[0079] The object recognition method provided in this application first obtains multiple candidate categories of the corresponding text modality for the image to be recognized. These multiple candidate categories contain the true categories of the objects in the image to be recognized. In subsequent steps, the true categories of the objects can be selected from these multiple subsequent categories. Specifically, this application uses a configured multimodal large model to extract the text features of each candidate category and to extract the visual features of the image to be recognized. Since the multimodal large model has been pre-trained on a large-scale dataset (text, image, video, and other multimodal data) and has a larger number of network parameters, it has a powerful ability to represent general knowledge. The text features of each candidate category extracted by the multimodal large model can more accurately represent the candidate category, and the visual features of the image to be recognized extracted by the multimodal large model can more accurately represent the image to be recognized. Based on this, the similarity between the visual features and the text features of each candidate category can be calculated, and the candidate category corresponding to the highest similarity is selected as the target category of the object in the image to be recognized. Clearly, this application leverages the powerful knowledge representation capabilities of multimodal large models to more accurately extract textual features of candidate categories and visual features of images to be identified. It then selects the target category of objects in the images to be identified by comparing feature similarity, thereby improving recognition accuracy. This avoids the problem of low recognition accuracy in traditional object recognition models when training data is insufficient, as well as the problem of low recognition accuracy when faced with images taken from special perspectives.

[0080] In some embodiments of this application, an optional application scenario of the above-described object recognition method is introduced. Specifically, the above-described object recognition method can be applied to the process of constructing a floor plan map of the robot's work area.

[0081] Specifically, the robot uses its onboard camera module to capture video streams of the work area, and can then use each frame or part of the frames in the video as images to be identified. The object recognition method in this case is then applied to identify the category of the object in each frame of the image to be identified.

[0082] Based on this, a floor plan map of the robot's work area can also be obtained. This floor plan map can be an environmental map constructed using the SLAM algorithm, such as... Figure 6 As shown, different objects in a map can be represented by different colors or shades of gray.

[0083] Furthermore, the location of the object in each frame of the image to be identified is determined within the floor plan map.

[0084] Specifically, by combining radar sensors on the robot, the actual position of the object in each frame of the image to be identified can be determined in the working environment, and the position of the object in the floor plan can be obtained through position mapping.

[0085] Based on the target category of the object in each frame of the image to be identified, an object texture of that target category is displayed at the corresponding location of the object in the floor plan map. For example... Figure 7 As shown, it exemplifies a floor plan map containing object textures, which includes textures of identified furniture and items such as sofas, dining tables, and cabinets, making it easier for users to browse the map more intuitively.

[0086] Taking the scenario where the robot captures some or all frames of the video stream of the work area as the image to be identified as an example, this application further introduces several different implementations of the object recognition method in this scenario.

[0087] Considering that the placement of different objects within the robot's working area generally has an inherent order, in order to fully explore the relationship information between objects and thus improve the accuracy of object recognition, this embodiment can further introduce a sequence model on the basis of the multimodal large model.

[0088] A sequence model is a model that processes sequential data (data that appears in a specific order). Common sequence models include Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs). The characteristic of sequence data is that the data at a given moment is not only related to the present but also to the past and future; the order in which the data appears is also important information, and sequence models can learn this information.

[0089] In this embodiment, considering that the placement of objects within the robot's working area generally has an inherent order, the advantage of the sequence model in naturally processing sequence data can be utilized to sequentially feed the visual features of each frame of the image to be identified in the video stream into the sequence model, thereby extracting the hidden state features of each frame of the image to be identified. These hidden state features can better characterize the feature information of the objects in the image to be identified, as well as the correlation information between the objects and the objects in the preceding and following frames of the image to be identified.

[0090] In this embodiment, a multimodal large model can be combined with a sequence model to perform object recognition.

[0091] An alternative combination scheme is referenced. Figure 4 As shown, it may include:

[0092] For each frame of the image to be identified, multiple candidate categories are obtained through an object recognition model, and then the text features of each candidate category are extracted by a multimodal large model text encoder.

[0093] For each frame of the image to be identified, its visual features are extracted by a multimodal large model visual encoder. Then, the visual features of each frame of the image to be identified are fed into the sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model. The hidden state features are used as the latest visual features of each frame of the image to be identified, and similarity is calculated with the text features of each candidate category. The candidate category with the highest similarity is selected as the target category of each frame of the image to be identified.

[0094] The sequence model is configured to extract the hidden state features of each frame of the input image sequence to be identified.

[0095] Sequence models can sequentially connect the visual features of each frame of the image to be recognized in the preceding and following inputs. The process of determining the hidden state features of the next frame of the image to be recognized can use the hidden state features of the previous frame of the image to be recognized, that is, it will take into account the sequential relationship between different frames of the image to be recognized in the preceding and following inputs.

[0096] Figure 4In the example, the hidden state features of the first frame of the image to be identified are passed to the hidden state feature extraction process of the (1+t)th frame of the image to be identified, where t is a positive integer greater than or equal to 1. That is, this application can perform object recognition on every frame of the video stream, in which case t is 1. In addition, this application can also extract several keyframes from the video stream at intervals for object recognition, where t is the number of frames between two extracted keyframes.

[0097] The scheme combining multimodal large model and sequence model provided in this embodiment extracts the visual features of each frame of the image to be identified using the multimodal large model visual encoder. Then, it can further extract the hidden state features of each frame of the image to be identified using the sequence model and use them as the latest visual features of each frame of the image to be identified. The latest visual features further consider the placement order information between objects in different frames of the image to be identified. Based on this, similarity is calculated with the text features of each candidate category. The candidate category with the highest similarity is taken as the target category of the object in each frame of the image to be identified. Since the inherent placement order information between different objects is considered in the process, the final recognition result is more accurate.

[0098] In Adoption Figure 4 Before performing object recognition, the model structure shown can be trained using training data. For example, training data consisting of training images and labeled text can be obtained. The training images and their corresponding labeled text form positive example pairs, while the training images and the labeled text of the other images form negative example pairs. The labeled text corresponding to the training images describes the objects contained within those images. Using… Figure 4 The structure shown processes the training data and updates the parameters of the multimodal large model and the sequence model with the goal of maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, until the set convergence condition is met.

[0099] Another alternative combination scheme is to... Figure 3 and Figure 4 The two model architectures in the example are further combined, using, for example... Figure 5 The model architecture shown performs object recognition for each frame of the image to be recognized.

[0100] like Figure 5 As shown, the entire model architecture can be divided into two modules (such as...). Figure 5 (The two dashed boxes at the top and bottom) The module above uses the following... Figure 3 The structure shown calculates the first similarity between the visual features of each frame of the image to be identified and the text features of each candidate category. For details, please refer to [link / reference needed]. Figure 3 The description of the corresponding embodiments will not be repeated here.

[0101] The following modules use, for example Figure 4 The structure shown calculates the second similarity between the hidden state features and the text features of each candidate category for each frame of the image to be identified. For details, please refer to [link / reference needed]. Figure 4 The description of the corresponding embodiments will not be repeated here.

[0102] After obtaining the first and second similarities as described above, we can proceed with the identification of each frame of the image to be identified:

[0103] Based on the first similarity and second similarity between each frame of the image to be identified and each candidate category, a third similarity is determined, and the candidate category corresponding to the highest third similarity is taken as the target category of the object contained in each frame of the image to be identified.

[0104] For example, for the i-th frame of the image to be recognized, by... Figure 5 The above module calculates the following:

[0105] The first similarity of candidate category x1 is q11, the first similarity of candidate category x2 is q12, and the first similarity of candidate category x3 is q13.

[0106] pass Figure 5 The following module calculates:

[0107] The second similarity of candidate category x1 is q21, the second similarity of candidate category x2 is q22, and the second similarity of candidate category x3 is q23.

[0108] Combining the first and second similarities, we obtain the third similarity:

[0109] The third similarity of candidate category x1 is q31, the third similarity of candidate category x2 is q32, and the third similarity of candidate category x3 is q33.

[0110] The candidate category corresponding to the highest third similarity is taken as the target category of the object contained in the i-th frame of the image to be identified. Assuming q31>q32>q33, then candidate category x1 can be taken as the target category of the object contained in the i-th frame of the image to be identified.

[0111] In one optional implementation, the process of determining the third similarity based on the first and second similarities between each frame of the image to be identified and each candidate category may include:

[0112] The first and second similarities of each frame of the image to be identified with each candidate category are averaged, and the average similarity is used as the third similarity.

[0113] In addition, other methods can be used to determine the third similarity, such as assigning different weights to the first and second similarities, and then taking a weighted average of the first and second similarities as the third similarity.

[0114] The above embodiments describe the process of combining a multimodal large model with a sequence model for object recognition. By leveraging the powerful knowledge representation capabilities of the multimodal large model, the arrangement order of different objects within the robot's working area can be considered. The sequence model can be used to mine the relationship information between these objects, thereby helping to further improve the accuracy of object recognition.

[0115] Based on this, in this embodiment, the robot's work area can be further divided according to rooms, that is, the work area includes several different rooms. In this embodiment, the placement order of objects is limited to the same room, that is, it is believed that there is a specific placement order between different objects within the same room, and there is no obvious placement order between objects across rooms. Based on this understanding, the process of feeding the visual features of each frame of the image to be recognized into the sequence model in the above embodiment may include:

[0116] If the image to be identified in the current frame is the first photo taken by the robot after entering a room in the work area, the hidden state features of the sequence model can be reset to 0 first, and then the visual features of the image to be identified in the current frame can be sent into the sequence model.

[0117] Based on this, sequence models can capture only the inherent placement logic between objects within the same room, while ignoring the placement logic between objects in different rooms, thus reducing the interference of other objects outside the room on the recognition of objects inside the room.

[0118] The object recognition device provided in the embodiments of this application is described below. The object recognition device described below can be referred to in correspondence with the object recognition method described above.

[0119] See Figure 8 , Figure 8 This is a schematic diagram of the structure of an object recognition device disclosed in an embodiment of this application.

[0120] like Figure 8 As shown, the device may include:

[0121] The candidate category acquisition unit 11 is used to acquire multiple candidate categories of the text modality corresponding to the image to be identified, wherein the multiple candidate categories include the true categories of objects in the image to be identified;

[0122] The feature extraction unit 12 is used to extract text features of each candidate category using the configured multimodal large model, and to extract visual features of the image to be identified using the multimodal large model.

[0123] The first similarity calculation unit 13 is used to calculate the similarity between the visual features of the image to be identified and the text features of each candidate category;

[0124] The target category determination unit is used to select the candidate category corresponding to the highest similarity as the target category of the object in the image to be identified.

[0125] Optionally, the process of the above feature extraction unit extracting text features of each candidate category using the configured multimodal large model, and extracting visual features of the image to be identified using the multimodal large model, includes:

[0126] Using the text encoder of the multimodal large model, text encoding is performed on each of the candidate categories to obtain the text features of each candidate category;

[0127] The visual encoder of the multimodal large model is used to visually encode the image to be identified to obtain visual features.

[0128] Optionally, the image to be identified can be several frames from a video stream captured by the robot of the work area. Based on this, the apparatus of this application may further include:

[0129] The first sequence model processing unit is configured to, before the first similarity calculation unit calculates the similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified, feed the visual features of each frame of the image to be identified into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model, and use the hidden state features as the latest visual features of each frame of the image to be identified. The sequence model is configured to extract the hidden state features of each frame of the image to be identified in the input sequence of images to be identified.

[0130] Optionally, the image to be identified can be several frames from a video stream captured by the robot of the work area. Based on this, the first similarity calculation unit calculates the similarity between the visual features of the image to be identified and the text features of each candidate category, including:

[0131] Calculate the first similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified. The apparatus of this application may further include:

[0132] The second sequence model processing unit is used to send the visual features of each frame of the image to be identified into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model, wherein the sequence model is configured to extract the hidden state features of each frame of the image to be identified in the input image sequence.

[0133] The second similarity calculation unit is used to calculate the second similarity between the hidden state features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified. The process by which the target category determination unit takes the candidate category corresponding to the highest similarity as the target category of the object contained in the image to be identified includes:

[0134] For each frame of the image to be identified:

[0135] Based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category, a third similarity is determined, and the candidate category corresponding to the highest third similarity is taken as the target category of the object contained in each frame of the image to be identified.

[0136] Optionally, the process by which the target category determination unit determines a third similarity based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category includes:

[0137] The first similarity and the second similarity of each frame of the image to be identified with each candidate category are averaged, and the average similarity is taken as the third similarity.

[0138] Optionally, the robot's work area includes at least one room. The process by which the first sequence model processing unit and the second sequence model processing unit input the visual features of each frame of the image to be recognized into the pre-configured sequence model includes:

[0139] If the image to be identified in the current frame is the first photo taken by the robot after entering a room in the work area, the hidden state features of the sequence model are reset to 0, and then the visual features of the image to be identified in the current frame are sent into the sequence model.

[0140] Optionally, the image to be identified is a number of frames from a video stream captured by the robot of the work area, and the device may further include:

[0141] A floor plan acquisition unit is used to acquire a floor plan of the work area corresponding to the robot.

[0142] An object location determination unit is used to determine the location of an object in each frame of the captured image to be identified within the floor plan map.

[0143] The mapping unit is used to display an object mapping of the target category at the location corresponding to the object in the floor plan map, according to the target category of the object in each frame of the image to be identified.

[0144] Optionally, the process by which the candidate category acquisition unit acquires multiple candidate categories for the text modality corresponding to the image to be recognized includes:

[0145] The image to be identified is fed into a pre-configured object recognition model to obtain multiple candidate categories output by the object recognition model; the object recognition model is trained using training images labeled with object category tags.

[0146] The object recognition device provided in this application embodiment can be applied to object recognition equipment, such as robots, computers, servers, and smart terminals. Optionally, Figure 9 The hardware structure block diagram of the object recognition device is shown below. Figure 9 The hardware structure of an object recognition device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0147] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0148] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0149] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0150] The memory stores a program, which the processor can call. The program is used for:

[0151] Obtain multiple candidate categories for the text modality corresponding to the image to be identified, wherein the multiple candidate categories include the true categories of objects in the image to be identified;

[0152] The text features of each candidate category are extracted using the configured multimodal large model, and the visual features of the image to be identified are extracted using the multimodal large model.

[0153] Calculate the similarity between the visual features of the image to be identified and the text features of each candidate category, and take the candidate category corresponding to the highest similarity as the target category of the object in the image to be identified.

[0154] Optionally, the refined and extended functions of the program can be found in the description above.

[0155] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0156] Obtain multiple candidate categories for the text modality corresponding to the image to be identified, wherein the multiple candidate categories include the true categories of objects in the image to be identified;

[0157] The text features of each candidate category are extracted using the configured multimodal large model, and the visual features of the image to be identified are extracted using the multimodal large model.

[0158] Calculate the similarity between the visual features of the image to be identified and the text features of each candidate category, and take the candidate category corresponding to the highest similarity as the target category of the object in the image to be identified.

[0159] Optionally, the refined and extended functions of the program can be found in the description above.

[0160] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, object, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, object, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, object, or apparatus that includes said element.

[0161] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0162] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An object recognition method, characterized in that, include: Obtain multiple candidate categories for the text modality corresponding to the image to be identified, wherein the multiple candidate categories include the true categories of objects in the image to be identified; The images to be identified are several frames from a video stream captured by the robot of the working area; The text features of each candidate category are extracted using the configured multimodal large model, and the visual features of the image to be identified are extracted using the multimodal large model. Calculate the first similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified; The visual features of each frame of the image to be identified are fed into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model. The sequence model is configured to extract the hidden state features of each frame of the image to be identified in the input image sequence. Calculate the second similarity between the hidden state features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified; For each frame of the image to be identified: based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category, a third similarity is determined, and the candidate category corresponding to the highest third similarity is taken as the target category of the object contained in each frame of the image to be identified.

2. The method according to claim 1, characterized in that, The text features of each candidate category are extracted using the configured multimodal large model, and the visual features of the image to be identified are extracted using the multimodal large model, including: Using the text encoder of the multimodal large model, text encoding is performed on each of the candidate categories to obtain the text features of each candidate category; The visual encoder of the multimodal large model is used to visually encode the image to be identified to obtain visual features.

3. The method according to claim 1, characterized in that, Before calculating the similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified, the method further includes: The visual features of each frame of the image to be identified are fed into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model, and the hidden state features are used as the latest visual features of each frame of the image to be identified. The sequence model is configured to extract the hidden state features of each frame of the image to be identified in the input image sequence.

4. The method according to claim 1, characterized in that, The process of determining the third similarity based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category includes: The first similarity and the second similarity of each frame of the image to be identified with each candidate category are averaged, and the average similarity is taken as the third similarity.

5. The method according to claim 1 or 3, characterized in that, The robot's work area includes at least one room; The process of feeding the visual features of each frame of the image to be identified into a pre-configured sequence model includes: If the image to be identified in the current frame is the first photo taken by the robot after entering a room in the work area, the hidden state features of the sequence model are reset to 0, and then the visual features of the image to be identified in the current frame are sent into the sequence model.

6. The method according to claim 1, characterized in that, The images to be identified are several frames from a video stream captured by the robot of the working area. The method further includes: Obtain a floor plan map of the work area corresponding to the robot; Determine the location of the object in each frame of the captured image to be identified within the floor plan map; According to the target category of the object in each frame of the image to be identified, the object texture of the target category is displayed at the corresponding location of the object in the floor plan map.

7. The method according to any one of claims 1 to 4, 6, characterized in that, The process of obtaining multiple candidate categories for the text modality corresponding to the image to be identified includes: The image to be identified is fed into a pre-configured object recognition model to obtain multiple candidate categories output by the object recognition model; the object recognition model is trained using training images labeled with object category tags.

8. An object recognition device, characterized in that, include: The candidate category acquisition unit is used to acquire multiple candidate categories of the text modality corresponding to the image to be identified. The multiple candidate categories include the true categories of objects in the image to be identified. The image to be identified is a number of frames in a video stream captured by the robot of the working area. The feature extraction unit is used to extract text features of each candidate category using the configured multimodal large model, and to extract visual features of the image to be identified using the multimodal large model. The first similarity calculation unit is used to calculate the first similarity between the visual features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified. The visual features of each frame of the image to be identified are fed into a pre-configured sequence model to obtain the hidden state features of each frame of the image to be identified extracted by the sequence model. The sequence model is configured to extract the hidden state features of each frame of the image to be identified from the input sequence of images to be identified. A second similarity is calculated between the hidden state features of each frame of the image to be identified and the text features of each candidate category corresponding to each frame of the image to be identified. For each frame of the image to be identified, a third similarity is determined based on the first similarity and the second similarity between each frame of the image to be identified and each candidate category. The target category determination unit is used to take the candidate category corresponding to the highest third similarity as the target category of the object contained in each frame of the image to be identified.

9. An object recognition device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the object recognition method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the object recognition method as described in any one of claims 1 to 7.

Images