Object recognition method, device, terminal device and computer readable storage medium

Abstract

The application discloses an object recognition method and device, a terminal device and a computer readable storage medium. The method comprises the following steps: acquiring a first image, region attention information corresponding to the first image and question information corresponding to the first image, the first image being a global image of an object to be recognized, the region attention information comprising position information of a region of interest in the first image specified by a user, and the question information being information of a question about the object to be recognized raised by the user; and recognizing the object to be recognized according to the first image, the region attention information and the question information, and determining an answer to the question. The method can utilize information of a region of interest prompted by the user to recognize the object, and significantly improves the object recognition accuracy.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to an object recognition method, device, terminal equipment and computer-readable storage medium. Background Technology

[0002] Object recognition is an important branch of computer vision, involving the identification and classification of objects in images. There are various object recognition methods, including multimodal recognition methods. These methods combine information from multiple modalities, such as images and text, to identify objects, providing richer contextual information and improving the robustness of recognition.

[0003] However, traditional multimodal recognition methods typically perform well only in recognizing a limited number of object categories. In recognizing multiple object categories, especially in recognizing all things, they suffer from high training costs and inaccurate recognition. Summary of the Invention

[0004] This application provides an object recognition method, apparatus, terminal device, and computer-readable storage medium, which can perform object recognition using information from user-provided regions of interest, significantly improving object recognition accuracy.

[0005] A first aspect of this application provides an object recognition method, comprising: acquiring a first image, region interest information corresponding to the first image, and question information corresponding to the first image, wherein the first image is a global image of an object to be recognized, the region interest information includes the location information of a region of interest in the first image specified by a user, and the question information is information about a question raised by the user regarding the object to be recognized; and recognizing the object to be recognized based on the first image, the region interest information, and the question information to determine the answer to the question.

[0006] In one implementation, identifying the object to be identified and determining the answer to the question based on a first image, region interest information, and question information includes: inputting multimodal information into a multimodal model to obtain the answer, wherein the multimodal information is determined based on the first image, region interest information, and question information, and the multimodal model includes a feature alignment module, which is used to align the global image feature information of the first image with the local image feature information of the region of interest.

[0007] In one implementation, the multimodal information includes a first image, a second image, and first text. The second image is an image of the region of interest, and the first text is the text of the question. The multimodal model further includes an image encoding module, an image convolution module, a text encoding module, and a decoding module. Inputting the multimodal information into the multimodal model includes: inputting the first text into the text encoding module to obtain the text feature information of the first text; using a feature alignment module to perform feature alignment on the global image feature information and the local image feature information to obtain aligned image feature information, wherein the global image feature information is obtained by encoding the first image using the image encoding module, and the local image feature information is obtained by convolving the second image using the image convolution module; concatenating the text feature information and the aligned image feature information to obtain concatenated text image feature information; and inputting the text image feature information into the decoding module to obtain the second text about the answer.

[0008] In one implementation, before inputting multimodal information into the multimodal model, the method further includes: acquiring a training dataset, wherein the training dataset includes text sample data, image sample data, and image-text pair sample data, the image-text pair sample data including global image samples, local image samples, and question text samples; and performing a training operation on the multimodal model based on the training dataset and a preset loss function, wherein the preset loss function includes a first loss term, the first loss term being used to represent the consistency between the aligned image feature information inferred by the feature alignment module and the text feature information inferred by the text encoding module.

[0009] In one implementation, the preset loss function includes a first loss function and a second loss function. Based on the training dataset and the preset loss function, a training operation on the multimodal model is performed, including: in the pre-training phase of the training operation, training the multimodal model using text sample data, image sample data, image-text pair sample data, and the first loss function, wherein the first loss function includes a first loss term and a second loss term, the second loss term representing the difference between the inference result of the multimodal model on the sample data and the true label corresponding to the sample data; and in the model fine-tuning phase of the training operation, training the multimodal model using image-text pair sample data and the second loss function, wherein the second loss function includes a second loss term.

[0010] In one implementation, the first loss term is expressed using the following formula:

[0011]

[0012] Where loss1 represents the first loss term, and x represents the global image sample. t represents a local image sample, b[0] represents the aligned image feature information inferred by the feature alignment module, and f[0] represents the text feature information inferred by the text encoder.

[0013] In one embodiment, the feature alignment module includes a first fully connected network layer and a second fully connected network layer. The feature alignment module performs feature alignment on global image feature information and local image feature information, including: superimposing global image feature information and local image feature information to obtain superimposed image feature information; inputting the superimposed image feature information into the first fully connected network layer to obtain transformed image feature information, wherein the number of feature dimensions of the transformed image feature information is greater than the number of feature dimensions of the superimposed image feature information; and inputting the transformed image feature information into the second fully connected network layer to obtain aligned image feature information, wherein the number of aligned image feature information is equal to the number of feature dimensions of the superimposed image feature information.

[0014] A second aspect of this application provides an object recognition device, comprising: an acquisition module, configured to acquire a first image, region interest information corresponding to the first image, and question information corresponding to the first image, wherein the first image is a global image of an object to be recognized, the region interest information includes location information of a region of interest in the first image specified by a user, and the question information is information about a question raised by a user regarding the object to be recognized; and a recognition module, configured to recognize the object to be recognized based on the first image, the region interest information, and the question information, and determine the answer to the question.

[0015] A third aspect of this application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the object recognition method described above.

[0016] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the object recognition method described above.

[0017] The fifth aspect of this application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps in the object recognition method described above.

[0018] The object recognition method provided in the first aspect of this application first acquires a global image of the object to be recognized, user question information, and user region attention information for regions of interest in the global image. Then, it identifies the object to be recognized based on the global image, region attention information, and question information, quickly and accurately determining the answer to the question. This scheme fully utilizes the user-provided region attention information to identify the object to be recognized, enabling rapid and accurate focus on the user-specified region for recognition. This allows for more accurate identification and analysis of objects within that region, thereby improving the accuracy and efficiency of recognition. Furthermore, focusing on a specific region based on the contextual information of the global image reduces the amount of data required for recognition computation, thereby increasing processing speed and optimizing the use of computing resources. In complex scenes, the user-specified region can contain clearer object features, maintaining a high recognition rate even when the global image is cluttered or the background is complex, thus improving the robustness of object recognition. Moreover, the user-specified region can serve as key information for the visual modality, combining with question information from other modalities (such as text or audio) to achieve a deeper level of understanding and recognition, making this recognition scheme suitable for various scenarios. Therefore, this scheme improves the accuracy, efficiency, robustness, and applicability of object recognition, resulting in a better user experience.

[0019] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an object recognition method provided in one embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a scenario for acquiring a first image and a second image according to an embodiment of this application;

[0023] Figure 3a This is a schematic diagram of the structure of a multimodal recognition model provided in one embodiment of this application;

[0024] Figure 3b This is a schematic diagram of the structure of an image encoding module provided in one embodiment of this application;

[0025] Figure 4This is a schematic diagram of the structure of an object recognition device provided in one embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of a terminal device provided in one embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information involved in this application embodiment all comply with the provisions of relevant laws and regulations, have obtained the user's authorization or consent, have taken necessary confidentiality measures, and do not violate public order and good morals.

[0033] As mentioned earlier, traditional multimodal recognition methods typically perform well only when recognizing a limited number of object categories. For recognizing multiple object categories, especially objects of all kinds, they suffer from high training costs and inaccurate recognition. For example, in the field of service robots, using traditional large multimodal models to recognize household items results in low accuracy, making it difficult to effectively empower robots, failing to meet user needs, and leading to a poor user experience.

[0034] Research has revealed that traditional multimodal large-scale models achieve object recognition primarily through visual question-answering tasks. This requires a large amount of labeled data, including triples of images, questions, and answers. To enable the model to learn to recognize multiple object categories, these datasets need to cover various question patterns and answer types, making dataset labeling extremely time-consuming, resulting in significant training time and difficulty. Furthermore, different visual question-answering tasks reduce the accuracy of object recognition.

[0035] To at least partially address the aforementioned technical problems, embodiments of this application provide an object recognition method, apparatus, terminal device, and computer-readable storage medium. These can be applied to various scenarios requiring object recognition, particularly those requiring the recognition of everything, including but not limited to object recognition scenarios in retail and e-commerce, intelligent warehousing and logistics, urban management and planning, autonomous driving, healthcare, security monitoring, environmental protection, intelligent transportation, and smart healthcare. The object recognition scheme of this application not only acquires a global image of the object to be recognized and the user's question information, but also fully acquires the user's regional interest information regarding regions of interest in the global image. By fully utilizing the regional interest information provided by the user, the object to be recognized is identified, and the answer to the question is obtained, significantly improving the accuracy of object recognition.

[0036] like Figure 1 As shown, the object recognition method provided in this application embodiment includes the following steps S110 and S220.

[0037] Step S110: Obtain a first image, corresponding region interest information and corresponding question information for the first image. The first image is a global image of the object to be identified. The region interest information includes the location information of the region of interest in the first image specified by the user. The question information is the information of the question raised by the user about the object to be identified.

[0038] In this embodiment, the object to be identified can be any suitable physical object in nature, and this application does not limit it. The first image can be a global scene image of the object to be identified. The first image may include at least the imaging area of ​​the object to be identified, and may also include the imaging areas of the environment and other objects surrounding the object to be identified. Figure 2As shown, the object to be identified is the dog or a part of the dog in the image. The first image may include not only the dog, but also the cat next to it and the surrounding grass, trees, etc. The region of interest (ROI) is the area of ​​the dog's body marked by the dashed line in the image. This ROI is the area that the user focuses on. The information for this region may include at least the location information of the ROI in the first image. The query information can be any question posed by the user about the object to be identified, including but not limited to information such as category, color, and shape. For example, the user could ask about the dog's scientific name, canine class, color, habits, personality traits, feeding method, and the shape of a part of the dog. This query information can take various forms, including but not limited to text information and audio information.

[0039] In this application embodiment, a variety of suitable methods can be used to obtain the above three types of information.

[0040] In one example, step S110 may include the following steps: providing a first interface, the first interface including an image capture control and a question control; in response to a first operation by the user on the image capture control, determining a first image and corresponding regional attention information of the first image; in response to a second operation by the user on the question control, determining question information corresponding to the first image.

[0041] For example, the first interface could be the interface of a device recognition app on a user's mobile phone. After the user clicks the image capture control, an image capture window is displayed. This image capture window can be a camera window or an album window. For example, the camera window not only displays the real-time image captured by the phone's camera and the shooting control, but also includes a region bounding box in the image. This region bounding box can have any suitable shape. Of course, the camera window can also include a region shape selection control or a region shape editing control, which the user can use to select or edit the shape of the region bounding box. That is, when the user operates the region shape selection control or the region shape editing control, the shape of the region bounding box in the image can change accordingly. When the field of view of the phone's camera changes, the real-world object in the region bounding box also changes accordingly. In this way, the user can control the camera to capture a global image of the object they want to recognize by moving the phone, operating the region shape selection control or the region shape editing control, and operating the shooting control, and accurately provide prompts for the local area of ​​interest. For example, in response to the user's click of the shooting control, a first image and the corresponding region of interest information for the first image can be determined. More specifically, one can obtain a first image and a second image of the region of interest obtained by cropping the first image according to the region bounding box. For example, one can obtain... Figure 2The image includes a global image (e.g., excluding the region bounding box shown by the dashed line in the figure) and a partial image cropped from that region bounding box, containing only the dog. Alternatively, the user can select the first image to be recognized from the album window and edit the region of interest in the subsequently displayed region editing window. The question control can be a text input control or a voice input control. Users can input their questions about the object to be recognized into the mobile device by operating the question control. For example, a user can click the voice input control and say, "What breed is the dog in the picture?" The mobile device will then receive the audio information "What breed is the dog in the picture?" as the corresponding... Figure 2 The question information.

[0042] In another example, step S110 may include the following steps: providing a second interface, which includes an image capture control, a region editing control, and a question control; determining a first image in response to a third user action on the image capture control; determining region interest information corresponding to the first image in response to a fourth user action on the region editing control; and determining question information corresponding to the first image in response to a fifth user action on the question control. In this example, the user can first capture or select a first image using the image capture control, and then select the region of interest using the region editing control.

[0043] Step S120: Based on the first image, region attention information, and question information, identify the object to be identified and determine the answer to the question.

[0044] In this embodiment of the application, after obtaining the first image, region attention information, and question information, various suitable information can be used to identify the object to be identified, and the answer to the user's question can be obtained based on the identification result.

[0045] In one example, a pre-trained multimodal model can be used to identify the object to be recognized. The multimodal model can be a large-scale multimodal model capable of recognizing everything. The multimodal model can have various suitable structures, and this embodiment does not limit it. Specifically, the input information of the model can be determined based on the first image, region of interest information, and question information. For example, the model can obtain the first image, a partial image of the region of interest selected based on user prompts (the second image below), and the question text asked by the user. These three can be used as input information to the multimodal model, and the answer to the question can be determined based on the model's output.

[0046] In other examples, other object recognition methods can be used to determine the answer to the question based on the first image, region of interest information, and question information. It should be noted that this object recognition method can fully consider the correlation between the image features of the first image, the image features of the region of interest, and the question features in the question information, and determine the answer to the question based on this correlation. For example, it can be combined with traditional computer vision techniques, utilizing template matching, feature extraction, and machine learning classifiers for object recognition.

[0047] For example, the answer can be displayed on the user's mobile app interface for the user to view. For instance, after a user asks, "What breed is the dog in the picture?", the reply "The dog in the picture is a Labrador" can be displayed in the chat box on the interface.

[0048] As mentioned earlier, traditional multimodal recognition methods typically perform well only in recognizing a limited number of object categories. For recognizing multiple object categories, especially objects of all kinds, they suffer from high training costs and inaccurate recognition. However, the object recognition method according to this application first acquires a global image of the object to be recognized, the user's question information, and the user's region of interest information within the global image. Then, it identifies the object based on the global image, region of interest information, and question information, quickly and accurately determining the answer to the question. This approach fully utilizes the user-provided region of interest information to identify the object, allowing for rapid and accurate focus on the user-specified region for recognition. This leads to more accurate identification and analysis of objects within that region, improving accuracy and efficiency. Furthermore, focusing on a specific region based on the contextual information of the global image reduces the amount of data required for recognition computation, thereby increasing processing speed and optimizing the use of computing resources. In complex scenes, the user-specified region can contain clearer object features, maintaining a high recognition rate even with interference in the global image or complex backgrounds, thus improving the robustness of object recognition. Furthermore, the user-specified region can serve as key information for the visual modality. Combined with question information from other modalities (such as text or audio), it enables deeper understanding and recognition, allowing the recognition scheme to meet the needs of multiple scenarios. Therefore, this scheme improves the accuracy, efficiency, robustness, and applicability of object recognition, resulting in a better user experience.

[0049] In one implementation, step S120 identifies the object to be identified based on the first image, region attention information, and question information, and determines the answer to the question, including step S121.

[0050] Step S121: Input the multimodal information into the multimodal model to obtain the answer. The multimodal information is determined based on the first image, region interest information, and question information. The multimodal model includes a feature alignment module, which aligns the global image feature information of the first image with the local image feature information of the region of interest.

[0051] In one example, the multimodal information may include a first image, a partial image of the region of interest in the first image, and the text of the user's question. In another example, the multimodal information may include a first image, the location coordinates of the region of interest in the first image, and the text of the user's question.

[0052] In this embodiment, the multimodal model can be a deep learning model capable of processing and integrating visual and textual information. The multimodal model can have various suitable model structures, as long as it has the ability to extract image features, extract text features, align global image features of the first image with local image features of the region of interest, and integrate and classify the aligned image features with text features. Specifically, the multimodal model includes a feature alignment module. The feature alignment module can have various suitable network structures, as long as it can align the global image feature information of the first image with the local image feature information of the region of interest. For example, the feature alignment module consists of one or more of a fully connected layer, an adaptive normalization layer, an adaptive convolutional layer, and an adaptive pooling layer.

[0053] In the above scheme, the feature alignment module enables the multimodal model to more accurately identify and analyze key regions in the image, improving recognition accuracy. Therefore, the scheme of integrating global image, region attention information, and question information using the above multimodal model for object recognition can further improve the accuracy and efficiency of multimodal recognition and also contribute to achieving real-time object recognition.

[0054] In one implementation, the multimodal information includes a first image, a second image, and first text. The second image is an image of the region of interest, and the first text is the text of the question. The multimodal model also includes an image encoding module, an image convolution module, a text encoding module, and a decoding module. Step S121 inputs the multimodal information into the multimodal model, including the following steps: Step S1211, inputs the first text into the text encoding module to obtain the text feature information of the first text; Step S1212, uses the feature alignment module to perform feature alignment on the global image feature information and the local image feature information to obtain aligned image feature information, wherein the global image feature information is obtained by encoding the first image using the image encoding module, and the local image feature information is obtained by convolution on the second image using the image convolution module; Step S1213, performs feature concatenation on the text feature information and the aligned image feature information to obtain concatenated text image feature information; Step S1214, inputs the text image feature information into the decoding module to obtain the second text about the answer.

[0055] In one example, such as Figure 3a As shown, the multimodal model may include a global image encoder (image encoding module), a user prompting region convolutional network (image convolution module), a text encoder (text encoding module), a feature alignment network (feature alignment module), and an object recognition decoder (decoding module).

[0056] For example, the infrastructure of the global image encoder, text encoder, and object recognition decoder can be based on Transformer, the feature alignment network can be a fully connected feedforward network, and the user prompt region convolutional network can be a traditional deep convolutional network.

[0057] For example, a global image encoder can be an encoder based on a transformer architecture. Assume the input is a first image (a color image or a grayscale image; RGB images will be used as an example below). The first image can be represented by x, where x∈R. h ×w×c Where h and w represent the length and width of the image, and c represents the number of RGB image channels. For example... Figure 3b As shown, a global image encoder can include a representation mapping layer, a multi-head self-attention layer, and a fully connected feedforward network layer. For example, the input to the multi-head self-attention layer is z = [x0, x1E, ..., x]. p E]+P, where x0 is a learnable global image classification vector, x1,…,x p To equally divide the matrix into blocks for x, Let P be a learnable mapping matrix, where P ∈ R. (p+1)×dLet be the learnable positional encoding matrix, l be the dimension of the equally divided blocks, and d be the dimension of the custom hidden feature layer. The number of multi-head self-attention layers and fully connected feedforward network layers is, for example, N. The computation of the i-th layer and the j-th attention head can be expressed by the following formula:

[0058] Retrieval Vector in

[0059] Keyword vector in

[0060] key-value vector in

[0061] The output of the multi-head self-attention layer is in

[0062] The input to the fully connected feedforward network layer is Where J represents the total number of self-attention heads;

[0063] The output of the fully connected feedforward network layer is in These are model parameters;

[0064] Assuming there are N blocks of self-attention, then we have

[0065] For example, to enhance the local feature learning ability of a multimodal model, a user-hint region convolutional network with a convolutional structure is configured. The second image, for example, uses... express, in and Here, represents the length and width of the user feedback region, respectively, and 'c' represents the number of channels in the RGC image. The user feedback region convolutional network can be a ResNet convolutional network structure. For example, a user feedback region convolutional network using... If it is indicated, then there is in

[0066] In this embodiment, the feature alignment network (feature alignment module) can have various suitable structures. In one embodiment, the feature alignment module includes a first fully connected network layer and a second fully connected network layer. Step S1212 uses the feature alignment module to perform feature alignment on global image feature information and local image feature information, including the following steps: Step S1212.1, superimposing global image feature information and local image feature information to obtain superimposed image feature information; Step S1212.2, inputting the superimposed image feature information into the first fully connected network layer to obtain transformed image feature information, wherein the number of feature dimensions of the transformed image feature information is greater than the number of feature dimensions of the superimposed image feature information; Step S1212.3, inputting the transformed image feature information into the second fully connected network layer to obtain aligned image feature information, wherein the number of aligned image feature information is equal to the number of feature dimensions of the superimposed image feature information.

[0067] For example, to align global image feature information, local image feature information, and text feature information, the global image feature information and local image feature information can be superimposed to obtain superimposed image feature information 'a'. For example, Where a∈R (p+1)×d The feature alignment network can employ two fully connected network layers. The weight matrix of the first fully connected layer can be represented by A1, and the weight matrix of the second fully connected layer can be represented by A2. Taking the example where the number of feature dimensions of the transformed image feature information is twice the number of feature dimensions of the superimposed image feature information, the aligned image feature information output by the feature alignment network, denoted by b, is then: Where A1∈R d×2d A2∈R 2d×d In the first fully connected network layer, each input feature is mapped to two new output features. In the second fully connected network layer, each 2D feature is mapped back to a d-dimensional image feature. The final output feature dimension is the same as the original input image feature dimension, achieving feature alignment.

[0068] In the above scheme, the alignment of global and local image features is achieved through two fully connected network layers. This not only improves the accuracy of the alignment but also enhances the model's adaptability to complex image transformations. Furthermore, the two fully connected network layers have a simple structure and low computational cost, thus saving computational resources and increasing computational speed. This improves both the accuracy and efficiency of object recognition.

[0069] For example, considering the full information of the text, a network text encoder can adopt a BERT structure. For instance, if the user prompt text input is 't', and its longest possible statement is set to 'L', when the length of the user prompt text is less than 'L', a BERT structure can be used. Padding is applied to length L. The feature representation of the user prompt text can be f = BERT(t), where f ∈ R. (L+1)×d .

[0070] For example, the structure of the object recognition decoder can be a Transformer-based decoder, specifically a GPT (Generative Pre-trained Transformer) structure. The object recognition decoder can convert multimodal features into a final text answer; this process can be a classification task (selecting from predefined answers) or a generation task (generating free-form answers).

[0071] For example, during training, the object recognition decoder can decode based on local image features and local text features. The input g of the object recognition decoder can be expressed by the following formula: g = [b[1:]; f[1:]], where g ∈ R (p +L)×d The output of the object recognition decoder can be the generated answer text, o, which can be represented by the formula o = F(g), where F represents the object recognition decoder. For example, during model inference, the object recognition decoder can integrate global image features, local image features, global text features, and local text features for decoding. The input g of the object recognition decoder can be represented by the following formula: g = [b; f], where g ∈ R. (p+l+2)×d .

[0072] In the above scheme, by integrating the first image (global image), the second image (region of interest image), and the first text (question text), the multimodal model can understand the question and image content from multiple perspectives. The image encoding module and the image convolution module are used to extract global and local image features, respectively, helping the multimodal model capture information from different levels of the image. Furthermore, the convolutional network structure has lower computational cost. The feature alignment module improves the consistency between global and local image features in the feature space, thereby enhancing the model's ability to identify key regions. The text encoding module extracts text features, enabling the model to deeply understand the textual content of the question. Through feature concatenation, the multimodal model can combine text features with aligned image features to form a rich multimodal feature representation. The decoding module accurately converts the multimodal features into the final textual answer. Through these steps, the multimodal model can more accurately identify key information in the image and, combined with the question text, provide a more accurate answer. Moreover, this model structure can process and understand multimodal data containing rich visual and textual information, thus supporting answers to complex questions and improving the applicability of the scheme.

[0073] In one embodiment, before inputting multimodal information into the multimodal model in step S121, the object recognition method of this application embodiment further includes the following steps: Step S101, obtaining a training dataset, wherein the training dataset includes text sample data, image sample data, and image-text pair sample data, and the image-text pair sample data includes global image samples, local image samples, and question text samples; Step S102, performing training operations on the multimodal model according to the training dataset and a preset loss function, wherein the preset loss function includes a first loss term, which is used to represent the consistency between the aligned image feature information inferred by the feature alignment module and the text feature information inferred by the text encoding module.

[0074] It can be understood that steps S101 and S102 above constitute the training process of the multimodal model. For example, for text sample data, the visual feature input can be considered empty; for image sample data, the text input can be considered empty. For image-text pair sample data, the image features and text features can be concatenated and then input into the model for training.

[0075] In the above scheme, by using a training dataset containing text sample data, image sample data, and image-text pair sample data, the multimodal model can quickly learn rich multimodal features and the relationships between them. The introduction of image-text pair sample data enables the model to learn the correspondence between image content and related text. The inclusion of a first loss term in the preset loss function can be used to optimize the feature alignment module, improving the consistency between image and text features, thereby enhancing the model's ability to fuse image and text information. By continuously optimizing feature alignment during training, the model can more accurately understand and answer image-related questions. Therefore, this scheme, by strengthening feature alignment and multimodal information fusion during the training phase, enables the model to exhibit higher accuracy and robustness in subsequent multimodal tasks, improving the precision of object recognition.

[0076] In one implementation, the preset loss function includes a first loss function and a second loss function. Step S102 performs a training operation on the multimodal model based on the training dataset and the preset loss function, including the following steps: Step S1021, in the pre-training phase of the training operation, the multimodal model is trained using text sample data, image sample data, image-text pair sample data, and the first loss function, wherein the first loss function includes a first loss term and a second loss term, and the second loss term is used to represent the difference between the inference result of the multimodal model on the sample data and the true label corresponding to the sample data; Step S1022, in the model fine-tuning phase of the training operation, the multimodal model is trained using image-text pair sample data and the second loss function, wherein the second loss function includes a second loss term.

[0077] In one implementation, the first loss term can be expressed using the following formula:

[0078]

[0079] Where loss1 represents the first loss term, and x represents the global image sample. Let t represent a local image sample, b[0] represent the problem text sample, b[0] represent the aligned image feature information inferred by the feature alignment module, and f[0] represent the text feature information inferred by the text encoder. Combining the aforementioned formula, b[0] and f[0] can represent the first set of features of b and the feature representation f of the user prompt text, respectively, namely the global image features and the global text features.

[0080] In one implementation, the second loss term can be expressed using the following formula:

[0081]

[0082] Where o represents the input image and text sample pair x, In case t, the generated information (i.e., the inference result) of the multimodal model can be understood as loss2 being the model-generated information o and the image-text sample pair x. The cross-entropy loss function between the true labels G corresponding to t.

[0083] In a specific example, the first loss function is, for example, loss... a In other words, loss a =loss1 + loss2; the second loss function is, for example, loss... b In other words, loss b =loss2. During the pre-training phase, a massive training dataset and the first loss function can be used for training. Figure 3a The multimodal model in [the context of the model]. This is achieved by minimizing the loss function [the model's] loss function]. a The model is trained using a gradient backpropagation method, training the entire network. This not only trains the model's overall recognition ability but also ensures consistency between text and images. During fine-tuning, the model's input can be limited to image-text pair samples, and categories can be directly generated. Therefore, only loss1 needs to be used. Similarly, the loss function can be minimized. b The training is performed in a manner that uses gradient backpropagation to train the entire network.

[0084] In the above scheme, the model can learn features and adjust parameters more effectively through two stages: pre-training and fine-tuning. During pre-training, the model simultaneously learns to process text, images, and image-text pairs, which helps it learn richer feature representations and improves its performance on multimodal tasks. Since the first loss term focuses on feature alignment, ensuring consistency between image and text features, while the second loss term focuses on the difference between the model's inference results and the true labels, training the multimodal model through the first loss function helps it quickly reach optimal performance in different aspects. This allows the model to learn how to accurately predict the labels of sample data during the pre-training stage, thereby improving the model's accuracy in object recognition. By using diverse sample data during pre-training, the model can learn a wider range of feature representations, which helps improve its generalization ability on unseen data. By learning general features during pre-training, the model can adapt to specific tasks more quickly during the fine-tuning stage, thus improving overall training efficiency. This staged training method makes the model easier to extend to new modalities or tasks. In summary, the training scheme for the aforementioned multimodal model, by introducing a finely designed loss function and a phased training strategy during the training process, enables the model to achieve higher accuracy and robustness in multimodal recognition tasks, thereby further improving the accuracy and efficiency of object recognition.

[0085] This application also provides an object recognition device. For example... Figure 4 As shown, the object recognition device 400 includes: an acquisition module 410, used to acquire a first image, region interest information corresponding to the first image, and question information corresponding to the first image, wherein the first image is a global image of the object to be recognized, the region interest information includes the location information of the region of interest in the first image specified by the user, and the question information is information about a question raised by the user about the object to be recognized; and a recognition module 420, used to recognize the object to be recognized based on the first image, the region interest information, and the question information, and determine the answer to the question.

[0086] In one embodiment, the recognition module includes: a model recognition submodule, used to input multimodal information into a multimodal model to obtain an answer, wherein the multimodal information is determined based on a first image, region interest information, and question information, and the multimodal model includes a feature alignment module, which is used to align the global image feature information of the first image with the local image feature information of the region of interest.

[0087] In one implementation, the multimodal information includes a first image, a second image, and first text. The second image is an image of the region of interest, and the first text is the text of the question. The multimodal model further includes an image encoding module, an image convolution module, a text encoding module, and a decoding module. The model recognition submodule includes: a text encoding unit, used to input the first text into the text encoding module to obtain the text feature information of the first text; a feature alignment unit, used to perform feature alignment on global image feature information and local image feature information using the feature alignment module to obtain aligned image feature information, wherein the global image feature information is obtained by encoding the first image using the image encoding module, and the local image feature information is obtained by convolving the second image using the image convolution module; a feature stitching unit, used to stitch the text feature information and the aligned image feature information to obtain stitched text image feature information; and a decoding unit, used to input the text image feature information into the decoding module to obtain the second text about the answer.

[0088] In one embodiment, the object recognition device further includes: a dataset acquisition module for acquiring a training dataset, wherein the training dataset includes text sample data, image sample data, and image-text pair sample data, and the image-text pair sample data includes global image samples, local image samples, and question text samples; and a training module for performing training operations on a multimodal model based on the training dataset and a preset loss function, wherein the preset loss function includes a first loss term, which is used to represent the consistency between the aligned image feature information inferred by the feature alignment module and the text feature information inferred by the text encoding module.

[0089] In one embodiment, the preset loss function includes a first loss function and a second loss function. The training module includes: a pre-training unit, used to train a multimodal model using text sample data, image sample data, image-text pair sample data, and the first loss function during the pre-training phase of the training operation, wherein the first loss function includes a first loss term and a second loss term, the second loss term being used to represent the difference between the inference result of the multimodal model on the sample data and the true label corresponding to the sample data; and a model fine-tuning unit, used to train the multimodal model using image-text pair sample data and the second loss function during the model fine-tuning phase of the training operation, wherein the second loss function includes a second loss term.

[0090] In one implementation, the first loss term is expressed using the following formula:

[0091]

[0092] Where loss1 represents the first loss term, and x represents the global image sample. t represents a local image sample, b[0] represents the aligned image feature information inferred by the feature alignment module, and f[0] represents the text feature information inferred by the text encoder.

[0093] In one embodiment, the feature alignment module includes a first fully connected network layer and a second fully connected network layer. The feature alignment unit includes: a superposition subunit for superimposing global image feature information with local image feature information to obtain superimposed image feature information; a transformation subunit for inputting the superimposed image feature information into the first fully connected network layer to obtain transformed image feature information, wherein the number of feature dimensions of the transformed image feature information is greater than the number of feature dimensions of the superimposed image feature information; and an alignment subunit for inputting the transformed image feature information into the second fully connected network layer to obtain aligned image feature information, wherein the number of aligned image feature information is equal to the number of feature dimensions of the superimposed image feature information.

[0094] like Figure 5 As shown, this application embodiment also provides a terminal device 500, including: at least one processor 510 ( Figure 5 The diagram shows only one processor, memory 520, and computer program 530 stored in memory 520 and executable on at least one processor 510. When processor 510 executes computer program 530, it implements the steps of the object recognition method described above.

[0095] In this embodiment, the terminal device may include, but is not limited to, a processor and a memory. Figure 5 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than illustrated, or combine certain components, or use different components. The processor may be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0096] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0097] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiments can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0098] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described object recognition method.

[0099] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps in the object recognition method described above.

[0100] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An object recognition method, characterized in that, include: Acquire a first image, corresponding region interest information and corresponding question information for the first image, wherein the first image is a global image of the object to be identified, the region interest information includes the location information of the region of interest in the first image specified by the user, and the question information is information about the question raised by the user regarding the object to be identified; The object to be identified is determined based on the first image, the region attention information, and the question information, and the answer to the question is determined.

2. The object recognition method as described in claim 1, characterized in that, The step of identifying the object to be identified based on the first image, the region attention information, and the question information, and determining the answer to the question, includes: The multimodal information is input into the multimodal model to obtain the answer. The multimodal information is determined based on the first image, the region of interest information, and the question information. The multimodal model includes a feature alignment module, which is used to align the global image feature information of the first image with the local image feature information of the region of interest.

3. The object recognition method as described in claim 2, characterized in that, The multimodal information includes the first image, the second image, and the first text, wherein the second image is the image of the region of interest, and the first text is the text of the question. The multimodal model further includes an image encoding module, an image convolution module, a text encoding module, and a decoding module. Inputting the multimodal information into the multimodal model includes: The first text is input into the text encoding module to obtain the text feature information of the first text; The feature alignment module is used to align the global image feature information and the local image feature information to obtain aligned image feature information. The global image feature information is obtained by encoding the first image using the image encoding module, and the local image feature information is obtained by convolving the second image using the image convolution module. The text feature information and the aligned image feature information are concatenated to obtain the concatenated text image feature information; The text image feature information is input into the decoding module to obtain the second text about the answer.

4. The object recognition method as described in claim 3, characterized in that, Before inputting the multimodal information into the multimodal model, the method further includes: Obtain a training dataset, wherein the training dataset includes text sample data, image sample data, and image-text pair sample data, and the image-text pair sample data includes global image samples, local image samples, and question text samples; The training operation of the multimodal model is performed based on the training dataset and the preset loss function, wherein the preset loss function includes a first loss term, which is used to represent the consistency between the aligned image feature information inferred by the feature alignment module and the text feature information inferred by the text encoding module.

5. The object recognition method as described in claim 4, characterized in that, The preset loss function includes a first loss function and a second loss function. The step of training the multimodal model based on the training dataset and the preset loss function includes: In the pre-training phase of the training operation, the multimodal model is trained using the text sample data, the image sample data, the image-text pair sample data, and the first loss function. The first loss function includes a first loss term and a second loss term, wherein the second loss term is used to represent the difference between the inference result of the multimodal model on the sample data and the true label corresponding to the sample data. During the model fine-tuning phase of the training operation, the multimodal model is trained using the image and text sample data and the second loss function, wherein the second loss function includes the second loss term.

6. The object recognition method as described in claim 5, characterized in that, The first loss term is expressed using the following formula: Where loss1 represents the first loss term, and x represents the global image sample. Let t represent the local image sample, t represent the question text sample, b[0] represent the aligned image feature information inferred by the feature alignment module, and f[0] represent the text feature information inferred by the text encoder.

7. The object recognition method according to any one of claims 3-6, characterized in that, The feature alignment module includes a first fully connected network layer and a second fully connected network layer. The step of using the feature alignment module to perform feature alignment on the global image feature information and the local image feature information includes: The global image feature information is superimposed with the local image feature information to obtain superimposed image feature information; The superimposed image feature information is input into the first fully connected network layer to obtain transformed image feature information, wherein the number of feature dimensions of the transformed image feature information is greater than the number of feature dimensions of the superimposed image feature information. The transformed image feature information is input into the second fully connected network layer to obtain the aligned image feature information, wherein the number of aligned image feature information is equal to the number of feature dimensions of the superimposed image feature information.

8. An object recognition device, characterized in that, include: The acquisition module is used to acquire a first image, region interest information corresponding to the first image, and question information corresponding to the first image. The first image is a global image of the object to be identified. The region interest information includes the location information of the region of interest in the first image specified by the user. The question information is information about the question raised by the user regarding the object to be identified. The recognition module is used to identify the object to be identified based on the first image, the region attention information, and the question information, and to determine the answer to the question.

9. A terminal device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the object recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the object recognition method as described in any one of claims 1 to 7.

Images