Multimodal scene recognition method and device, computer device and storage medium

Abstract

The application relates to a multi-modal scene recognition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: determining a pre-trained first multi-modal scene recognition model; the first multi-modal scene recognition model comprises a first image coding network, and the second multi-modal scene recognition model comprises a second image coding network; inputting sample images into the first image coding network and the second image coding network respectively for coding processing; inputting the corresponding coding processing results into a pre-trained first auxiliary branch and a trained second auxiliary branch respectively for image recognition, so as to obtain a first image recognition result and a second image recognition result; adjusting the first image coding network based on the difference between the first image recognition result and the second image recognition result, so as to obtain a trained first multi-modal scene recognition model; and performing scene recognition based on the trained first multi-modal scene recognition model. The method can improve the accuracy of scene recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a multi-modal scene recognition method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] With the development of computer and Internet technology, the application of scene recognition is more and more extensive. For example, the scene recognition can be performed on the photographed image or video, and the scene recognition can be used to add the text matched with the scene to the image or video.

[0003] In the traditional technology, the scene recognition can be performed by using a multi-modal representation model. The multi-modal representation model refers to a technology that machines extract information from multiple field data such as images, texts, videos and voices, realize the conversion and fusion of the information, and then improve the performance of the model. Since the structure of the multi-modal representation model is usually complex, the complexity of the model can be reduced by using knowledge distillation.

[0004] However, the distillation scheme of the multi-modal representation model mainly uses a teacher network to train a student network, mutual distillation between multiple student networks, self-distillation of a single student network, and the like. The model obtained by using the traditional distillation scheme needs to be improved in terms of accuracy. SUMMARY

[0005] Therefore, it is necessary to provide a multi-modal scene recognition method, device, computer equipment, computer readable storage medium and computer program product capable of improving the recognition accuracy in view of the above technical problems.

[0006] In a first aspect, the present application provides a multi-modal scene recognition method. The method comprises: determining a pre-trained first multi-modal scene recognition model; the pre-trained first multi-modal scene recognition model comprises a first image encoding network and a first text encoding network, the pre-trained first multi-modal scene recognition model is trained based on a trained second multi-modal scene recognition model, the second multi-modal scene recognition model comprises a second image encoding network and a second text encoding network, and the network complexity of the first image encoding network is less than that of the second image encoding network; inputting a sample image into the first image encoding network for encoding processing, inputting the result of the encoding processing into a pre-trained first auxiliary branch for image recognition, and obtaining a first image recognition result; inputting the sample image into the second image encoding network for encoding processing, inputting the result of the encoding processing into a trained second auxiliary branch for image recognition, and obtaining a second image recognition result; adjusting the first image encoding network based on the difference between the first image recognition result and the second image recognition result, and obtaining a trained first multi-modal scene recognition model; and performing scene recognition based on the trained first multi-modal scene recognition model.

[0007] In a second aspect, the present application also provides a multi-modal scene recognition device. The device comprises: a model determination module configured to determine a pre-trained first multi-modal scene recognition model; the pre-trained first multi-modal scene recognition model comprises a first image encoding network and a first text encoding network, the pre-trained first multi-modal scene recognition model is trained based on a trained second multi-modal scene recognition model, the second multi-modal scene recognition model comprises a second image encoding network and a second text encoding network, and the network complexity of the first image encoding network is less than that of the second image encoding network; a first image recognition module configured to input a sample image into the first image encoding network for encoding processing, input the result of the encoding processing into a pre-trained first auxiliary branch for image recognition, and obtain a first image recognition result; a second image recognition module configured to input the sample image into the second image encoding network for encoding processing, input the result of the encoding processing into a trained second auxiliary branch for image recognition, and obtain a second image recognition result; a model adjustment module configured to adjust the first image encoding network based on the difference between the first image recognition result and the second image recognition result, and obtain a trained first multi-modal scene recognition model; and a scene recognition module configured to perform scene recognition based on the trained first multi-modal scene recognition model.

[0008] In some embodiments, the model adjustment module is further configured to: obtain a first loss value based on a difference between the first image recognition result and the second image recognition result; input a sample image into a first image encoding network in the pre-trained first multi-modal scene recognition model for encoding processing to obtain a first encoding feature; input the sample image into a second image encoding network in the trained second multi-modal scene recognition model for encoding processing to obtain a second encoding feature; obtain a second loss value based on a feature difference value between the first encoding feature and the second encoding feature; and adjust the first image encoding network in the pre-trained first multi-modal scene recognition model based on the first loss value and the second loss value to obtain the trained first multi-modal scene recognition model.

[0009] In some embodiments, the multi-modal scene recognition apparatus further comprises a first training module configured to: input a sample image into a second image encoding network in the trained second multi-modal scene recognition model for encoding processing, and input a result of the encoding processing into a to-be-trained second auxiliary branch for image recognition to obtain a third image recognition result of the sample image; and adjust the to-be-trained second auxiliary branch based on a difference between the third image recognition result and a standard image recognition result of the sample image to obtain the trained second auxiliary branch.

[0010] In some embodiments, the first training module is further configured to: adjust the to-be-trained second auxiliary branch based on a difference between the third image recognition result and the standard image recognition result to obtain a preliminarily trained second auxiliary branch; input a sample image into a second image encoding network in the trained second multi-modal scene recognition model for encoding processing, and input a result of the encoding processing into the to-be-trained second auxiliary branch for image recognition to obtain a fourth image recognition result of the sample image; and adjust the preliminarily trained second auxiliary branch based on a difference between the fourth image recognition result and a standard image recognition result of the sample image to obtain the trained second auxiliary branch.

[0011] In some embodiments, the multi-modal scene recognition apparatus further comprises a second training module configured to: input a sample image and a sample text into the trained second multi-modal scene recognition model for similarity calculation to generate a first similarity; the first similarity representing a similarity between the sample image and the sample text; input the sample image and the sample text into a to-be-trained first multi-modal scene recognition model for similarity calculation to generate a second similarity; and adjust parameters of the to-be-trained first multi-modal scene recognition model based on a difference between the first similarity and the second similarity to obtain a pre-trained first multi-modal scene recognition model.

[0012] In some embodiments, the scene recognition module is further configured to: input a target scene image into a first image encoding network of the trained first multi-modal scene recognition model to obtain a target image feature; input a candidate scene text into a first text encoding network of the trained first multi-modal scene recognition model to obtain a candidate text feature; and determine the candidate scene text as a target scene text matched with the target scene image if a similarity between the target image feature and the candidate text feature is greater than a similarity threshold.

[0013] In a third aspect, the present application provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the multi-modal scene recognition method when executing the computer program.

[0014] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the steps of the multi-modal scene recognition method when executed by a processor.

[0015] In a fifth aspect, the present application provides a computer program product. The computer program product includes a computer program, and the computer program implements the steps of the multi-modal scene recognition method when executed by a processor.

[0016] The multi-modal scene recognition method, device, computer device, storage medium and computer program product, the first multi-modal scene recognition model includes a first image encoding network, the second multi-modal scene recognition model includes a second image encoding network, and the network complexity of the first image encoding network is less than that of the second image encoding network. The trained second multi-modal scene recognition model is trained by a large amount of data, can calculate the similarity between different modal data to realize scene recognition and has high recognition accuracy. By adjusting the first image encoding network based on the difference between the first image recognition result and the second image recognition result, the image expression ability of the first image encoding network is strengthened, so that the trained first similarity model has higher recognition accuracy, that is, the accuracy of the first similarity recognition model is improved, thereby improving the accuracy of scene recognition. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An application environment of a multi-modal scene recognition method provided by an embodiment of the present application;

[0018] Figure 2 A flowchart of a multi-modal scene recognition method provided by an embodiment of the present application;

[0019] Figure 3A This is a schematic diagram of the structure of the first multimodal scene recognition model provided in an embodiment of this application;

[0020] Figure 3B This is a schematic diagram of the structure of the second multimodal scene recognition model provided in the embodiments of this application;

[0021] Figure 3C A schematic diagram of the structure of the auxiliary model provided in the embodiments of this application;

[0022] Figure 4 A schematic flowchart of a model distillation method provided in this application embodiment;

[0023] Figure 5 This is a structural block diagram of a multimodal scene recognition device provided in an embodiment of this application;

[0024] Figure 6 This is an internal structural diagram of a computer device according to an embodiment of this application;

[0025] Figure 7 This is an internal structural diagram of another computer device in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] The multimodal scene recognition method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0028] Specifically, the server 104 determines a pre-trained first multi-modal scene recognition model; the pre-trained first multi-modal scene recognition model includes a first image encoding network and a first text encoding network, the pre-trained first multi-modal scene recognition model is trained based on a trained second multi-modal scene recognition model, the second multi-modal scene recognition model includes a second image encoding network and a second text encoding network, and the network complexity of the first image encoding network is less than that of the second image encoding network; then the sample image is input into the first image encoding network for encoding processing, the result of the encoding processing is input into the pre-trained first auxiliary branch for image recognition, and a first image recognition result is obtained; the sample image is input into the second image encoding network for encoding processing, the result of the encoding processing is input into the trained second auxiliary branch for image recognition, and a second image recognition result is obtained; based on the difference between the first image recognition result and the second image recognition result, the first image encoding network is adjusted to obtain the trained first multi-modal scene recognition model; the server 104 receives the scene image sent by the terminal 102, performs scene recognition based on the trained first multi-modal scene recognition model, obtains a scene recognition result, and the server 104 can send the scene recognition result to the terminal 102. The terminal 102 can store the scene recognition result sent by the server 104.

[0029] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0030] In some embodiments, as shown in Figure 2 , a multi-modal scene recognition method is provided. The method is applied to the server 104 in Figure 1 for example, and includes the following steps:

[0031] Step 202, determining a pre-trained first multi-modal scene recognition model; the pre-trained first multi-modal scene recognition model includes a first image encoding network and a first text encoding network, the pre-trained first multi-modal scene recognition model is trained based on a trained second multi-modal scene recognition model, the second multi-modal scene recognition model includes a second image encoding network and a second text encoding network, and the network complexity of the first image encoding network is less than that of the second image encoding network.

[0032] The first multi-modal scene recognition model and the second multi-modal scene recognition model are both multi-modal representation models, which can process multi-modal data. The multi-modal data is data including at least two modalities, and the modalities of the data include at least one of an image, text, a video, or speech. For example, data including two different modalities of an image and text is multi-modal data, and image data is single-modal data. The first multi-modal scene recognition model includes a first image encoding network and a first text encoding network. As shown in FIG. 1, a structural diagram of the first multi-modal scene recognition model is shown. The input is an image and text. The first image encoding network is used to encode and process the image to obtain image encoding features. The first text encoding network is used to encode and process the text to obtain text encoding features. Then, the similarity between the image encoding features and the text encoding features is calculated, and the similarity between the image and the text is output. Figure 3A

[0033] The pre-trained first multi-modal scene recognition model is trained based on the trained second multi-modal scene recognition model and has the function of scene recognition. However, the number of samples used for pre-training is small, and therefore the recognition accuracy is low. The second multi-modal scene recognition model includes a second image encoding network and a second text encoding network. The second multi-modal scene recognition model has the same input and output as the first multi-modal scene recognition model, that is, the functions of the two models are the same, but the model structures and the number of parameters are different. The network complexity of the first image encoding network is less than that of the second image encoding network, and therefore the network complexity of the first multi-modal scene recognition model is less than that of the second similarity generation network. The first text encoding network is determined based on the second text encoding network. The first text encoding network and the second text encoding network can be the same or different.

[0034] Specifically, the server trains the first similarity model to be trained by using sample images and sample texts to obtain the pre-trained first multi-modal scene recognition model. The sample images and the sample texts are used for model training and can be images and description texts of different scenes. The sample images and the sample texts used for training different networks can be the same or different.

[0035] In some embodiments, the server can input the sample images and the sample texts into the first multi-modal scene recognition model to be trained to perform similarity calculation and generate a first similarity. The sample images and the sample texts are input into the second multi-modal scene recognition model to perform similarity calculation and generate a second similarity. Then, the difference between the first similarity and the second similarity is calculated, and the parameters of the first multi-modal scene recognition model are adjusted by using the difference to obtain the pre-trained first multi-modal scene recognition model.

[0036] ​In some embodiments, the trained second multi-modal scene recognition model is obtained by training a to-be-trained second multi-modal scene recognition model using sample images and sample texts in a super large data set. Since the data set used for training has a very large amount of data, the training time is relatively long, and the network complexity is relatively high, the trained second multi-modal scene recognition model has relatively high recognition accuracy. However, due to the high network complexity of the second multi-modal scene recognition model, it is difficult to deploy the second multi-modal scene recognition model to a mobile terminal. Therefore, the trained second multi-modal scene recognition model can be used as a teacher model, the pre-trained first multi-modal scene recognition model can be used as a student model, and then the teacher model can be used to perform knowledge distillation on the student model, so as to improve the recognition accuracy of the first multi-modal scene recognition model.

[0037] In step 204, the sample image is input into the first image encoding network for encoding processing, and the result of the encoding processing is input into the pre-trained first auxiliary branch for image recognition to obtain a first image recognition result.

[0038] The first auxiliary branch is used to recognize scene features in the image. For example, if the image is a beach image including a sky, sea waves, a beach, and shells, the image recognition result is "sky", "sea waves", "beach", and "shells". The first auxiliary branch can assist in distillation training of the model. The first image recognition result is obtained by image recognition of the sample image by the pre-trained first auxiliary branch, and the pre-trained first auxiliary branch is trained based on the trained second auxiliary branch.

[0039] Specifically, the server obtains a sample image, as shown in FIG. 4, inputs the sample image into the first image encoding network for encoding processing, and then inputs the result of the encoding processing into the first auxiliary branch, i.e., the auxiliary branch, for image recognition to obtain a first image recognition result. Figure 3A

[0040] In some embodiments, the output layer corresponding to the first image encoding network is a full connection layer, and the first auxiliary branch is connected to the first image encoding network in a full connection operator manner. Therefore, the branch in the first multi-modal scene recognition model for processing the image includes two output branches. The output layer corresponding to the first image encoding network outputs image encoding features, and the first auxiliary branch outputs an image recognition result.

[0041] ​In some embodiments, the server can utilize the trained second auxiliary branch to perform distillation training on the to-be-trained second auxiliary branch to obtain a pre-trained first auxiliary branch having a function of identifying scene features in an image. The server inputs a sample image into a first image encoding network in the pre-trained first multi-modal scene identification model for encoding processing, inputs a result of the encoding processing into the to-be-trained first auxiliary branch for image identification to obtain a sample identification result, and inputs the sample image into the trained second image encoding network for encoding processing, inputs a result of the encoding processing into the trained second auxiliary branch for image identification to obtain a target identification result, and then adjusts parameters of the to-be-trained first auxiliary branch based on a difference between the sample identification result and the target identification result until the network converges to obtain the pre-trained first auxiliary branch.

[0042] In step 206, the sample image is input into the second image encoding network for encoding processing, and a result of the encoding processing is input into the trained second auxiliary branch for image identification to obtain a second image identification result.

[0043] The second auxiliary branch is configured to identify scene features in an image.

[0044] Specifically, the server inputs a sample image into a second image encoding network in the trained second multi-modal scene identification model for encoding processing, and inputs a result of the encoding processing into the trained second auxiliary branch for image identification to obtain a second image identification result.

[0045] In some embodiments, a model combining the second auxiliary branch and the second image encoding network can be regarded as an auxiliary model corresponding to the second multi-modal scene identification model. The auxiliary model is obtained on the basis of the second multi-modal scene identification model. As shown in Figure 3B FIG. 2 shows a structural schematic diagram of the second multi-modal scene identification model, Figure 3C FIG. 3 shows a structural schematic diagram of the auxiliary model. The second auxiliary branch can be connected after the second image encoding network in the second multi-modal scene identification model in a full-connection operator manner, and an output layer corresponding to the second image encoding network and a part after the output layer in the second multi-modal scene identification model are discarded to obtain the auxiliary model as shown in Figure 3C

[0046] In step 208, the first image encoding network is adjusted based on a difference between the first image identification result and the second image identification result to obtain the trained first multi-modal scene identification model.

[0047] ​Specifically, the server can adjust the parameters of the first auxiliary branch and the parameters of the first image encoding network in the first multi-modal scene recognition model based on the difference between the first image recognition result and the second image recognition result, to obtain the trained first multi-modal scene recognition model.

[0048] In some embodiments, the pre-trained first multi-modal scene recognition model is a pre-trained student model, the pre-trained first auxiliary branch is an auxiliary branch of the student model, and the first image encoding network is an original branch of the student model. The trained second multi-modal scene recognition model is a teacher model, and the trained second auxiliary branch and the second image encoding network constitute an auxiliary model. The auxiliary model and the teacher model are two independent models. The server inputs the sample image into the pre-trained student model, the trained teacher model, and the trained auxiliary model respectively. The first image encoding network and the corresponding output layer in the student model output the first encoding feature, and the first image recognition result is output by the auxiliary branch. The second image encoding network and the corresponding output layer in the teacher network output the second encoding feature, and the second image recognition result is output by the auxiliary model. The server obtains the first loss value based on the difference between the first image recognition result and the second image recognition result, and obtains the second loss value based on the feature difference value between the first encoding feature and the second encoding feature. Then the server performs weighted calculation on the first loss value and the second loss value to obtain a weighted loss value, and adjusts the first image encoding network in the student network and the auxiliary branch by using the weighted loss value. Since the teacher model and the auxiliary model are used, the parameters of the two output branches of the student model are iteratively updated, so this distillation process is called mixed distillation training of the student model.

[0049] In some embodiments, as shown in Figure 4 The server trains the auxiliary model by using the sample image, to obtain the trained auxiliary model. Then the server takes the pre-trained first multi-modal scene recognition model as a student model, accesses the first auxiliary branch after the first image encoding network in the student model, constructs the auxiliary branch, distills and trains the auxiliary branch of the student model by using the trained auxiliary model, to obtain the pre-trained auxiliary branch. Then the server performs mixed distillation training on the original branch and the auxiliary branch of the student model by using the teacher model and the auxiliary model, to obtain the trained student model and the trained auxiliary branch. The auxiliary branch of the student model is removed, to obtain the trained first multi-modal scene recognition model.

[0050] In step 210, scene recognition is performed based on the trained first multi-modal scene recognition model.

[0051] The scene is a landmark place, including a beach, a flower shop, a station, a park, a school, etc., and the scene recognition can be based on scene features in a scene image to determine corresponding scene text. The scene image is an image corresponding to the scene, for example, when the scene is a station, the scene image can be a photo taken at the station. The scene text can be a scene name or a scene description text.

[0052] Specifically, the server determines a target scene image and a candidate scene text, inputs the target scene image and the candidate scene text into the trained first multi-modal scene recognition model, obtains a similarity between the target scene image and the candidate scene text, and then determines a target scene text based on the similarity, thereby realizing scene recognition. For example, the similarity between the target scene image and multiple candidate scene texts can be calculated respectively, and the candidate scene text corresponding to the maximum similarity can be taken as the target scene text. Alternatively, the similarity can be compared with a similarity threshold, and the candidate scene text can be taken as the target scene text when the similarity is greater than the similarity threshold. The target scene image is a scene image to be recognized, the candidate scene text can be determined from a scene text library, the scene text library is pre-set and includes multiple scene texts, and the target scene text is a scene text matched with the target scene image.

[0053] In the above multi-modal scene recognition method, the first multi-modal scene recognition model includes a first image encoding network, the second multi-modal scene recognition model includes a second image encoding network, the network complexity of the first image encoding network is less than that of the second image encoding network, the trained second multi-modal scene recognition model is trained by a large amount of data and can calculate the similarity between different modal data to realize scene recognition and has high recognition accuracy, the first image encoding network is adjusted based on the difference between the first image recognition result and the second image recognition result, and the image expression ability of the first image encoding network is strengthened, so that the trained first similarity model has higher recognition accuracy, that is, the recognition accuracy of the first similarity recognition model is improved, thereby improving the accuracy of scene recognition.

[0054] In some embodiments, step 208 further includes obtaining a first loss value based on the difference between the first image recognition result and the second image recognition result, inputting the sample image into the first image encoding network in the pre-trained first multi-modal scene recognition model for encoding processing to obtain first encoding features, inputting the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing to obtain second encoding features, obtaining a second loss value based on a feature difference value between the first encoding features and the second encoding features, and adjusting the first image encoding network in the pre-trained first multi-modal scene recognition model based on the first loss value and the second loss value to obtain the trained first multi-modal scene recognition model.

[0055] The first loss value is obtained based on a difference between the first image recognition result and the second image recognition result, the first encoding feature is obtained by encoding the sample image by using the first image encoding network, and the second encoding feature is obtained by encoding the sample image by using the second image encoding network.

[0056] Specifically, the server obtains the first loss value based on the difference between the first image recognition result and the second image recognition result, for example, the first loss value can be calculated by using a BCE loss (Binary Cross Entropy Loss) loss function; and obtains the second loss value based on a feature difference value between the first encoding feature and the second encoding feature, for example, the second loss value can be calculated by using a MSE loss (Mean Square Error) loss function; then the first loss value and the second loss value are weighted and calculated to obtain a weighted loss value, and the first image encoding network in the pre-trained first multi-modal scene recognition model is adjusted by using the weighted loss value until the model converges, the first auxiliary branch output branch connected to the first multi-modal scene recognition model is removed, the structure and parameters of the first image encoding network after training are retained, and a trained first multi-modal scene recognition model is obtained.

[0057] In the embodiment, the first loss value is obtained based on the first image recognition result and the second image recognition result, the second loss value is obtained based on the first encoding feature and the second encoding feature, the first image encoding network in the pre-trained first multi-modal scene recognition model is adjusted based on the first loss value and the second loss value, the expression ability of the first image encoding network for images is improved, so that the recognition accuracy of the trained first multi-modal scene recognition model is higher, and the accuracy of scene recognition is improved.

[0058] In some embodiments, the step of determining the trained second auxiliary branch includes: inputting the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing, inputting the result of the encoding processing into the to-be-trained second auxiliary branch for image recognition, and obtaining a third image recognition result of the sample image; and adjusting the to-be-trained second auxiliary branch based on a difference between the third image recognition result and a standard image recognition result of the sample image, to obtain the trained second auxiliary branch.

[0059] The third image recognition result is obtained by image recognition of the second auxiliary branch to be trained on the sample image. The standard recognition result is obtained by manual labeling of the sample image, also known as the label of the sample image. The standard recognition result of one sample image can be one or multiple. For example, the sample image includes sky, sea wave, beach and shell, and the standard recognition result of the sample image is "sky", "sea wave", "beach" and "shell". That is, the sample image can also be referred to as multi-label single-modal data.

[0060] Specifically, the server first trains the second auxiliary branch to be trained of the second multi-modal scene recognition model, that is, trains the auxiliary model. The server can input the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing, input the result of the encoding processing into the second auxiliary branch to be trained for image recognition, and obtain the third image recognition result of the sample image. Based on the difference between the third image recognition result and the standard image recognition result of the sample image, the second auxiliary branch to be trained is adjusted to obtain the trained second auxiliary branch.

[0061] In this embodiment, the second auxiliary branch to be trained is adjusted based on the difference between the third image recognition result and the standard image recognition result of the sample image to obtain the trained second auxiliary branch, so that the first auxiliary branch can be distilled and trained by using the trained second auxiliary branch, and the recognition accuracy of the first similarity generation network is further improved.

[0062] In some embodiments, adjusting the second auxiliary branch to be trained based on the difference between the third image recognition result and the standard image recognition result of the sample image to obtain the trained second auxiliary branch includes: adjusting the second auxiliary branch to be trained based on the difference between the third image recognition result and the standard image recognition result to obtain a preliminary trained second auxiliary branch; inputting the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing, inputting the result of the encoding processing into the second auxiliary branch to be trained for image recognition, and obtaining a fourth image recognition result of the sample image; adjusting the preliminary trained second auxiliary branch based on the difference between the fourth image recognition result and the standard image recognition result of the sample image to obtain the trained second auxiliary branch.

[0063] The fourth image recognition result is obtained by image recognition of the preliminary trained second auxiliary branch on the sample image.

[0064] Specifically, the server first fixes the parameters of the second image encoding network in the second multi-modal scene recognition model, adjusts the second auxiliary branch to be trained based on the difference between the third image recognition result and the standard image recognition result, obtains the second auxiliary branch after preliminary training, and the parameters of the second image encoding network in the second multi-modal scene recognition model are unchanged in the training process; then the parameters of the second image encoding network in the second multi-modal scene recognition model are released to participate in the training with a preset learning rate, that is, the preliminary trained second auxiliary branch and the second image encoding network are adjusted based on the difference between the fourth image recognition result and the standard image recognition result of the sample image, and then the second auxiliary branch with the highest accuracy in the training process is taken as the trained second auxiliary branch.

[0065] In the embodiment, the second auxiliary branch to be trained is adjusted based on the difference between the third image recognition result and the standard image recognition result to obtain the second auxiliary branch after preliminary training, and the preliminary trained second auxiliary branch is adjusted based on the difference between the fourth image recognition result and the standard image recognition result of the sample image to obtain the trained second auxiliary branch, which further improves the recognition accuracy of the trained second auxiliary branch and makes the distillation training effect better.

[0066] In some embodiments, determining the pre-trained first multi-modal scene recognition model comprises: inputting the sample image and the sample text into the first multi-modal scene recognition model to be trained for similarity calculation to generate a first similarity; inputting the sample image and the sample text into the trained second multi-modal scene recognition model for similarity calculation to generate a second similarity; the second similarity represents the similarity between the sample image and the sample text; and adjusting the parameters of the first multi-modal scene recognition model to be trained based on the difference between the first similarity and the second similarity to obtain the pre-trained first multi-modal scene recognition model.

[0067] The first similarity is generated by the first multi-modal scene recognition model to be trained, and the second similarity is generated by the trained second multi-modal scene recognition model. The sample image and the sample text used for training different networks can be the same or different.

[0068] Specifically, the server adjusts the parameters of the first image encoding network and the parameters of the first text encoding network in the first multi-modal scene recognition model to be trained based on the difference between the first similarity and the second similarity until the model converges to obtain the pre-trained first multi-modal scene recognition model.

[0069] In this embodiment, the trained second multi-modal scene recognition model is used to train the first multi-modal scene recognition model to be trained to obtain a pre-trained first multi-modal scene recognition model, so that the pre-trained first multi-modal scene recognition model has the ability to calculate the similarity between an image and text.

[0070] In some embodiments, step 210 further includes: inputting the target scene image into the first image encoding network of the trained first multi-modal scene recognition model to obtain a target image feature; inputting the candidate scene text into the first text encoding network of the trained first multi-modal scene recognition model to obtain a candidate text feature; and in a case where the similarity between the target image feature and the candidate text feature is greater than a similarity threshold, determining the candidate scene text as the target scene text matched with the target scene image.

[0071] The target scene image is a scene image to be recognized, and the target scene text is a scene text matched with the target scene image. The similarity threshold is pre-set, for example, the similarity threshold can be set to 0.9.

[0072] Specifically, the server can obtain a target scene image to be recognized from a terminal, and determine a candidate scene text from a scene text library, then input the target scene image into the first image encoding network of the trained first multi-modal scene recognition model to perform encoding processing on the target scene image, and pass through the output layer corresponding to the first image encoding network to obtain a target image feature; input the candidate scene text into the first text encoding network of the trained first multi-modal scene recognition model to perform encoding processing to obtain a candidate text feature, then calculate the similarity between the target image feature and the candidate text feature, the similarity between the target image feature and the candidate text feature representing the matching degree between the target scene image and the candidate scene text, in a case where the similarity is greater than a similarity threshold, determine the candidate scene text as the target scene text, the target scene text being a scene name or a scene description text corresponding to a scene in the scene image; in a case where the similarity is less than the similarity threshold, return to the step of determining the candidate scene text from the scene text library until the similarity is greater than the similarity threshold.

[0073] In this embodiment, by obtaining the similarity between the target scene image and the candidate scene text based on the target image feature and the candidate text feature, and in a case where the similarity is greater than a similarity threshold, determining the candidate scene text as the target scene text, scene recognition is realized.

[0074] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0075] Based on the same inventive concept, the embodiments of the present application also provide a multi-modal scene recognition device for implementing the multi-modal scene recognition method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-modal scene recognition device embodiments provided below can refer to the limitations of the multi-modal scene recognition method described above, which will not be repeated here.

[0076] In some embodiments, as shown in Figure 5 A multi-modal scene recognition device is provided, comprising: a model determination module 502, a first image recognition module 504, a second image recognition module 506, a model adjustment module 508, and a scene recognition module 510, wherein:

[0077] The model determination module 502 is configured to determine a pre-trained first multi-modal scene recognition model. The pre-trained first multi-modal scene recognition model comprises a first image encoding network and a first text encoding network. The pre-trained first multi-modal scene recognition model is trained based on a trained second multi-modal scene recognition model. The second multi-modal scene recognition model comprises a second image encoding network and a second text encoding network. The network complexity of the first image encoding network is less than that of the second image encoding network.

[0078] The first image recognition module 504 is configured to input a sample image into the first image encoding network for encoding processing, input the result of the encoding processing into the pre-trained first auxiliary branch for image recognition, and obtain a first image recognition result.

[0079] The second image recognition module 506 is configured to input a sample image into the second image encoding network for encoding processing, input the result of the encoding processing into the trained second auxiliary branch for image recognition, and obtain a second image recognition result.

[0080] The model adjustment module 508 is configured to adjust the first image encoding network based on the difference between the first image recognition result and the second image recognition result, to obtain the trained first multi-modal scene recognition model.

[0081] The scene recognition module 510 is configured to perform scene recognition based on the trained first multi-modal scene recognition model.

[0082] In some embodiments, the model adjustment module 508 is further configured to: obtain a first loss value based on the difference between the first image recognition result and the second image recognition result; input the sample image into the first image encoding network in the pre-trained first multi-modal scene recognition model for encoding processing, to obtain a first encoding feature; input the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing, to obtain a second encoding feature; obtain a second loss value based on a feature difference value between the first encoding feature and the second encoding feature; and adjust the first image encoding network in the pre-trained first multi-modal scene recognition model based on the first loss value and the second loss value, to obtain the trained first multi-modal scene recognition model.

[0083] In some embodiments, the multi-modal scene recognition apparatus further includes a first training module. The first training module is configured to: input the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing, and input the result of the encoding processing into the to-be-trained second auxiliary branch for image recognition, to obtain a third image recognition result of the sample image; and adjust the to-be-trained second auxiliary branch based on the difference between the third image recognition result and a standard image recognition result of the sample image, to obtain the trained second auxiliary branch.

[0084] In some embodiments, the first training module is further configured to: adjust the to-be-trained second auxiliary branch based on the difference between the third image recognition result and the standard image recognition result, to obtain a preliminarily trained second auxiliary branch; input the sample image into the second image encoding network in the trained second multi-modal scene recognition model for encoding processing, and input the result of the encoding processing into the to-be-trained second auxiliary branch for image recognition, to obtain a fourth image recognition result of the sample image; and adjust the preliminarily trained second auxiliary branch based on the difference between the fourth image recognition result and the standard image recognition result of the sample image, to obtain the trained second auxiliary branch.

[0085] In some embodiments, the multimodal scene recognition device further includes a second training model. The second training module is configured to: input sample images and sample text into a trained second multimodal scene recognition model to calculate similarity and generate a first similarity; the first similarity characterizes the similarity between the sample images and the sample text; input sample images and sample text into a first multimodal scene recognition model to be trained to calculate similarity and generate a second similarity; and adjust the parameters of the first multimodal scene recognition model to be trained based on the difference between the first similarity and the second similarity to obtain a pre-trained first multimodal scene recognition model.

[0086] In some embodiments, the scene recognition module 510 is further configured to: input the target scene image into the first image encoding network of the trained first multimodal scene recognition model to obtain target image features; input the candidate scene text into the first text encoding network of the trained first multimodal scene recognition model to obtain candidate text features; and determine the candidate scene text as the target scene text that matches the target scene image if the similarity between the target image features and the candidate text features is greater than a similarity threshold.

[0087] Each module in the aforementioned multimodal scene recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0088] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data related to the multimodal scene recognition method. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal scene recognition method.

[0089] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a multimodal scene recognition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0090] Those skilled in the art will understand that Figure 6 and Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0091] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the multimodal scene recognition method described above.

[0092] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the multimodal scene recognition method described above.

[0093] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the multimodal scene recognition method described above.

[0094] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0095] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0096] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0097] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A multimodal scene recognition method, characterized in that, The method includes: A pre-trained first multimodal scene recognition model is determined; the pre-trained first multimodal scene recognition model includes a first image encoding network and a first text encoding network. The pre-trained first multimodal scene recognition model is trained based on a pre-trained second multimodal scene recognition model, which includes a second image encoding network and a second text encoding network. The network complexity of the first image encoding network is less than that of the second image encoding network. The first text encoding network is determined based on the second text encoding network. The sample image is input into the first image encoding network for encoding processing, and the result of the encoding processing is input into the pre-trained first auxiliary branch for image recognition to obtain the first image recognition result; wherein, the first auxiliary branch is used to identify scene features in the image; the first auxiliary branch assists in the distillation training of the model; The sample image is input into the second image coding network for encoding processing, and the result of the encoding processing is input into the trained second auxiliary branch for image recognition to obtain the second image recognition result; wherein, the pre-trained first auxiliary branch is trained based on the trained second auxiliary branch; Based on the difference between the first image recognition result and the second image recognition result, the first image coding network is adjusted to obtain the trained first multimodal scene recognition model; Scene recognition is performed based on the trained first multimodal scene recognition model.

2. The method according to claim 1, characterized in that, The step of adjusting the first image coding network based on the difference between the first image recognition result and the second image recognition result to obtain the trained first multimodal scene recognition model includes: Based on the difference between the first image recognition result and the second image recognition result, a first loss value is obtained; The sample image is input into the first image encoding network of the pre-trained first multimodal scene recognition model for encoding processing to obtain the first encoded feature; The sample image is input into the second image coding network of the trained second multimodal scene recognition model for coding processing to obtain the second coding feature; Based on the feature difference value between the first coding feature and the second coding feature, a second loss value is obtained; Based on the first loss value and the second loss value, the first image coding network in the pre-trained first multimodal scene recognition model is adjusted to obtain the trained first multimodal scene recognition model.

3. The method according to claim 1, characterized in that, The steps for determining the trained second auxiliary branch include: The sample image is input into the second image encoding network of the trained second multimodal scene recognition model for encoding processing. The result of the encoding processing is input into the second auxiliary branch to be trained for image recognition, and a third image recognition result of the sample image is obtained. Based on the difference between the third image recognition result and the standard image recognition result of the sample image, the second auxiliary branch to be trained is adjusted to obtain the trained second auxiliary branch.

4. The method according to claim 3, characterized in that, The step of adjusting the second auxiliary branch to be trained based on the difference between the third image recognition result and the standard image recognition result of the sample image to obtain the trained second auxiliary branch includes: Based on the difference between the third image recognition result and the standard image recognition result, the second auxiliary branch to be trained is adjusted to obtain a pre-trained second auxiliary branch; The sample image is input into the second image encoding network in the trained second multimodal scene recognition model for encoding processing. The result of the encoding processing is input into the second auxiliary branch to be trained for image recognition, and a fourth image recognition result of the sample image is obtained. Based on the difference between the fourth image recognition result and the standard image recognition result of the sample image, the second auxiliary branch of the initial training is adjusted to obtain the trained second auxiliary branch.

5. The method according to claim 1, characterized in that, The first pre-trained multimodal scene recognition model includes: The sample image and sample text are input into the first multimodal scene recognition model to be trained for similarity calculation, and the first similarity is generated. The sample image and sample text are input into the trained second multimodal scene recognition model to calculate similarity and generate a second similarity; the second similarity represents the similarity between the sample image and the sample text. Based on the difference between the first similarity and the second similarity, the parameters of the first multimodal scene recognition model to be trained are adjusted to obtain the pre-trained first multimodal scene recognition model.

6. The method according to claim 1, characterized in that, The scene recognition based on the trained first multimodal scene recognition model includes: The target scene image is input into the first image encoding network of the trained first multimodal scene recognition model to obtain the target image features; The candidate scene text is input into the first text encoding network of the trained first multimodal scene recognition model to obtain candidate text features; If the similarity between the target image features and the candidate text features is greater than a similarity threshold, the candidate scene text is determined as the target scene text that matches the target scene image.

7. A multimodal scene recognition device, characterized in that, The device includes: The model determination module is used to determine a pre-trained first multimodal scene recognition model. The pre-trained first multimodal scene recognition model includes a first image encoding network and a first text encoding network. This pre-trained first multimodal scene recognition model is trained based on a pre-trained second multimodal scene recognition model. The second multimodal scene recognition model includes a second image encoding network and a second text encoding network. The network complexity of the first image encoding network is less than that of the second image encoding network. The first text encoding network is determined based on the second text encoding network. The first image recognition module is used to input sample images into the first image encoding network for encoding processing, and input the encoding processing result into a pre-trained first auxiliary branch for image recognition to obtain a first image recognition result; wherein, the first auxiliary branch is used to identify scene features in the image; the first auxiliary branch assists in the distillation training of the model; The second image recognition module is used to input sample images into the second image coding network for encoding processing, and input the encoding processing result into the trained second auxiliary branch for image recognition to obtain the second image recognition result; wherein, the pre-trained first auxiliary branch is trained based on the trained second auxiliary branch; The model adjustment module is used to adjust the first image coding network based on the difference between the first image recognition result and the second image recognition result to obtain the trained first multimodal scene recognition model. The scene recognition module is used to perform scene recognition based on the trained first multimodal scene recognition model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images