Image recognition model training method and image recognition model application method

Abstract

The application provides an image recognition model training method and an image recognition model application method. The image recognition model is applied to a blind aid system and comprises an image label recognition encoder and an image depth recognition encoder. The method comprises the following steps: pre-processing a first sample image to obtain a first target image label corresponding to the first sample image and a plurality of middle-layer prompt images corresponding to the first sample image; training the image label recognition encoder based on the first target image label, the first sample image and a preset label semantic library to obtain a trained image label recognition encoder; and training the image depth recognition encoder based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder. The image recognition model in the embodiment of the application improves the generalization ability of the model by combining semantic information and simultaneously obtains the depth of the image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to an image recognition model training method and an image recognition model application method. BACKGROUND

[0002] Currently, some products can help the visually impaired, such as guide the visually impaired to avoid obstacles by using a blind stick or smart glasses. However, these products can usually only identify whether there is an obstacle in front of the visually impaired person, and guide the visually impaired person to avoid the obstacle in the case of an obstacle. In addition, these products cannot well help the visually impaired to perceive more external information, and the visually impaired can obtain too little external information. SUMMARY

[0003] The application provides an image recognition model training method and an image recognition model application method, which can solve the technical problem that current blind guiding products cannot well help the visually impaired to perceive more external information, and the visually impaired can obtain too little external information.

[0004] The first aspect embodiment of the application provides an image recognition model training method, which is applied to a blind guiding system, and the image recognition model comprises an image label recognition encoder and an image depth recognition encoder. The method comprises the following steps:

[0005] Preprocessing a first sample image to obtain a first target image label corresponding to the first sample image and a plurality of middle-layer prompt images corresponding to the first sample image; the plurality of middle-layer prompt images are images of different collection angles corresponding to the first sample image under different camera poses;

[0006] Training the image label recognition encoder based on the first target image label, the first sample image and a preset label semantic library to obtain a trained image label recognition encoder; the preset label semantic library comprises semantic information of image labels corresponding to a plurality of sample images respectively;

[0007] Training the image depth recognition encoder based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder.

[0008] The second aspect embodiment of the application provides an image recognition model application method, which is applied to a blind guiding system and comprises the following steps:

[0009] Receiving a target image collected by a blind guiding device;

[0010] input the target image into a pre-trained image recognition model to obtain a first target image label corresponding to the target image and a target image depth; the image recognition model is obtained by using the image recognition model training method of the first aspect;

[0011] output text description information for the target image based on the first target image label and the target image depth;

[0012] send the text description information to the blind assistance device.

[0013] The third aspect of the present application proposes a blind assistance system, comprising a blind assistance device and a server,

[0014] The blind assistance device is configured to collect a target image and send the target image to the server.

[0015] The server is configured to input the target image into a pre-trained image recognition model to obtain a first target image label corresponding to the target image and a target image depth, output text description information for the target image based on the first target image label and the target image depth, and send the text description information to the blind assistance device; the image recognition model is obtained by using the image recognition model training method of the first aspect.

[0016] The blind assistance device is further configured to receive the text description information and perform a corresponding blind assistance operation based on the text description information, wherein the blind assistance operation includes at least one of playing the text description information by voice and performing path navigation based on the text description information.

[0017] The fourth aspect of the present application proposes a blind assistance device, comprising a wearable component and a navigation component.

[0018] The wearable component is configured to collect a target image and send the target image to a server, receive text description information of the target image returned by the server, and send a navigation instruction to the navigation component based on the text description information, wherein the text description information is used to describe image content of the target image, the text description information is determined based on a first target image label and a target image depth, the first target image label and the target image depth are obtained by a pre-trained image recognition model based on the target image, and the image recognition model is obtained by using the image recognition model training method of the first aspect.

[0019] The navigation component is configured to receive the navigation instruction and perform a navigation operation corresponding to the navigation instruction.

[0020] The embodiment of the fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method of the first aspect.

[0021] The embodiment of the sixth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method of the first aspect.

[0022] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0023] The present application provides an image recognition model training method and an image recognition model application method. The image recognition model is applied to a blind aid system. The image recognition model includes an image label recognition encoder and an image depth recognition encoder. The image recognition model includes: pre-processing a first sample image to obtain a first target image label corresponding to the first sample image and a plurality of middle-layer prompt images corresponding to the first sample image; the plurality of middle-layer prompt images are images of different collection angles corresponding to the first sample image under different camera poses; training the image label recognition encoder based on the first target image label, the first sample image, and a preset label semantic library to obtain a trained image label recognition encoder; the preset label semantic library includes semantic information of image labels corresponding to a plurality of sample images respectively; and training the image depth recognition encoder based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder. The image recognition model in the embodiment of the present application improves the generalization ability of the model by combining semantic information, and at the same time obtains the depth of the image.

[0024] Additional aspects and advantages will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0025] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application. Moreover, the same reference numerals are used throughout the same figures. In the drawings:

[0026] Figure 1 A flowchart of an image recognition model training method provided by an embodiment of the present application is shown;

[0027] Figure 2 A structural schematic diagram of model optimization provided by an embodiment of the present application is shown;

[0028] Figure 3A structural schematic diagram of model training provided by an embodiment of the present application is shown.

[0029] Figure 4 A flow chart of an image recognition model application method provided by an embodiment of the present application is shown.

[0030] Figure 5 A structural schematic diagram of a blind aid system provided by an embodiment of the present application is shown.

[0031] Figure 6 A structural schematic diagram of a blind aid device provided by an embodiment of the present application is shown.

[0032] Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown.

[0033] Figure 8 A schematic diagram of a storage medium provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thoroughly and completely understood, and will fully convey the scope of the application to those skilled in the art.

[0035] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the usual meaning understood by those skilled in the art to which the present application belongs.

[0036] In the light of the above background, according to statistics of the WHO, at least 2.2 billion people worldwide are visually impaired or blind. There are currently some applications and tools that can help the visually impaired, but they usually only provide a limited function, and are not always easy to use, such as the physical tool of a blind aid stick used by the visually impaired to help them predict obstacles around them to some extent; or a guide dog type tool that guides the visually impaired to avoid obstacles or guides the visually impaired to plan a route. Help can also be sought from others, but most people prefer to explore the world as independently as possible. The existing tools for helping the visually impaired cannot well help the visually impaired to perceive more information from the outside world, and the way for the visually impaired to obtain information from the outside world is either too little information or not strong enough subjective initiative.

[0037] Previous smart headsets can only do simple text recognition and limited object recognition, but cannot perform accurate positioning, and voice interaction is also limited to a limited set.

[0038] And the current intelligent head-mounted device is generally combined with a picture recognition function, that is, the intelligent head-mounted device collects images, and the picture recognition function analyzes the collected images, and sends the recognized objects to the intelligent head-mounted device, so as to assist the visually impaired person with the intelligent head-mounted device.

[0039] Picture recognition aims to provide semantic labels by identifying multiple labels of a given image, which can be understood as giving multiple labels to describe the picture, including objects, scenes, attributes, and actions, and is a multi-label classification.

[0040] The previous labeling method is based on known labels for closed set image recognition, and in addition, because there is a domain shift between the source training data and the target test data, there is no way to implement recognition detection in an open domain that changes frequently, such as being unable to recognize an object that has never been seen, and a model trained in sunny conditions will significantly decrease when tested in foggy conditions.

[0041] To solve the above technical problems, the present application provides an image recognition model training method and an image recognition model application method, the image recognition model is applied to a blind aid system, and the image recognition model comprises: an image label recognition encoder and an image depth recognition encoder, which comprise: pre-processing a first sample image to obtain a first target image label corresponding to the first sample image and a plurality of middle-layer prompt images corresponding to the first sample image; the plurality of middle-layer prompt images are images of different collection angles corresponding to the first sample image under different camera poses; training the image label recognition encoder based on the first target image label, the first sample image and a preset label semantic library to obtain a trained image label recognition encoder; the preset label semantic library comprises semantic information of image labels corresponding to a plurality of sample images respectively; and training the image depth recognition encoder based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder. In the embodiment of the present application, the corresponding label is obtained by pre-processing the sample image, which can improve the number of sample sets compared with manual labeling or training based on manually labeled labels. The image recognition model of the embodiment of the present application combines semantic information, which can improve the generalization ability of the model based on the context in the semantic information when the image recognition model recognizes, and at the same time, the image depth recognition encoder can be trained based on the same training set, so that the image recognition model trained can obtain the depth of the image.

[0042] The image recognition model training method of the present application can be executed by a computing device, which can be a server, for example, one server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, for example, a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, a billboard, an all-in-one machine, etc. The present application does not limit the device type and the number of devices of the computing device

[0043] An image recognition model training method according to an embodiment of the present application is described below in combination with the accompanying drawings. The image recognition model is applied to a blind aid system and includes an image label recognition encoder and an image depth recognition encoder.

[0044] Referring to Figure 1 The method specifically includes the following steps:

[0045] S101, pre-process the first sample image to obtain a first target image label corresponding to the first sample image and a plurality of middle-level prompt images corresponding to the first sample image.

[0046] The plurality of middle-level prompt images are images of different collection angles of the first sample image under different camera poses.

[0047] The first sample image can be image data such as a picture or a video.

[0048] The source of the first sample image can be the Internet, social media, etc.

[0049] The pre-processing includes automatically labeling the image label of the first sample image and obtaining the plurality of middle-level prompt images corresponding to the first sample image, etc.

[0050] Compared with manual labeling and using labeled image data, a larger number of training sets can be obtained by pre-processing the first sample image, thereby improving the recognition accuracy of the model.

[0051] The camera pose can be the position and attitude of the camera. The pixel coordinates of the same object in the first sample image obtained by different positions and attitudes are different, and the difference in pixel coordinates of a point in the same world coordinate system in different images is the parallax. The parallax between different images can be converted into the distance between the object and the shooting point, that is, the depth. That is, the image depth of the first sample image can be obtained by pre-processing the first sample image.

[0052] S102, train the image label recognition encoder based on the first target image label, the first sample image, and a preset label semantic library to obtain a trained image label recognition encoder.

[0053] The preset label semantic library includes semantic information of image labels corresponding to the plurality of sample images respectively.

[0054] In each training process, after obtaining the image label of the sample image, the image label is encoded, the semantic information of the image label is obtained, and the semantic information is stored in the preset label semantic library. Since the training process generally requires multiple training, the semantic information in the preset label semantic library is relatively rich. Further, in the process of training the image label recognition encoder, the semantic information in the preset label semantic library can make the image label recognition encoder have a generalization ability.

[0055] In each training process, the image label recognition encoder outputs the corresponding sample image label based on the input first sample image, the network structure of the image label recognition encoder, and the semantic information in the preset label semantic library, so as to supervise the model parameters of the image label recognition encoder by the first target image label.

[0056] After multiple training, the trained image label recognition encoder can be obtained.

[0057] S103, based on the plurality of middle layer prompt images and the first sample image, training the image depth recognition encoder to obtain a trained image depth recognition encoder.

[0058] The image depth is the number of bits used to represent the accuracy of color information of each pixel in an image, and the pixels at different positions are inconsistent. Therefore, the image depth can represent the distance of each object in the image to the photographer.

[0059] While training the image label recognition encoder, the image depth recognition encoder can be trained based on the same training set.

[0060] Based on the same reason, in each training process, the image depth recognition encoder can output the corresponding sample image depth based on the input first sample image, so as to supervise the sample image depth based on the plurality of middle layer prompt images to train the image depth recognition encoder.

[0061] The application provides an image recognition model training method. The image recognition model is applied to a blind aid system. The image recognition model comprises an image label recognition encoder and an image depth recognition encoder. The method comprises the following steps: pre-processing a first sample image to obtain a first target image label corresponding to the first sample image and a plurality of middle-layer prompt images corresponding to the first sample image; the plurality of middle-layer prompt images are images of different collection angles corresponding to the first sample image under different camera poses; training the image label recognition encoder based on the first target image label, the first sample image and a preset label semantic library to obtain a trained image label recognition encoder; the preset label semantic library comprises semantic information of image labels corresponding to a plurality of sample images; and training the image depth recognition encoder based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder. In the embodiment of the application, the corresponding label is obtained by pre-processing the sample image. Compared with manual labeling or training based on manually labeled labels, the number of sample sets can be increased. The image recognition model in the embodiment of the application combines semantic information. When the image recognition model performs recognition, the context in the semantic information can be used to improve the generalization ability of the model. Meanwhile, the image depth recognition encoder can be trained based on the same training set, so that the image recognition model obtained by training can obtain the depth of the image.

[0062] In some embodiments, training the image label recognition encoder based on the first target image label, the first sample image and the preset label semantic library to obtain a trained image label recognition encoder comprises: performing an encoding operation on the first target image label to obtain semantic information of the first target image label; storing the semantic information in the preset label semantic library; inputting the first sample image into the image encoder to obtain image features; inputting the image features into the image label recognition encoder, so that the image label recognition encoder outputs a first sample image label based on the network structure of the image label recognition encoder and the semantic information of the preset label semantic library; calculating a first loss function value based on the first sample image label and the first target image label; adjusting the model parameters of the image label recognition encoder based on the first loss function value, continuing the training until a preset first training completion condition is met, and obtaining the trained image label recognition encoder.

[0063] In some embodiments, the encoding operation on the first target image label can be implemented by performing the encoding operation on the first target image label based on a contrastive text-image pretraining (CLIP) model, so as to obtain the semantic information of the first target image label.

[0064] Specifically, for an input image, the CLIP model converts it into a semantic vector that captures the semantic information contained in the image. For an input text description, the CLIP model also generates a semantic vector representing the semantic information contained in the text.

[0065] Further, the semantic information is stored in the preset label semantic library to enrich the semantic information in the preset label semantic library.

[0066] In each training process, the first sample image is input into the image encoder to obtain an image feature, and further, the image label recognition encoder outputs a first sample image label corresponding to the image feature based on the network structure of the image label recognition encoder and the semantic information of the preset label semantic library.

[0067] Then, a first loss function value is calculated based on the first sample image and the first target image label, where the first loss function value can be a cross-entropy loss value or the like.

[0068] The model parameters of the image label recognition decoder are updated using a backpropagation algorithm and an optimization method, and the training is continued until a preset first training completion condition is met, to obtain a trained image label recognition encoder.

[0069] The first training completion condition can be that the training is performed for a preset number of times or that the first loss function value is less than a preset threshold value of the image label recognition encoder, where the preset threshold value can be flexibly set based on actual conditions.

[0070] In some embodiments, to meet the requirement of outputting text by the model, the image recognition model further includes an image label interaction encoder and an image label text generation decoder.

[0071] The image label recognition encoder is trained based on the first target image label, the first sample image, and the preset label semantic library to obtain a trained image label recognition encoder, including: inputting the first sample image label and the image feature into the image label interaction encoder to obtain a text feature vector; inputting the text feature vector into the image label text generation decoder to obtain a sample text; parsing the sample text to obtain a second sample image label; calculating a second loss function value based on the second sample image label and the first sample image label; adjusting the model parameters corresponding to the image label recognition encoder, the image label interaction encoder, and the image label text generation decoder based on the first loss function value and the second loss function value, and continuing the training until a preset second training completion condition is met to obtain the trained image label recognition encoder, the image label interaction encoder, and the image label text generation decoder.

[0072] In each training process, after the image label recognition encoder obtains the first sample image label, the image label interaction encoder and the image label text generation decoder can be supervised trained through the first sample image label.

[0073] Specifically, the first sample image label and the image feature can be input into the image label interaction encoder to obtain a text feature vector; the text feature vector can be input into the image label text generation decoder to obtain a sample text; and the sample text can be parsed to obtain a second sample image label.

[0074] Further, based on the second sample image label and the first sample image label, a second loss function value can be calculated.

[0075] The second loss function value can also be a cross-entropy loss value.

[0076] Further, the first loss function value and the second loss function value can be used for joint training to adjust the model parameters corresponding to the image label recognition encoder, the image label interaction encoder and the image label text generation decoder, and the training can be continued until a preset second training completion condition is met to obtain the trained image label recognition encoder, the image label interaction encoder and the image label text generation decoder.

[0077] The second training completion condition can be a preset number of training times or the first loss function value and the second loss function value being less than a respective preset threshold, wherein the preset threshold can be flexibly set based on actual conditions.

[0078] In some embodiments, the image depth recognition encoder is trained based on a plurality of middle-layer prompt images to obtain a trained image depth recognition encoder, including: reconstructing the plurality of middle-layer prompt images to obtain reconstructed images and reconstructed image depths; inputting a first sample image into the image depth recognition encoder to obtain a sample image depth; calculating a third loss function value based on the reconstructed image depth and the sample image depth; adjusting the model parameters of the image depth recognition encoder based on the third loss function value, and continuing the training until a preset third training completion condition is met to obtain the trained image depth recognition encoder.

[0079] In some embodiments, the process of reconstructing the plurality of middle-layer prompt images to obtain the reconstructed images and the reconstructed image depths can be implemented as:

[0080] Feature extraction: using computer vision algorithms and techniques, features of the middle-layer prompt image are extracted. This can include edge detection, texture analysis, color distribution, etc.

[0081] Feature encoding: the extracted features are encoded into digital representations for subsequent processing and analysis. Common encoding methods include vector quantization, hash encoding, etc.

[0082] Reconstruction model training: Train a reconstruction model using machine learning or deep learning techniques. This model can be an autoencoder, generative adversarial network (GAN), or other suitable model for the task.

[0083] Reconstructed image generation: Use the trained reconstruction model to map the encoded features back to the image space, generating a reconstructed image. This step involves decoding, deconvolution, and other operations.

[0084] Further, input the first sample image into the image depth recognition encoder to obtain a sample image depth; calculate a third loss function value based on the reconstructed image depth and the sample image depth; adjust the model parameters of the image depth recognition encoder based on the third loss function value, continue training until a preset third training completion condition is met, and obtain a trained image depth recognition encoder.

[0085] The third loss function value can also be a cross-entropy loss value, and the third training completion condition can be training a preset number of times or the third loss function value being less than a preset threshold value of the image depth recognition encoder, wherein the preset threshold value can be flexibly set based on actual conditions.

[0086] In some embodiments, the sample image is preprocessed to obtain target label data corresponding to the sample image and a plurality of middle-level prompt images corresponding to the sample image, including: obtaining a first image label by automatically parsing the semantic of the first sample image, the first label data being an unannotated image label; annotating the first image label through a preset annotation model to obtain a second image label; correcting the annotations in the second image label through a data engine to obtain a target image label; inputting the first sample image into an image annotator to obtain a plurality of middle-level prompt images, the image annotator including a plurality of camera pose information.

[0087] Wherein, the process of automatic text semantic parsing can be implemented as:

[0088] Text extraction: Extract semantic information related to images from text data, such as picture descriptions, labels, keywords, etc.

[0089] Semantic understanding: Perform semantic understanding on the extracted text information, including part-of-speech tagging, entity recognition, syntax analysis, etc., to understand the meaning and structure of the text.

[0090] Text-image association: Establish the association between text and image, such as matching the corresponding image content through the description or label in the text.

[0091] Image feature extraction: Extract features from the associated image, converting the image into a digital representation, such as using a convolutional neural network (CNN) to extract an image feature vector.

[0092] Text-image association modeling: using machine learning or deep learning techniques to establish a model of association between text and image, such as a text-to-image mapping model.

[0093] Text semantic parsing: combining text semantic information with image features to achieve semantic parsing of text on image. This may involve judging and reasoning the consistency between text description and image content.

[0094] Image semantic understanding: based on the results of text semantic parsing, further semantic understanding of the image, such as recognizing objects, scenes and their relationships in the image.

[0095] Further, the first image label can be annotated by a preset annotation model to obtain a second image label, wherein the preset annotation model can be obtained by supervised training of sample images of Yushu River and corresponding image labels.

[0096] Since the second image label may have incorrect annotations and missed annotations, the annotations in the second image label need to be corrected, that is, the incorrect annotations in the second image label need to be removed and the missed annotations need to be added.

[0097] In some embodiments, the correction of annotations in the second image label by the data engine can be implemented as follows:

[0098] Label preprocessing: pre-processing of original label data, including deduplication, standardization, error correction, etc., to ensure the accuracy and consistency of label data.

[0099] Label cleaning: cleaning label data through algorithms, including the following aspects:

[0100] Spelling check: spelling check of labels to correct possible spelling errors.

[0101] Label integration: integrating or merging similar labels to eliminate redundancy between labels.

[0102] Label filtering: excluding irrelevant or inappropriate labels, such as noise labels or error labels.

[0103] Label standardization: uniform naming specification of labels to ensure consistency and comparability of labels.

[0104] At the same time, the first sample image can also be input into the image annotator to obtain multiple middle-level prompt images.

[0105] In some embodiments, to improve the accuracy of the trained image recognition model, the method further comprises: obtaining a second sample image and first target data corresponding to the second sample image, the first target data being label data of the second sample image, the label data including a second target image label and target image depth; and performing secondary training on the trained image recognition model based on the first target data and the second sample image.

[0106] The second sample image and the first target data corresponding to the second sample image are a small quantity but high quality data set, so that the image recognition model can be subjected to secondary training, the model parameters can be fine-tuned, and the accuracy of the image recognition model can be further improved.

[0107] In some embodiments, in the real world, the visual perception system operates in a non-stationary and constantly changing environment, and the distribution of the target domain can change over time. Therefore, the recognition ability of the image recognition model can be affected by these non-stationary environments. Due to the change in the distribution of the target domain over time, the labels obtained by the algorithm become unreliable. The method further comprises: obtaining a third sample image and a fourth sample image obtained by data augmentation on the third sample image; obtaining a teacher model based on the trained image recognition model; inputting the third sample image into the target image recognition model to obtain a first output result; inputting the fourth sample image into the teacher model to obtain a second output result; performing an averaging operation on the second output result to obtain a third output result; calculating a fourth loss function value based on the third output result and the first output result; adjusting the model parameters of the target image recognition model based on the fourth loss function value, and continuing training until a preset fourth training completion condition is met, to obtain an optimized image recognition model.

[0108] The third sample image can be any sample image.

[0109] In the training of the deep learning algorithm model of the present method, by performing weight averaging on the models at multiple time steps during the training process, a more accurate model than the final model can usually be obtained. This is because weight averaging can reduce noise and fluctuations during the training process, thereby improving the robustness and generalization performance of the model.

[0110] The fourth training completion condition can be that the training reaches a preset number of times, or the fourth loss function value is less than a preset threshold value of the student model. The preset threshold value can be flexibly set based on actual conditions, and will not be described here.

[0111] In the initial state, the teacher model is consistent with the image recognition model, and further, the fourth sample image and the corresponding supervision data are used to train the teacher model.

[0112] Further, the third sample image is input into the target image recognition model to obtain a first output result, and the fourth sample image is input into the teacher model to obtain a second output result.

[0113] In some embodiments, in the initial stage, the student model and the teacher model have the same architecture and initial parameters, so the prediction result of the student model is very close to the prediction result of the teacher model. These similarities can be used to train the student model.

[0114] Specifically, the teacher model can be used to predict the training data to obtain the output result of the teacher model. Then, these output results are used as the target labels of the student model, and the student model is trained based on these target labels. During the training process, the loss function can be calculated according to the prediction result of the teacher model and the output result of the student model, and the weights of the student model are updated according to the value of the loss function. That is, the fourth loss function value is calculated based on the third output result and the first output result; the model parameters of the target image recognition model are adjusted based on the fourth loss function value, and the training is continued until the fourth training completion condition is met, and the optimized image recognition model is obtained.

[0115] In subsequent training, multiple different teacher models can be used to predict the training data to obtain multiple different target labels, and then these target labels are averaged to obtain the target labels of the student model. In this way, the performance and stability of the student model can be further improved.

[0116] It should be noted that the trained image recognition model can also be optimized during the use of the model.

[0117] For the convenience of understanding, the application provides a structure diagram of a teacher model training an image recognition model. As shown in Figure 2 , the structure diagram of the teacher model training the image recognition model includes a student model and a teacher model.

[0118] First, based on the trained image recognition model, the student model and the teacher model are obtained, and in the initial stage, the student model and the teacher model have the same architecture and initial parameters, that is, both are image recognition models. The third sample image is input into the student model, and the fourth sample image is input into the teacher model, and further, the first output result output by the student model and the second output result output by the teacher model are obtained. The second output result is averaged to obtain the third output result, and the consistency loss, that is, the fourth loss function value, is calculated by backward propagation of the third output result and the first output result. The model parameters of the target image recognition model are adjusted based on the fourth loss function value, that is, the weights of the student model are adjusted, and further, the weights of the teacher model are updated using the weights of the student model, wherein the updating method can be a simple average method or an exponential average method.

[0119] Further, the teacher model with the updated weight is used to train the student model with the updated weight next time until a preset fourth training completion condition is met, and an optimized image recognition model is obtained.

[0120] Since the weight of the student model is updated to update the weight of the teacher model each time, and the weight of the teacher model is used for reverse propagation to update the weight of the student model in the next training process, that is, in the multiple training processes, the weight of the student model at multiple time steps is averaged, so that a more accurate image recognition model can be obtained.

[0121] In order to describe the training process of the image recognition model in detail, the embodiment of the present application also provides a structural schematic diagram of model training, as shown in Figure 3

[0122] The structural schematic diagram includes an image annotator, an image depth recognition encoder, an image encoder, an image label interaction encoder, an image label text generation encoder and an image label recognition encoder.

[0123] Firstly, the training processes of the image label interaction encoder, the image label text generation encoder and the image label recognition encoder are described.

[0124] After the first sample image is preprocessed, a first target image label corresponding to the first sample image is obtained, and the first image label can include a kitchen, a stove, a pot and a watermelon, etc. The first target image label is encoded to obtain semantic information of the first target image label; and the semantic information is stored in a preset label semantic library. Further, the first sample image is input into the image encoder, so that the image encoder outputs image features; the image features are input into the image label recognition encoder, so that the image label recognition encoder outputs a first sample image label based on the network structure of the image label recognition encoder and the semantic information of the preset label semantic library; and a first loss function value is calculated based on the first sample image label and the first target image label.

[0125] Meanwhile, the first sample image label and the image features are input into the image label interaction encoder to obtain a text feature vector; the text feature vector is input into the image label text generation decoder to obtain a sample text; a second loss function value is calculated based on the second sample image label and the first sample image label; and model parameters corresponding to the image label recognition encoder, the image label interaction encoder and the image label text generation decoder are adjusted based on the first loss function value and the second loss function value, and the training is continued until a trained image label recognition encoder, image label interaction encoder and image label text generation decoder are obtained by meeting a preset second training completion condition.

[0126] Further, the training process of the image depth recognition encoder is described. ​

[0127] The first sample image is input into the image annotator to obtain multiple intermediate cue images; the multiple intermediate cue images are reconstructed to obtain reconstructed images and reconstructed image depths; the first sample image is input into the image depth recognition encoder to obtain the sample image depth; based on the reconstructed image depth and the sample image depth, the third loss function value is calculated; based on the third loss function value, the model parameters of the image depth recognition encoder are adjusted, and training continues until the preset third training completion condition is met, resulting in a trained image depth recognition encoder.

[0128] The following description, in conjunction with the accompanying drawings, describes an image recognition model application method based on an embodiment of this application. This image recognition model application method is applied to an assistive system for the blind. (See attached figures.) Figure 4 The method specifically includes the following steps:

[0129] S401, Receive the target image acquired by the assistive device for the blind.

[0130] S402. Input the target image into the pre-trained image recognition model to obtain the first target image label and target image depth corresponding to the target image.

[0131] Image recognition models utilize Figure 1 The image recognition model was obtained using the training method shown.

[0132] S403. Output textual description information for the target image based on the first target image label and the target image depth.

[0133] S404. Send the text description information to the assistive device for the visually impaired.

[0134] After the server determines the objects and their depth information in the current target image, this image recognition model can be applied in the following scenarios:

[0135] 1. Locating a specific object: For example, if the wearer says, "Help me find my headphones," the device uses its speech-to-text capability to convert the wearer's voice into a text command and transmits it to the computing center for processing. Once the device recognizes that the user's intention is to find the specific object "headphones," it activates the camera to capture the surrounding environment. The video is then processed by algorithms through the cloud / local computing center to pinpoint the exact location of the "headphones" in the environment. Finally, the device informs the user of the exact location of the "headphones" via voice.

[0136] 2. Pilot blind: when the wearer says where he wants to go, the device converts the wearer's voice into text instructions through the voice-to-text capability and transmits them to the operation center for processing, identifies the user's intention to go to a specific place, and starts the camera to shoot the surrounding environment at this time. Plan the route, guide the wearer to walk according to the established route, remind the road conditions in front of the navigation process, crossroads, traffic light conditions, etc., to ensure that the wearer can reach the destination smoothly.

[0137] 3. Environment description:

[0138] When a visually impaired user enters a space he is not very familiar with, he can describe the surrounding environment through this product, such as describing the current state as a whole and reminding the user of what people or objects are around him and where the specific direction is.

[0139] 4. Environmental space-time awareness reminder:

[0140] The visually impaired user can perceive the surrounding environment and space-time changes such as weather and time through this device

[0141] 5. Identify friends around:

[0142] When using the device for the first time, you can record relevant information such as family, friends, colleagues, classmates, etc. with the help of staff or family members. When you perceive people you know around you, you can remind the visually impaired user who the people you know around you are and where they are.

[0143] 6. Perception of people's emotions around:

[0144] If the visually impaired user wants to know the facial expressions and emotions of others around him, he can easily perceive them through this device.

[0145] 7. Text reading:

[0146] The visually impaired user can read text through this device, such as reminding signs, menus, on-screen words, electronic screens, paper and electronic documents, etc.

[0147] 8. Intelligent question and answer of the surrounding environment:

[0148] The visually impaired user can interact with the device through voice and conduct intelligent question and answer, such as asking if there are any historical sites in the current area, etc. Get knowledge anytime, anywhere.

[0149] Next, an assistive blind system according to an embodiment of the present application will be described in conjunction with the accompanying drawings, as shown in the drawings. The assistive blind system includes an assistive blind device and a server, Figure 5

[0150] The assistive blind device is used to collect target images and send the target images to the server;​

[0151] The server is configured to input the target image into a pre-trained image recognition model to obtain a first target image label and a target image depth corresponding to the target image, output text description information for the target image based on the first target image label and the target image depth, and send the text description information to the blind assistance device.

[0152] The blind assistance device is further configured to receive the text description information and perform a corresponding blind assistance operation based on the text description information, the blind assistance operation including at least one of playing the text description information in voice and performing path navigation based on the text description information.

[0153] The blind assistance device can collect the target image using the collection equipment and send the target image to the server upon receiving a voice instruction from the user.

[0154] The server is deployed with an image recognition model that can output a first target image label and a target image depth based on a target image, and further output text description information for the target image based on the first target image label and the target image depth, and send the text description information to the blind assistance device.

[0155] The image recognition model can also output a first target image label, a target image depth, and text description information based on a target image, and the server further sends the text description information to the blind assistance device.

[0156] The output of the specific image recognition model can be flexibly set based on actual conditions.

[0157] In addition, the blind assistance system can further include a terminal configured to receive user operations and send image collection instructions to the blind assistance device.

[0158] A blind assistance device according to an embodiment of the present application is described below in conjunction with the accompanying drawings, which includes a wearable component and a navigation component.

[0159] The wearable component is configured to collect a target image, send the target image to a server, receive text description information of the target image returned by the server, the text description information being used to describe the image content of the target image, and send a navigation instruction to the navigation component based on the text description information.

[0160] The navigation component is configured to receive the navigation instruction and perform a navigation operation corresponding to the navigation instruction.

[0161] The wearable component cooperates with the navigation component to reach the destination more smoothly. The navigation component can have a 360° travel ability driven by electricity, and is connected to the wearer of the head-mounted device through a stable and light solid object (such as a walking stick, a blind stick, etc.). When the wearer speaks the place he wants to go, the head-mounted device converts the voice of the wearer into a text instruction through the voice-to-text capability, and transmits the text instruction to the operation center for processing. The operation center identifies that the intention of the user is to go to a specific place, and at this time, the camera is started to shoot the environment around the place, and the travel route of the navigation device is planned. The operation center controls the navigation device to move through the intelligent control algorithm, and continuously collects the surrounding environment for feedback and calculation to generate a new travel route. The travel speed is controlled through the link with the user, and effective obstacle avoidance is performed. The head-mounted device cooperates with the navigation device to issue voice instructions for warning and guiding when changing lanes and avoiding obstacles, so as to ensure that the wearer can reach the destination smoothly.

[0162] The embodiment of the present application also provides an electronic device for executing the image recognition model training method. Figure 7 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in Figure 7 The electronic device 7 includes a processor 700, a memory 701, a bus 702 and a communication interface 703, the processor 700, the communication interface 703 and the memory 701 are connected through the bus 702; the memory 701 stores a computer program executable on the processor 700, and the processor 700 executes the computer program to execute the image recognition model training method provided by any of the preceding embodiments of the present application.

[0163] The memory 701 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication between the device network element and at least one other network element is realized through at least one communication interface 703 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0164] The bus 702 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 701 is used to store programs, and the processor 700 executes the programs after receiving execution instructions. The image recognition model training method disclosed in any of the preceding embodiments of the present application can be applied to the processor 700 or realized by the processor 700.

[0165] The processor 700 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuits or instructions in the form of software in the processor 700. The processor 700 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701, and combines the hardware to complete the steps of the above method.

[0166] The electronic device provided by the embodiments of the present application and the image recognition model training method provided by the embodiments of the present application have the same beneficial effects as the methods they adopt, run or implement.

[0167] The present application also provides a computer readable storage medium corresponding to the image recognition model training method provided by the preceding embodiments. Please refer to Figure 8 The computer readable storage medium shown is an optical disc 30, and a computer program (i.e. program product) is stored on the optical disc 30. When the computer program is run by the processor, the image recognition model training method provided by any of the preceding embodiments will be executed.

[0168] It should be noted that examples of computer readable storage media can also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical, magnetic storage media, which will not be described one by one here.

[0169] The computer readable storage medium provided by the above embodiments of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the image recognition model training method provided by the embodiments of the present application.

[0170] It should be noted that:

[0171] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

[0172] Similarly, it is to be understood that the various features of the application described herein are sometimes grouped together in a single embodiment, figure or description of a related art for the purpose of streamlining the disclosure. This method of disclosure, however, is not to be interpreted as reflecting an intention that the application requires more features than are explicitly recited in each claim. Rather, inventive aspects lie in less than all features of a single disclosed embodiment. Thus, the claims following, are hereby expressly incorporated into this detailed description, with each claim acting as a separate embodiment of the application. Furthermore, the claims can be combined with one another in any manner to form additional embodiments of the application.

[0173] Furthermore, to one of ordinary skill in the art, these and other modifications to, and

[0174] The above merely provides the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training an image recognition model, characterized in that, The image recognition model is applied to a blind aid system, and the image recognition model comprises an image label recognition encoder and an image depth recognition encoder. The first sample image is preprocessed to obtain a first target image label corresponding to the first sample image and a plurality of middle-layer prompt images corresponding to the first sample image; the plurality of middle-layer prompt images are images of different collection angles corresponding to the first sample image under different camera poses; The image label recognition encoder is trained based on the first target image label, the first sample image, and a preset label semantic library to obtain a trained image label recognition encoder; The preset label semantic library comprises semantic information of image labels corresponding to a plurality of sample images respectively; The image depth recognition encoder is trained based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder; The image label recognition encoder is trained based on the first target image label, the first sample image, and a preset label semantic library to obtain a trained image label recognition encoder, comprising: The first target image label is encoded to obtain semantic information of the first target image label; The semantic information is stored in a preset label semantic library; The first sample image is input into an image encoder to obtain image features; The image features are input into the image label recognition encoder, so that the image label recognition encoder outputs a first sample image label based on its own network structure and the semantic information of the preset label semantic library; A first loss function value is calculated based on the first sample image label and the first target image label; The first sample image label and the image features are input into an image label interaction encoder to obtain a text feature vector; The text feature vector is input into an image label text generation decoder to obtain a sample text; The sample text is parsed to obtain a second sample image label; A second loss function value is calculated based on the second sample image label and the first sample image label; Based on the first loss function value and the second loss function value, the model parameters corresponding to the image label recognition encoder, the image label interaction encoder, and the image label text generation decoder are adjusted for continuous training until a preset second training completion condition is met to obtain a trained image label recognition encoder, image label interaction encoder, and image label text generation decoder.

2. The method of claim 1, wherein, The image depth recognition encoder is trained based on the plurality of middle-layer prompt images and the first sample image to obtain a trained image depth recognition encoder, comprising: The plurality of middle-layer prompt images are reconstructed to obtain reconstructed images and reconstructed image depths; The first sample image is input into the image depth recognition encoder to obtain a sample image depth; A third loss function value is calculated based on the reconstructed image depth and the sample image depth; Adjusting model parameters of the image depth recognition encoder based on the third loss function value, continuing training until a preset third training completion condition is met, and obtaining a trained image depth recognition encoder.

3. The method of claim 1, wherein, The pre-processing of the first sample image, obtaining the first target image label corresponding to the first sample image, and obtaining a plurality of middle-layer prompt images corresponding to the first sample image, comprises: Obtaining a first image label by automatically analyzing the semantic of the first sample image, wherein the first image label is an unannotated image label; Annotating the first image label by using a preset annotation model to obtain a second image label; Correcting the annotations in the second image label by using a data engine to obtain a target image label; Inputting the first sample image into an image annotator to obtain a plurality of middle-layer prompt images, wherein the image annotator comprises a plurality of camera pose information.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: Obtaining a second sample image and a first target data corresponding to the second sample image, wherein the first target data is annotation data of the second sample image, and the annotation data comprises a second target image label and a target image depth; Based on the first target data and the second sample image, the trained image recognition model is trained again.

5. The method of claim 4, wherein, The method further comprises: Obtaining a third sample image and a fourth sample image obtained by data enhancement on the third sample image; Based on the trained image recognition model, a student model and a teacher model are obtained; Inputting the third sample image into the student model to obtain a first output result; Inputting the fourth sample image into the teacher model to obtain a second output result; Performing an average operation on the second output result to obtain a third output result; Based on the third output result and the first output result, a fourth loss function value is calculated; Based on the fourth loss function value, the model parameters of the student model are adjusted, and the training is continued until a preset fourth training completion condition is met, and an optimized image recognition model is obtained.

6. An image recognition model application method, characterized by, Applied to a blind aid system, comprising: Receiving a target image collected by a blind aid device; Inputting the target image into a pre-trained image recognition model to obtain a first target image label corresponding to the target image and a target image depth; the image recognition model is obtained by using the image recognition model training method of any one of claims 1-5; Based on the first target image label and the target image depth, outputting text description information for the target image; Sending the text description information to the blind aid device.

7. A system for assisting the blind, characterized in that Comprising: A blind aid device and a server, The blind aid device is configured to collect a target image and send the target image to the server; The server is configured to input the target image into a pre-trained image recognition model to obtain a first target image label and a target image depth corresponding to the target image, output text description information of the target image based on the first target image label and the target image depth, and send the text description information to the blind assistance device, wherein the image recognition model is obtained by using the image recognition model training method in any one of claims 1-5. The blind assistance device is further configured to receive the text description information and perform a corresponding blind assistance operation based on the text description information, wherein the blind assistance operation includes at least one of playing the text description information by voice and performing path navigation based on the text description information.

8. A blind aid device characterized by, The method comprises: a wearable component and a navigation component; The wearable component is configured to collect a target image, send the target image to a server, receive text description information of the target image returned by the server, and send a navigation instruction to the navigation component based on the text description information, wherein the text description information is used to describe image content of the target image, the text description information is determined based on a first target image label and a target image depth, the first target image label and the target image depth are obtained by a pre-trained image recognition model based on the target image, the image recognition model is obtained by using the image recognition model training method in any one of claims 1-5, and the navigation component is configured to receive the navigation instruction and perform a corresponding navigation operation. The processor runs the computer program to implement the method in any one of claims 1-5 or claim 6.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The program is executed by the processor to implement the method in any one of claims 1-5 or claim 6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images