Prompt information generation method, object recognition method, device and unmanned vehicle

Abstract

The present disclosure provides a prompt information generation method, an object recognition method, a device and an unmanned vehicle, which relate to the field of artificial intelligence technology, specifically to the technical fields of computer vision, deep learning, large models, etc., and can be applied to scenarios such as intelligent transportation and autonomous driving. The specific implementation scheme is: performing multi-type image attribute conversion on the acquired initial sample image to obtain sample images with multiple image attributes, the initial sample image is related to the sample object to be recognized, and the initial sample image is associated with a label of the object category representing the sample object; obtaining a sample text for describing the sample object, and initial prompt information related to each of the multiple image attributes; determining the sample object recognition result according to the sample text, multiple sample images and multiple initial prompt information; and updating multiple initial prompt information according to the sample object recognition result and the label to generate multiple prompt information for object recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as intelligent transportation and autonomous driving. Background Art

[0002] In application scenarios such as vehicle autonomous driving, the image of the vehicle's driving environment can be collected, and target objects such as pedestrians, vehicles, and traffic signs in the collected images can be identified based on artificial intelligence technology. Based on the type, size, and other attributes of the identified target object, control information for controlling vehicle movement can be further generated to achieve the vehicle's autonomous driving function. Summary of the invention

[0003] The present disclosure provides a prompt information generating method, an object recognition method, a device, an electronic device, a storage medium and an unmanned vehicle.

[0004] According to one aspect of the present disclosure, a prompt information generating method is provided, comprising: performing multi-type image attribute conversion on an acquired initial sample image to obtain sample images respectively associated with multiple image attributes, the initial sample image being associated with a sample object to be identified, and the initial sample image being associated with a label representing an object category of the sample object; acquiring sample text for describing the sample object, and initial prompt information respectively associated with multiple image attributes; determining a sample object recognition result based on the sample text, multiple sample images, and multiple initial prompt information; and updating multiple initial prompt information based on the sample object recognition result and the label to generate multiple prompt information for object recognition.

[0005] According to another aspect of the present disclosure, there is provided an object recognition method, comprising: acquiring a target image related to a target object, a description text for describing the target object, and prompt information related to each of a plurality of image attributes, wherein the prompt information is determined according to a prompt information generation method provided in an embodiment of the present disclosure; performing object recognition according to the description text, the target image and the plurality of prompt information to obtain a target object recognition result.

[0006] According to another aspect of the present disclosure, a prompt information generating device is provided, comprising: a first attribute conversion module, used to perform multi-type image attribute conversion on an acquired initial sample image, to obtain sample images respectively associated with multiple image attributes, the initial sample image being associated with a sample object to be identified, and the initial sample image being associated with a label representing an object category of the sample object; a first acquisition module, used to acquire sample text for describing the sample object, and initial prompt information respectively associated with multiple image attributes; a sample object recognition result determination module, used to determine a sample object recognition result based on the sample text, multiple sample images and multiple initial prompt information; and a prompt information generating module, used to update multiple initial prompt information based on the sample object recognition result and the label, to generate multiple prompt information for object recognition.

[0007] According to another aspect of the present disclosure, an object recognition device is provided, including: a second acquisition module, used to acquire a target image related to a target object, a description text for describing the target object, and prompt information related to each of a plurality of image attributes, wherein the prompt information is determined according to a prompt information generation method provided in an embodiment of the present disclosure; and a recognition module, used to perform object recognition based on the description text, the target image and a plurality of prompt information to obtain a target object recognition result.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided according to an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method provided according to the embodiment of the present disclosure is implemented.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 An exemplary system architecture to which the prompt information generating method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0014] Figure 2 A flowchart of a method for generating prompt information according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3A The schematic diagram of the deep learning model according to the embodiment of the present disclosure is shown;

[0016] Figure 3B The schematic diagram of the text feature fusion network according to the embodiment of the present disclosure is shown;

[0017] Figure 4 A flowchart of an object recognition method according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 5 The schematic diagram shows a principle diagram of an object recognition method according to an embodiment of the present disclosure;

[0019] Figure 6 A block diagram of a prompt information generating device according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 7 A block diagram of an object recognition device according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 8 A schematic diagram showing an application scenario of an unmanned vehicle according to an embodiment of the present disclosure; and

[0022] Fig. 9 A block diagram of an electronic device suitable for implementing a prompt information generating method and an object identifying method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0025] With the development of artificial intelligence technology, in application scenarios such as vehicle autonomous driving and smart transportation, the collected images can be processed based on machine learning models to identify the types of target objects such as vehicles and pedestrians in the images, so that autonomous driving strategies and smart traffic indication strategies can be generated based on attribute information such as the types of the identified target objects, thereby improving the traffic efficiency. At the same time, in industrial application scenarios such as smart logistics and automated manufacturing, images of the working environment can be collected based on intelligent robots, and the automated operation process can be automatically executed by identifying the object categories in the collected images. Therefore, the recognition accuracy and efficiency of objects in the image have become key factors affecting the work efficiency in various application scenarios. The inventors have found that the types of target objects in application scenarios are diversified, and it is difficult to accurately identify the diverse object categories.

[0026] The embodiments of the present disclosure provide a prompt information generation method, an object recognition method, a device, an electronic device, a storage medium, and an unmanned vehicle. The prompt information generation method includes: performing multi-type image attribute conversion on the acquired initial sample image to obtain sample images related to multiple image attributes, the initial sample image is related to the sample object to be recognized, and the initial sample image is associated with a label representing the object category of the sample object; obtaining a sample text for describing the sample object, and initial prompt information related to each of the multiple image attributes; determining the sample object recognition result based on the sample text, multiple sample images, and multiple initial prompt information; and updating multiple initial prompt information based on the sample object recognition result and the label to generate multiple prompt information for object recognition.

[0027] According to an embodiment of the present disclosure, by performing multi-type image attribute conversion on an initial sample image, a sample image related to the image attributes is obtained, and a sample object recognition result is determined based on initial prompt information and sample images related to each image attribute, as well as a sample text for describing the sample object. The initial prompt information is updated based on the sample object recognition result and the label representing the object category. The updated multiple prompt information can respectively fuse the image semantic features of the sample object in each image attribute condition, so that the prompt information can fully learn the object attributes of the sample object and the image attributes of the sample image in multiple image attribute conditions, thereby performing object recognition based on multiple prompt information, which can improve the recognition accuracy of the object category of the object, reduce the training overhead required for training the target object recognition model, and improve the object recognition efficiency.

[0028] Figure 1 An exemplary system architecture to which the prompt information generating method and device according to an embodiment of the present disclosure can be applied is schematically shown.

[0029] It should be noted that Figure 1The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the prompt information generation method and apparatus can be applied may include a terminal device, but the terminal device may implement the prompt information generation method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0032] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0033] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the prompt information generating method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the prompt information generating device provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0035] Alternatively, the prompt information generation method provided in the embodiment of the present disclosure may also be generally executed by the server 15. Accordingly, the prompt information generation device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The prompt information generation method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the prompt information generation device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] Figure 2 The flowchart of the prompt information generating method according to the embodiment of the present disclosure is schematically shown.

[0038] like Figure 2 As shown, the prompt information generating method includes operations S210 to S240.

[0039] In operation S210, the acquired initial sample image is converted into multiple types of image attributes to obtain sample images with respective image attributes, the initial sample image is related to the sample object to be identified, and the initial sample image is associated with a label representing the object category of the sample object.

[0040] In operation S220, a sample text for describing the sample object and initial prompt information related to each of a plurality of image attributes are obtained.

[0041] In operation S230, a sample object recognition result is determined according to the sample text, the plurality of sample images, and the plurality of initial prompt information.

[0042] In operation S240, a plurality of initial prompt information is updated according to the sample object recognition result and the label to generate a plurality of prompt information for object recognition.

[0043] According to an embodiment of the present disclosure, the initial sample image may include an image obtained by capturing the vehicle's traffic environment, the sample object may include obstacles in the traffic environment, and the object categories may include pedestrians, bicycles, motorcycles, traffic signs, and the like.

[0044] According to an embodiment of the present disclosure, image attributes may include style attributes of the image, but are not limited thereto. They may also include environmental attributes such as lighting and weather conditions of the environment represented by the image, or may also include other types of image attributes such as color and size. The embodiment of the present disclosure does not limit the specific types of image attributes.

[0045] According to an embodiment of the present disclosure, multiple types of image attribute conversions can be performed on the initial sample image based on an image attribute conversion tool. For example, multiple types of image attribute conversions can be performed on the initial sample image based on a model constructed based on a convolutional neural network algorithm, or multiple types of image attribute conversions can be performed on the initial sample image based on other methods.

[0046] According to an embodiment of the present disclosure, the sample text used to describe the sample object may include text describing the object category of the sample object. For example, when a sample image records a bicycle, the sample text may be "a photo of aBicycle". The initial prompt information may include any type of information used to prompt the sample object to be identified, such as text information, vector information, etc. The embodiment of the present disclosure does not limit the specific information type of the initial prompt information.

[0047] According to an embodiment of the present disclosure, determining a sample object recognition result based on a sample text, a plurality of sample images, and a plurality of initial prompt information may include processing the sample text, a plurality of sample images, and a plurality of initial prompt information according to a pre-trained deep learning model to obtain a sample object recognition result. The pre-trained deep learning model may include one trained based on a supervised training method, or may also include one trained based on any type of training method such as an unsupervised training method, a semi-supervised training method, etc., and the embodiments of the present disclosure do not limit this.

[0048] According to the embodiments of the present disclosure, the sample recognition result can characterize the object category of the sample object, and the initial prompt information corresponding to each image attribute can be iteratively updated according to the difference between the sample object recognition result and the label, until the difference between the obtained sample object recognition result and the label meets the preset difference condition, and the prompt information corresponding to each image attribute can be obtained. In the case where the difference between the sample object recognition result and the label meets the preset difference condition, the prompt information can fully learn the object attributes of the sample object and the image attributes of the sample image in multiple image attribute conditions. By introducing the updated prompt information in object recognition, the learning capabilities of images of multiple image attribute types and multiple object categories can be transferred in the object recognition task, so that object recognition can be performed according to multiple prompt information, which can improve the recognition accuracy of the object category of the object, reduce the training overhead required for training the target object recognition model, and improve the object recognition efficiency.

[0049] It should be noted that the prompt information generation method provided in the embodiments of the present disclosure can be applied to any application scenario. For example, multiple prompt information generated by the method provided in the embodiments of the present disclosure can be used to identify target objects such as pedestrians, vehicles, traffic signs, etc. in a vehicle traffic environment, or it can also be applied to logistics operation scenarios to perform object recognition on target objects such as goods. The embodiments of the present disclosure do not limit the specific application scenarios of the prompt information generation method.

[0050] It should be noted that in the prompt information generation method provided by the embodiment of the present disclosure, information acquisition authorization is obtained before information (including but not limited to sample images, labels) is acquired, and necessary encryption methods are used during storage, interaction, etc. to hide relevant privacy information to avoid data leakage.

[0051] According to an embodiment of the present disclosure, the plurality of image attributes includes a kth image attribute associated with both the kth initial prompt information and the kth sample image.

[0052] According to an embodiment of the present disclosure, determining a sample object recognition result based on a sample text, multiple sample images, and multiple initial prompt information may include: performing image feature extraction on the kth sample image for the kth image attribute to obtain the kth sample image feature; performing text feature fusion on the sample text and the kth initial prompt information to obtain the kth sample text fusion feature; and determining the kth sample object recognition sub-result based on the sample image feature and the kth sample text fusion feature, the sample object recognition result including multiple sample object recognition sub-results, k≥1, and k is an integer.

[0053] Figure 3A The schematic diagram shows a principle diagram of a deep learning model according to an embodiment of the present disclosure.

[0054] like Figure 3A As shown, the deep learning model 300 may include an image feature extraction network 310, a text feature fusion network 320, and an object recognition network 330. The image feature extraction network 310 may be constructed based on a neural network algorithm, for example, the image feature extraction network 310 may be constructed based on a convolutional neural network algorithm. The text feature fusion network 320 may be constructed based on an attention network algorithm, for example, the text feature fusion network 320 may be constructed based on a Transformer algorithm. The object recognition network 330 may be constructed based on a multi-layer perceptron algorithm. The first sample image T301 to the third sample image T303 may be obtained by performing multi-type image attribute conversion on the sample images, and the first sample image T301 to the third sample image T303 may correspond to the first image attribute, the second image attribute, and the third image attribute, respectively.

[0055] like Figure 3A As shown, for example, in the case of k=1, for the kth image attribute, performing image feature extraction on the kth sample image to obtain the kth sample image feature may include inputting the first sample image T301 into the image feature extraction network 310 and outputting the first sample image feature. For another example, performing text feature fusion on the sample text and the kth initial prompt information to obtain the kth sample text fusion feature may include inputting the sample text 301 and the first initial prompt information P301 into the text feature fusion network 320 and outputting the first sample text fusion feature. For another example, determining the kth sample object recognition sub-result based on the sample image feature and the kth sample text fusion feature may include inputting the sample image feature and the first sample text fusion feature into the object recognition network 330 and outputting the first sample object recognition sub-result 3021. The first sample object recognition sub-result 3021 may include a prediction result of the object category of the sample object.

[0056] like Figure 3A As shown, for another example, when k=2, for the kth image attribute, performing image feature extraction on the kth sample image to obtain the kth sample image feature may include inputting the second sample image T302 into the image feature extraction network 310 and outputting the second sample image feature. For another example, performing text feature fusion on the sample text and the kth initial prompt information to obtain the kth sample text fusion feature may include inputting the sample text 301 and the second initial prompt information P302 into the text feature fusion network 320 and outputting the second sample text fusion feature. For another example, determining the kth sample object recognition sub-result based on the sample image feature and the kth sample text fusion feature may include inputting the sample image feature and the second sample text fusion feature into the object recognition network 330 and outputting the second sample object recognition sub-result 3022. The second sample object recognition sub-result 3022 may also include a prediction result of the object category of the sample object.

[0057] like Figure 3AAs shown, for another example, in the case of K=k=3, for the kth image attribute, performing image feature extraction on the kth sample image to obtain the kth sample image feature may include inputting the third sample image T303 into the image feature extraction network 310 and outputting the third sample image feature. For another example, performing text feature fusion on the sample text and the kth initial prompt information to obtain the kth sample text fusion feature may include inputting the sample text 301 and the third initial prompt information P303 into the text feature fusion network 320 and outputting the third sample text fusion feature. For another example, determining the kth sample object recognition sub-result based on the sample image feature and the kth sample text fusion feature may include inputting the sample image feature and the third sample text fusion feature into the object recognition network 330 and outputting the third sample object recognition sub-result 3022. The third sample object recognition sub-result 3023 may also include a prediction result of the object category of the sample object.

[0058] It should be noted that K can be any preset integer greater than 1, and the K sample images can correspond to K image attributes respectively. Those skilled in the art can set any number of image attributes according to actual needs, as long as the number of set image attributes is greater than 1.

[0059] It should be noted that Figure 3A The deep learning model 300 shown in the figure may be a pre-trained deep learning model. The deep learning model may be pre-trained based on a supervised method, an unsupervised method, or a semi-supervised method. The embodiments of the present disclosure do not limit the training method of the deep learning model.

[0060] According to an embodiment of the present disclosure, text feature fusion of sample text and kth initial prompt information may include: extracting features of the sample text to obtain sample text features; and fusing the sample text features and the kth initial prompt information based on an attention mechanism.

[0061] Figure 3B The schematic diagram shows a principle diagram of a text feature fusion network according to an embodiment of the present disclosure.

[0062] like Figure 3B As shown, the text feature fusion network 320 may include a text feature embedding layer 321 and a text encoder 322. The text feature fusion network 320 may be constructed based on a pre-trained Transformer model. The first initial prompt information P301, the second initial prompt information P302, and the third initial prompt information P303 may be represented based on an initialized feature vector.

[0063] like Figure 3BAs shown, extracting features from the sample text to obtain sample text features may include inputting the sample text 301 into the text feature embedding layer 321 and outputting a sample text feature vector. In the case of k=1, for the first image attribute, fusing the sample text features and the kth initial prompt information based on the attention mechanism may include: inputting the first initial prompt information P301 and the sample text feature vector into the text encoder 322, and outputting the first sample text fusion feature P301'. In the case of k=2, for the second image attribute, fusing the sample text features and the kth initial prompt information based on the attention mechanism may include: inputting the second initial prompt information P302 and the sample text feature vector into the text encoder 322, and outputting the second sample text fusion feature P302'. In the case of K=k=3, for the third image attribute, fusing the sample text features and the kth initial prompt information based on the attention mechanism may include: inputting the third initial prompt information P303 and the sample text feature vector into the text encoder 322, and outputting the third sample text fusion feature P303'.

[0064] According to an embodiment of the present disclosure, the image feature extraction network may include an image feature embedding layer and an image encoder. The image feature extraction network may be constructed based on a pre-trained Transformer model.

[0065] According to an embodiment of the present disclosure, updating multiple initial prompt information based on sample object recognition results and labels may include: for the kth image attribute, processing the kth sample object recognition sub-result and label according to the loss function to obtain the kth sub-loss value; and updating the kth initial prompt information based on the kth sub-loss value.

[0066] According to an embodiment of the present disclosure, the kth sample object recognition sub-result can characterize the predicted probability of the object category of the sample object in the sample image, and the kth sample object recognition sub-result and the label are processed according to the loss function, and the obtained kth sub-loss value can characterize the degree of difference between the kth sample object recognition sub-result and the true object category, and the kth initial prompt information is iteratively updated according to the kth sub-loss value. Each time the updated kth initial prompt information is obtained, the updated kth initial prompt information and the sample text, as well as the kth sample image corresponding to the kth image attribute are input into the deep learning model, and a new kth sample object recognition sub-result is output, and the kth initial prompt information is iteratively updated according to the kth sub-loss value between the new kth sample object recognition sub-result and the label, until the loss function converges, and the kth initial prompt information corresponding to the convergence of the loss function is determined as the updated kth prompt information.

[0067] It should be noted that in the process of updating the initial prompt information, the model parameters of the deep learning model can remain fixed to reduce the computational overhead caused by adjusting the parameters of the deep learning model, improve the efficiency of prompt information generation and improve the overall efficiency of object recognition.

[0068] According to the embodiments of the present disclosure, the prompt information generation method provided by the embodiments of the present disclosure aligns image features and text features through a pre-trained multimodal deep learning model, and updates the prompt information corresponding to K image attributes through sample object recognition results and labels. It can realize open set classification and recognition of sample objects through the deep learning model, and by updating the prompt information corresponding to multiple image attributes, the sample text features and the image semantic features of the sample image can be learned into the prompt information in a fine-grained manner, thereby improving the migration effect of the deep learning model for object recognition tasks and improving the object recognition capability.

[0069] According to an embodiment of the present disclosure, the image attributes may include image style attributes.

[0070] According to an embodiment of the present disclosure, performing multi-type image attribute conversion on the acquired initial sample image to obtain sample images related to multiple image attributes may include: performing style attribute conversion on the initial sample image to obtain a sample style image corresponding to the image style attribute, and the multiple sample images include the sample style image.

[0071] According to an embodiment of the present disclosure, the initial sample image can be processed based on a pre-trained style transfer model, such as a VGGNet (Visual Geometry Group Net) model, to obtain a sample style image corresponding to the image style attribute. By performing style attribute conversion on the initial sample image, the sample style image can characterize the sample object through a style corresponding to the image style attribute, so that the updated prompt information can fully learn the sample objects related to the object category and the image semantic features in different image style attributes, so that object recognition based on the prompt information can improve the object recognition accuracy for different image style attributes.

[0072] According to an embodiment of the present disclosure, the image style attribute includes at least one of the following: a sketch style attribute, a cartoon style attribute, and an oil painting style attribute.

[0073] According to the embodiments of the present disclosure, a sample style image with sketch style attributes can characterize the sample object and environmental information related to the sample object through relatively obvious lines, and can also ignore the color attributes in the image, thereby generating prompt information corresponding to the sketch style attributes. The contour details of the sample object in the sample style image can be fully learned, and the contour recognition capability between the sample object and the environment can be improved, so that object recognition can be performed according to the prompt information corresponding to the sketch style attributes, which can improve the recognition accuracy for the object category.

[0074] According to an embodiment of the present disclosure, a sample style image with cartoon style attributes can characterize the sample object through lines that are significantly different from standard sample object lines, as well as characterize environmental information related to the sample object, thereby generating prompt information corresponding to the cartoon style attributes. The semantic features of the sample object with obvious contour line changes can be learned, and object recognition can be performed based on the prompt information corresponding to the cartoon style attributes, so that accurate recognition can be performed on a variety of target objects of the same object category.

[0075] According to an embodiment of the present disclosure, a sample style image with oil painting style attributes can characterize the sample object through lines that are significantly different in color style from a standard sample object, as well as characterize environmental information related to the sample object, thereby generating prompt information corresponding to the oil painting style attributes. The semantic features of the sample object with more obvious color changes can be learned, and object recognition can be performed based on the prompt information corresponding to the oil painting style attributes, so that accurate recognition can be performed on a variety of target objects of the same object category.

[0076] According to the embodiments of the present disclosure, the image style attributes may also include other types, such as artistic style attributes such as Gothic style. The embodiments of the present disclosure do not limit the specific types of image style attributes, and those skilled in the art may select them according to actual needs.

[0077] According to an embodiment of the present disclosure, the image attributes further include image lens attributes, and the image lens attributes represent that the image attributes of the sample image match the image attributes of the image captured by the lens having the image lens attributes.

[0078] According to an embodiment of the present disclosure, performing multiple types of image attribute conversion on the acquired initial sample image to obtain sample images respectively related to multiple image attributes may include: performing lens attribute conversion on the initial sample image to obtain sample lens attribute images corresponding to the image lens attributes, wherein the multiple sample images include the sample lens attribute image.

[0079] According to the embodiments of the present disclosure, the sample lens attribute image obtained after the lens attribute conversion can represent the image captured by the lens having the image lens attribute, so that by generating prompt information corresponding to the sample lens attribute image, the prompt information can fully learn the size change information of the sample object in the image corresponding to the image lens attribute, and the ratio change law between the sample object and the environment image area in the image corresponding to the image lens attribute, so that object recognition is performed according to the prompt information corresponding to the sample lens attribute image, which can improve the object recognition ability of images captured by lenses with different image attributes and improve the generalization ability of object recognition.

[0080] According to an embodiment of the present disclosure, the image lens attribute includes at least one of the following: fisheye lens attribute, wide-angle lens attribute, but is not limited thereto, and may also include other types of image lens attributes, such as telephoto lens attribute, etc.

[0081] According to the embodiments of the present disclosure, the sample lens attribute image related to the fisheye lens attribute can make the object contour attributes such as the shape and size of the sample object in the sample lens attribute image adapt to the image contour attributes collected by the fisheye lens, so that the prompt information corresponding to the sample lens attribute image is generated, which can improve the object recognition capability for the image with fisheye lens attributes.

[0082] According to an embodiment of the present disclosure, the image attributes may also include image weather condition attributes. The weather condition attributes may include, for example, rainy conditions, snowy conditions, foggy conditions, sandy conditions, etc. The embodiments of the present disclosure do not limit the specific types of weather condition attributes, and those skilled in the art may select them according to actual needs.

[0083] According to an embodiment of the present disclosure, performing multi-type image attribute conversion on the acquired initial sample image to obtain sample images related to multiple image attributes may include: performing weather condition attribute conversion on the initial sample image to obtain a sample weather condition image corresponding to the image weather condition attribute, wherein the multiple sample images include a sample weather condition image.

[0084] According to an embodiment of the present disclosure, a sample weather condition image may include a background environment related to weather conditions, and the weather conditions may at least partially block a sample object to be identified, for example, the sample weather condition image may include a bicycle in a rainy environment. By generating a sample weather condition image corresponding to the image weather condition attribute, and further generating prompt information corresponding to the image weather condition attribute based on the sample weather condition image, the image features of the sample objects in the sample images collected under different weather conditions can be fully learned, and the recognition capability of the target objects under different weather conditions can be improved, so that the object recognition can be performed according to the prompt information corresponding to the image weather condition attribute, and the recognition accuracy rate for the object category can be improved.

[0085] According to an embodiment of the present disclosure, the image attribute may further include at least one of the following: image illumination attribute, image size attribute.

[0086] According to the embodiments of the present disclosure, the image illumination attribute can characterize the illumination conditions for the sample object in the sample image, for example, it can include the illumination intensity, illumination angle, shadow length, etc. for the sample object. By performing image attribute conversion on the image illumination attribute of the initial sample image, the sample image can characterize the sample object under various illumination conditions, so that the prompt information generated according to the sample image can further target the recognition capability of the target object under different illumination conditions, thereby improving the recognition accuracy.

[0087] According to the embodiments of the present disclosure, the image size attribute of the initial sample image can be converted by cropping, stretching, etc., to obtain a sample image corresponding to the image size attribute. By generating a sample image corresponding to the image size attribute, the updated prompt information can fully learn the contour attribute of the sample object under the condition that the size of the sample object contour in the sample image is deformed or the contour is missing, and then the object is recognized according to the prompt information corresponding to the image size attribute, which can improve the robustness and generalization ability of the target object recognition model and improve the recognition accuracy.

[0088] According to an embodiment of the present disclosure, image attributes may also include image perspective attributes, etc., and a sample image corresponding to the image perspective attributes can be obtained by performing perspective conversion on a three-dimensional sample image, thereby improving the ability of the prompt information to learn the contours of sample objects at different angles, thereby improving object recognition accuracy.

[0089] According to the embodiment of the present disclosure, before extracting image features from a sample image, the sample image may be segmented to obtain sample image patches. Each patch is input to the image feature embedding layer of the image feature extraction network, and the sample image patch code E = {e1, e2, ..., e M},in is the sample image block encoding, M is the number of patches. M sample image block encodings are input to the image encoder E of the image feature extraction network I ,

[0090] According to an embodiment of the present disclosure, the sample text can be input into the text feature embedding layer of the text feature fusion network, and the text feature embedding layer can be constructed based on the word encoder. The sample text is input into the word encoder, and the word vector T corresponding to the word in the text is outputted. N}, where t i ∈ is the word vector output by the word encoder, and N is the number of word vectors. By introducing K learnable initial prompt information (Prompt vectors), each Prompt vector corresponds to each type of image attribute, where the initial prompt information is represented as is a learnable prompt vector, L is the length of Prompt, and word vector T = {t1, t2, ..., t N} and the learnable prompt vector are input to the text encoder E of the text feature fusion network T , output the sample text features

[0091] According to an embodiment of the present disclosure, the sample texts include C pieces, the C sample texts correspond to C object categories one by one, the kth sample object recognition sub-result includes the prediction probabilities corresponding to the C object categories respectively, C is greater than 1, and C is an integer; wherein, according to the sample image features and the kth sample text fusion features, determining the kth sample object recognition sub-result may include: calculating the similarity between the kth sample text fusion features and the cth sample text fusion features to obtain the cth similarity information, the C object categories include the cth object category, and determining the prediction probability of the kth sample object recognition sub-result for the cth object category according to the similarity between the kth sample text fusion features and the cth sample text fusion features.

[0092] For example, when there are C object categories, each object category can correspond to a sample text describing the object category, and then C sample text features Y={y0, y1, ..., y C}, where y c Represents the sample text feature. For the k-th image attribute, the k-th sample object recognition sub-result can be characterized based on the prediction probability for the c-th category among the C object categories. The recognition probability of the c-th category can be determined by the following formula (1).

[0093]

[0094] In formula (1), τ is the temperature coefficient, p c The predicted probability of the k-th sample object recognition sub-result for the c-th category, sim() represents the cosine similarity. The predicted probability and label can be processed by the cross entropy loss function, and the initial prompt information can be updated according to the obtained sub-loss value to obtain the updated prompt information.

[0095] According to the embodiments of the present disclosure, after generating prompt information corresponding to each of the K image attributes, the generated prompt information can be tested based on an integration strategy. For example, the K prompt information and the test text can be fused with text features based on formula (2) to obtain the test text fusion feature.

[0096]

[0097] In formula (2), T represents the test text feature corresponding to the test text, P i represents the prompt information feature vector corresponding to the i-th prompt information, E T () represents the text encoder. Based on formula (2), the knowledge learned from different prompt information can be integrated, so as to realize the integration of the image semantic information represented by the K image attributes of the test object, and then the test text fusion feature can represent the object category center of the test object to be identified. According to the object category center and the image features of the test image, the object category of the test object can be accurately identified, thereby effectively improving the recognition accuracy of the deep learning model in identifying open set categories and effectively improving the generalization ability of the deep learning model.

[0098] An embodiment of the present disclosure also provides an object recognition method.

[0099] Figure 4 The flowchart of the object recognition method according to the embodiment of the present disclosure is schematically shown.

[0100] like Figure 4 As shown, the object recognition method may include S410 to S430.

[0101] In operation S410, a target image related to a target object, a description text for describing the target object, and prompt information related to each of a plurality of image attributes are acquired.

[0102] In operation S420, object recognition is performed according to the description text, the target image, and the plurality of prompt information to obtain a target object recognition result.

[0103] According to an embodiment of the present disclosure, the target image may include an image acquired after image acquisition of the vehicle's traffic environment, or may also include an image acquired after image acquisition of the working environment. The target image may include target objects of one or more object categories. The description text may include text information for describing the object category, and the number of the description texts may include one or more. In the case where the description text includes multiple description texts, each of the multiple description texts may be associated with a different object category.

[0104] According to an embodiment of the present disclosure, the prompt information is determined according to the prompt information generating method provided by the embodiment of the present disclosure.

[0105] According to an embodiment of the present disclosure, object recognition is performed based on description text, target image and multiple prompt information, and obtaining the target object recognition result may include inputting the description text, target image and multiple prompt information into a pre-trained deep learning model, and outputting the target object recognition result representing the object category of the target object. Since the prompt information generated by the method provided in the embodiment of the present disclosure can fully learn the object attributes of the sample object, as well as the image attributes of the sample image in multiple image attribute conditions, object recognition is performed based on the target image, multiple prompt information and description text processed by the deep learning model, so that the deep learning model can obtain the target object recognition result under the condition of learning the object attributes and diversified image attributes, thereby improving the recognition accuracy and recognition efficiency of the object category of the target object.

[0106] According to an embodiment of the present disclosure, multiple image attributes include a kth image attribute associated with a kth prompt information, k≥1, and k is an integer; wherein, target object recognition based on the description text, the target image and the multiple prompt information includes: for the kth image attribute, performing text feature fusion on the description text and the kth prompt information to obtain a kth text fusion feature corresponding to the kth image attribute; performing feature fusion on the text fusion features corresponding to each of the multiple image attributes to obtain a target text fusion feature.

[0107] According to an embodiment of the present disclosure, identifying a target object according to the description text, the target image and a plurality of prompt information may further include: extracting image features from the target image to obtain target image features.

[0108] According to an embodiment of the present disclosure, performing target object recognition based on the description text, the target image and the plurality of prompt information may further include: determining a target object recognition result based on a fusion feature of the target text and a feature of the target image.

[0109] Figure 5 The principle diagram of the object recognition method according to the embodiment of the present disclosure is schematically shown.

[0110] like Figure 5 As shown, the object recognition method can be implemented based on a deep learning model 500, and the deep learning model 500 may include an image feature extraction network 510, a text feature fusion network 520, and an object recognition network 530. The image feature extraction network 510 may be constructed based on a neural network algorithm, for example, the image feature extraction network 510 may be constructed based on a Transformer algorithm. The text feature fusion network 520 may be constructed based on an attention network algorithm, for example, the text feature fusion network 520 may be constructed based on a Transformer algorithm. The object recognition network 530 may be constructed based on a multi-layer perceptron algorithm. The target image T501 may include an image representing a target object to be recognized, and the first prompt information P501, the second prompt information P502, and the third prompt information P503 may correspond to the first image attribute, the second image attribute, and the third image attribute, respectively. The description text 501 may include text describing the object category of the target object.

[0111] like Figure 5 As shown, performing image feature extraction on the target image to obtain the target image feature may include inputting the target image T501 into an image feature extraction network 510 and outputting the target image feature.

[0112] like Figure 5 As shown, in the case of k=1, for the kth image attribute, performing text feature fusion on the description text and the kth prompt information to obtain the kth text fusion feature may include inputting the description text 501 and the first prompt information P501 into the text feature fusion network 520, and outputting the first text fusion feature. For another example, in the case of k=2, for the kth image attribute, performing text feature fusion on the description text and the kth prompt information to obtain the kth text fusion feature may include inputting the description text 501 and the second prompt information P502 into the text feature fusion network 520, and outputting the second text fusion feature. For another example, in the case of K=k=3, for the kth image attribute, performing text feature fusion on the description text and the kth initial prompt information to obtain the kth text fusion feature may include inputting the description text 501 and the third initial prompt information P503 into the text feature fusion network 520, and outputting the third text fusion feature.

[0113] like Figure 5 As shown, for another example, performing feature fusion on the text fusion features corresponding to the multiple image attributes to obtain the target text fusion feature may include: inputting the first text fusion feature, the second text fusion feature and the third text fusion feature into the text feature fusion layer of the object recognition network, and outputting the target text fusion feature. The text feature fusion layer may be constructed based on a neural network algorithm, or may be constructed based on formula (2) in the above embodiment.

[0114] According to the embodiments of the present disclosure, Figure 5 As shown, object recognition is performed according to the description text, the target image and the multiple prompt information to obtain the target object recognition result, which may also include inputting the target image features output by the image feature extraction network 510 into the object recognition network 530, so that the object recognition layer of the object recognition network processes the target image features and the target text fusion features, and outputs the target object recognition result 502. The object recognition layer may be constructed based on a multi-layer perceptron algorithm.

[0115] It should be noted that Figure 5 The image feature extraction network and text feature fusion network in the deep learning model shown may be the same as the image feature extraction network and text feature fusion network in the above-mentioned embodiment.

[0116] According to the embodiments of the present disclosure, since the target text fusion feature can be obtained by fusion of the description text and the prompt information corresponding to each of the multiple image attributes, the target text fusion feature can be used as the object category center for characterizing the target object, and the target text fusion feature can fully learn the image semantic features of the target object in multiple image attributes, so the object category of the target object is identified according to the target image feature and the target text fusion feature. Under the condition that the target object in the target image has differences due to the image attributes, the target object can be identified by comparing the target image feature and the target text fusion feature, so that the target object recognition result obtained can more accurately characterize the object category, and improve the recognition accuracy without increasing the computational overhead and computational time. At the same time, the recognition accuracy of the target object under different image acquisition environments and different image acquisition device attributes is improved, and the generalization ability of target object recognition is improved.

[0117] According to an embodiment of the present disclosure, the image attribute includes at least one of the following: image lighting attribute, image size attribute, image weather condition attribute, image lens attribute, and image style attribute.

[0118] Figure 6 A block diagram of a prompt information generating device according to an embodiment of the present disclosure is schematically shown.

[0119] like Figure 6 As shown, the prompt information generating device 600 may include: a first attribute conversion module 610 , a first acquisition module 620 , a sample object recognition result determination module 630 and a prompt information generating module 640 .

[0120] The first attribute conversion module 610 is used to perform multi-type image attribute conversion on the acquired initial sample image to obtain sample images with multiple image attributes respectively related. The initial sample image is related to the sample object to be identified, and the initial sample image is associated with a label of the object category representing the sample object.

[0121] The first acquisition module 620 is used to acquire a sample document for describing the sample object and initial prompt information related to each of a plurality of image attributes.

[0122] The sample object recognition result determination module 630 is used to determine the sample object recognition result according to the sample text, multiple sample images and multiple initial prompt information.

[0123] The prompt information generation module 640 is used to update multiple initial prompt information according to the sample object recognition results and labels, and generate multiple prompt information for object recognition.

[0124] According to an embodiment of the present disclosure, the multiple image attributes include a kth image attribute associated with both the kth initial prompt information and the kth sample image.

[0125] According to an embodiment of the present disclosure, the sample object recognition result determination module includes: a first extraction submodule, a first fusion submodule and a first determination submodule.

[0126] The first extraction submodule is used to extract image features from the kth sample image according to the kth image attribute to obtain the kth sample image feature.

[0127] The first fusion submodule is used to fuse the text features of the sample text and the k-th initial prompt information to obtain the k-th sample text fusion feature.

[0128] The first determination submodule is used to determine the kth sample object recognition subresult according to the sample image feature and the kth sample text fusion feature, the sample object recognition result includes multiple sample object recognition subresults, k≥1, and k is an integer.

[0129] According to an embodiment of the present disclosure, the prompt information generation module includes: a sub-loss value acquisition sub-module and an update sub-module.

[0130] The sub-loss value obtaining sub-module is used to process the k-th sample object recognition sub-result and label according to the loss function for the k-th image attribute to obtain the k-th sub-loss value.

[0131] The update submodule is used to update the kth initial prompt information according to the kth sub-loss value.

[0132] According to an embodiment of the present disclosure, the first fusion submodule includes: a first extraction unit and a fusion unit.

[0133] The first extraction unit is used to extract features from the sample text to obtain features of the sample text.

[0134] The fusion unit is used to fuse the sample text features and the kth initial prompt information based on the attention mechanism.

[0135] According to an embodiment of the present disclosure, the image attributes include image style attributes.

[0136] According to an embodiment of the present disclosure, the first attribute conversion module includes a first attribute conversion submodule.

[0137] The first attribute conversion submodule is used to perform style attribute conversion on the initial sample image to obtain a sample style image corresponding to the image style attribute, and the multiple sample images include the sample style image.

[0138] According to an embodiment of the present disclosure, the image style attribute includes at least one of the following: a sketch style attribute, a cartoon style attribute, and an oil painting style attribute.

[0139] According to an embodiment of the present disclosure, the image attributes further include image lens attributes, and the image lens attributes represent that the image attributes of the sample image match the image attributes of the image captured by the lens having the image lens attributes.

[0140] According to an embodiment of the present disclosure, the first attribute conversion module includes a second attribute conversion submodule.

[0141] The second attribute conversion submodule is used to perform lens attribute conversion on the initial sample image to obtain a sample lens attribute image corresponding to the image lens attribute, wherein the multiple sample images include the sample lens attribute image.

[0142] According to an embodiment of the present disclosure, the image lens attribute includes at least one of the following: a fisheye lens attribute, a wide-angle lens attribute.

[0143] According to an embodiment of the present disclosure, the image attributes further include image weather condition attributes.

[0144] According to an embodiment of the present disclosure, the first attribute conversion module includes a third attribute conversion submodule.

[0145] The third attribute conversion submodule is used to convert the weather condition attributes of the initial sample image to obtain a sample weather condition image corresponding to the weather condition attribute of the image, wherein the multiple sample images include the sample weather condition image.

[0146] In the third attribute conversion submodule, the image attribute further includes at least one of the following: image illumination attribute and image size attribute.

[0147] Figure 7The block diagram of the object recognition device according to the embodiment of the present disclosure is schematically shown.

[0148] like Figure 7 As shown, the object recognition device 700 includes: a second acquisition module 710 and a recognition module 720 .

[0149] The second acquisition module 710 is used to acquire a target image related to the target object, a description text for describing the target object, and prompt information related to each of a plurality of image attributes, wherein the prompt information is determined according to the prompt information generation method described in the above embodiment.

[0150] The recognition module 720 is used to perform object recognition according to the description text, the target image and multiple prompt information to obtain a target object recognition result.

[0151] According to an embodiment of the present disclosure, the plurality of image attributes include a kth image attribute associated with the kth prompt information, k≥1, and k is an integer.

[0152] According to an embodiment of the present disclosure, the identification module includes: a second fusion submodule and a third fusion submodule.

[0153] The second fusion submodule is used to fuse the description text and the kth prompt information with respect to the kth image attribute to obtain the kth text fusion feature corresponding to the kth image attribute.

[0154] The third fusion submodule is used to fuse the text fusion features corresponding to the multiple image attributes to obtain the target text fusion features.

[0155] According to an embodiment of the present disclosure, the image attribute includes at least one of the following: image lighting attribute, image size attribute, image weather condition attribute, image lens attribute, and image style attribute.

[0156] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an unmanned vehicle.

[0157] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0158] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0159] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0160] According to an embodiment of the present disclosure, an unmanned vehicle includes the electronic device as described in the above embodiment.

[0161] Figure 8 The application scenario diagram of the unmanned vehicle according to the embodiment of the present disclosure is schematically shown.

[0162] like Figure 8 As shown, the application scenario may include vehicles 801 and 802, wherein vehicle 801 may be equipped with an image acquisition device to acquire a target image in a traffic environment, and vehicle 801 may also be equipped with an electronic device provided by an embodiment of the present disclosure, and the electronic device includes: at least one processor and a memory connected to the at least one processor in communication. The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that at least one processor can execute the object recognition method provided by the embodiment of the present disclosure. Since the processor of the electronic device can obtain multiple prompt information according to the prompt information generation method provided by the embodiment of the present disclosure, obtain the target image containing vehicle 802, and perform object recognition according to the preset description text, the vehicle 802 can be timely identified in heavy rain weather conditions to avoid recognition errors.

[0163] Fig. 9 A block diagram of an electronic device suitable for implementing a prompt information generation method and an object recognition method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0164] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0165] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0166] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a prompt information generation method or an object recognition method. For example, in some embodiments, the prompt information generation method or the object recognition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the prompt information generation method or the object recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the prompt information generating method or the object identifying method in any other appropriate manner (eg, by means of firmware).

[0167] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0169] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0171] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0172] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0173] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0174] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating prompt information, comprising: Performing multi-type image attribute conversion on the acquired initial sample image to obtain sample images with multiple image attributes respectively associated therewith, wherein the initial sample image is associated with a sample object to be identified, the initial sample image is associated with a label representing an object category of the sample object, and the image attribute includes at least one of an image illumination attribute and a weather condition attribute; Acquire sample text for describing the sample object and initial prompt information related to each of the plurality of image attributes; Determine a sample object recognition result according to the sample text, the plurality of sample images and the plurality of initial prompt information; as well as A plurality of the initial prompt information are updated according to the sample object recognition result and the label to generate a plurality of prompt information for object recognition, at least one of the prompt information corresponds to the image illumination attribute or the weather condition attribute.

2. The method according to claim 1, wherein: The plurality of image attributes include a kth image attribute associated with both the kth initial prompt information and the kth sample image; Wherein, determining the sample object recognition result according to the sample text, the plurality of sample images and the plurality of initial prompt information comprises: According to the k-th image attribute, extract image features from the k-th sample image to obtain a k-th sample image feature; Performing text feature fusion on the sample text and the kth initial prompt information to obtain a kth sample text fusion feature; and According to the sample image feature and the kth sample text fusion feature, a kth sample object recognition sub-result is determined, the sample object recognition result includes a plurality of the sample object recognition sub-results, k≥1, and k is an integer.

3. The method according to claim 2, wherein: The updating of the plurality of initial prompt information according to the sample object recognition result and the label comprises: For the kth image attribute, processing the kth sample object recognition sub-result and the label according to a loss function to obtain a kth sub-loss value; and Update the kth initial prompt information according to the kth sub-loss value.

4. The method according to claim 2, wherein: The text feature fusion of the sample text and the kth initial prompt information comprises: Extracting features from the sample text to obtain sample text features; and The sample text features and the kth initial prompt information are fused based on the attention mechanism.

5. The method according to any one of claims 1 to 4, wherein: The image attributes include image style attributes; The step of converting the acquired initial sample image into multiple types of image attributes to obtain sample images having multiple image attributes that are respectively related to each other includes: The initial sample image is subjected to style attribute conversion to obtain a sample style image corresponding to the image style attribute, wherein the plurality of sample images include the sample style image.

6. The method according to claim 5, wherein: The image style attribute includes at least one of the following: Sketch style attributes, cartoon style attributes, oil painting style attributes.

7. The method according to any one of claims 1 to 4, wherein: The image attributes further include image lens attributes, and the image lens attributes represent that the image attributes of the sample image match the image attributes of an image captured by a lens having the image lens attributes; The step of converting the acquired initial sample image into multiple types of image attributes to obtain sample images having multiple image attributes that are respectively related to each other includes: The initial sample image is subjected to lens attribute conversion to obtain a sample lens attribute image corresponding to the image lens attribute, wherein the plurality of sample images include the sample lens attribute image.

8. The method according to claim 7, wherein: The image lens attribute includes at least one of the following: Fisheye lens properties, wide-angle lens properties.

9. The method according to any one of claims 1 to 4, in, The step of converting the acquired initial sample image into multiple types of image attributes to obtain sample images with multiple image attributes respectively related to each other includes: The initial sample image is subjected to weather condition attribute conversion to obtain a sample weather condition image corresponding to the weather condition attribute of the image, wherein the plurality of sample images include the sample weather condition image.

10. The method according to any one of claims 1 to 4, wherein: The image attributes also include an image size attribute.

11. A method for object recognition, comprising: Acquire a target image related to a target object, a description text for describing the target object, and prompt information related to each of a plurality of image attributes, wherein the prompt information is determined according to the method according to any one of claims 1 to 10; Object recognition is performed according to the description text, the target image and the plurality of prompt information to obtain a target object recognition result.

12. The method according to claim 11, wherein: The plurality of image attributes include a kth image attribute associated with the kth prompt information, k≥1, and k is an integer; The step of identifying the target object according to the description text, the plurality of target images and the plurality of prompt information includes: For the k-th image attribute, performing text feature fusion on the description text and the k-th prompt information to obtain a k-th text fusion feature corresponding to the k-th image attribute; The text fusion features corresponding to the plurality of image attributes are fused to obtain a target text fusion feature.

13. The method according to claim 11, wherein: The image attributes include at least one of the following: Image lighting attributes, image size attributes, image weather condition attributes, image lens attributes, and image style attributes.

14. A prompt information generating device, comprising: a first attribute conversion module, configured to perform multi-type image attribute conversion on the acquired initial sample image to obtain sample images with multiple image attributes, wherein the initial sample image is related to a sample object to be identified, the initial sample image is associated with a label representing an object category of the sample object, and the image attribute includes at least one of an image illumination attribute and a weather condition attribute; A first acquisition module, used for acquiring a sample text for describing the sample object and initial prompt information related to each of the plurality of image attributes; A sample object recognition result determination module, used to determine a sample object recognition result according to the sample text, the plurality of sample images and the plurality of initial prompt information; as well as A prompt information generation module is used to update the multiple initial prompt information according to the sample object recognition result and the label, and generate multiple prompt information for object recognition, at least one of which corresponds to the image lighting attribute or the weather condition attribute.

15. The device according to claim 14, wherein: The plurality of image attributes include a kth image attribute associated with both the kth initial prompt information and the kth sample image; Wherein, the sample object recognition result determination module includes: A first extraction submodule is used to extract image features from the kth sample image according to the kth image attribute to obtain a kth sample image feature; A first fusion submodule is used to fuse text features of the sample text and the k-th initial prompt information to obtain a k-th sample text fusion feature; and The first determination submodule is used to determine the kth sample object recognition subresult according to the sample image feature and the kth sample text fusion feature, the sample object recognition result includes multiple sample object recognition subresults, k≥1, and k is an integer.

16. The device according to claim 15, wherein: The prompt information generating module comprises: a sub-loss value obtaining sub-module, for processing the k-th sample object recognition sub-result and the label according to a loss function for the k-th image attribute to obtain a k-th sub-loss value; and An updating submodule is used to update the kth initial prompt information according to the kth sub-loss value.

17. The device according to claim 15, wherein: The first fusion submodule includes: A first extraction unit is used to extract features from the sample text to obtain sample text features; and A fusion unit is used to fuse the sample text features and the kth initial prompt information based on an attention mechanism.

18. The device according to any one of claims 14 to 17, wherein: The image attributes include image style attributes; Wherein, the first attribute conversion module includes: The first attribute conversion submodule is used to perform style attribute conversion on the initial sample image to obtain a sample style image corresponding to the image style attribute, and the plurality of sample images include the sample style image.

19. The device according to claim 18, wherein The image style attribute includes at least one of the following: Sketch style attributes, cartoon style attributes, oil painting style attributes.

20. The device according to any one of claims 14 to 17, wherein: The image attributes further include image lens attributes, and the image lens attributes represent that the image attributes of the sample image match the image attributes of an image captured by a lens having the image lens attributes; Wherein, the first attribute conversion module includes: The second attribute conversion submodule is used to perform lens attribute conversion on the initial sample image to obtain a sample lens attribute image corresponding to the image lens attribute, wherein the plurality of sample images include the sample lens attribute image.

21. The device according to claim 20, wherein: The image lens attribute includes at least one of the following: Fisheye lens properties, wide-angle lens properties.

22. The device according to any one of claims 14 to 17, in, The first attribute conversion module includes: The third attribute conversion submodule is used to perform weather condition attribute conversion on the initial sample image to obtain a sample weather condition image corresponding to the weather condition attribute of the image, wherein the multiple sample images include the sample weather condition image.

23. The device according to any one of claims 14 to 17, wherein: The image attributes also include an image size attribute.

24. An object recognition device, comprising: a second acquisition module, used for acquiring a target image related to a target object, a description text for describing the target object, and prompt information related to each of a plurality of image attributes, wherein the prompt information is determined according to the method according to any one of claims 1 to 10; The recognition module is used to perform object recognition according to the description text, the target image and the plurality of prompt information to obtain a target object recognition result.

25. The device according to claim 24, wherein: The plurality of image attributes include a kth image attribute associated with the kth prompt information, k≥1, and k is an integer; Wherein, the identification module includes: The second fusion submodule is used to fuse the description text and the kth prompt information with respect to the kth image attribute to obtain a kth text fusion feature corresponding to the kth image attribute; The third fusion submodule is used to perform feature fusion on the text fusion features corresponding to each of the plurality of image attributes to obtain a target text fusion feature.

26. The device according to claim 24, wherein: The image attributes include at least one of the following: Image lighting attributes, image size attributes, image weather condition attributes, image lens attributes, and image style attributes.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

30. An unmanned vehicle, comprising: The electronic device according to claim 27.

Images