Automatic driving scene image generation method, training method, device and equipment

Abstract

The disclosure provides an automatic driving scene image generation method, a training method, an apparatus and a device, relates to the technical field of artificial intelligence, and in particular to the technical field of image processing, deep learning, automatic driving and the like. The specific implementation scheme of the automatic driving scene image generation method is as follows: image features of an initial image are extracted, wherein the image features represent features corresponding to a description text of the initial image; text features of change content relative to the initial image are extracted; and the image features and the text features are processed to generate a target image; wherein the target image represents an image after the initial image is changed according to the change content.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of image processing, deep learning, and autonomous driving, specifically to methods, training methods, devices, and equipment for generating images of autonomous driving scenarios. Background Technology

[0002] Autonomous driving technology refers to the technology that can perceive the driving environment around a vehicle and automatically plan a driving route for the vehicle without human intervention, thus achieving autonomous driving.

[0003] Because real-world driving environments are complex and varied, autonomous driving systems need to learn how to avoid risks and reduce collisions in diverse driving scenarios. Therefore, to improve the risk avoidance performance of autonomous driving systems, a large number of diverse driving scenario images are required for the systems to learn from. Summary of the Invention

[0004] This disclosure provides a method, training method, apparatus, and device for generating images of autonomous driving scenarios.

[0005] According to one aspect of this disclosure, an autonomous driving scene image generation method is provided, comprising: extracting image features of an initial image, wherein the image features represent features corresponding to descriptive text of the initial image; extracting text features of changes relative to the initial image; and processing the image features and text features to generate a target image; wherein the target image represents an image after the initial image has been modified according to the changed content.

[0006] According to another aspect of this disclosure, a training method for a deep learning model is provided. The deep learning model includes a feature extraction module and a feature processing module. The training method includes: using the feature extraction module to extract sample image features of an initial sample image; wherein the sample image features represent features corresponding to descriptive text of the initial sample image; using the feature extraction module to extract sample text features representing sample changes relative to the initial sample image; using the feature processing module to process the sample image features and sample text features to generate a target sample image, wherein the target sample image represents an image after the initial sample image has been modified according to the sample image changes; based on a loss function, obtaining a first loss value based on the initial sample image, sample text features, and target sample image; and based on the first loss value, fixing the parameters of the feature extraction module and adjusting the parameters of the feature processing module to obtain a trained deep learning model.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods described above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This illustration schematically shows an exemplary system architecture of a training method and apparatus for generating autonomous driving scene images or deep learning models, which can be applied according to embodiments of the present disclosure.

[0013] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically;

[0014] Figure 3a A schematic diagram of initial image 300a according to an embodiment of the present disclosure is shown;

[0015] Figure 3b This schematically illustrates a target image after the initial image 300a has been modified according to the changes made according to an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of an image generation method according to an embodiment of the present disclosure is shown;

[0017] Figure 5 A schematic diagram of an image generation method according to another embodiment of the present disclosure is shown;

[0018] Figure 6 A flowchart illustrating a method for training a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0019] Figure 7 A schematic diagram illustrating a training method for a deep learning model according to an embodiment of the present disclosure is shown.

[0020] Figure 8 The diagram illustrates a training method for a feature extraction module of a deep learning model according to an embodiment of the present disclosure.

[0021] Figure 9 A block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0022] Figure 10 A block diagram schematically illustrates a training apparatus for a deep learning model according to an embodiment of the present disclosure; and

[0023] Figure 11 The diagram illustrates a block diagram of an electronic device suitable for implementing an autonomous driving scene image generation method or a deep learning model training method according to embodiments of the present disclosure. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] The diversity of driving scenarios is a crucial factor in determining whether autonomous driving models can accurately avoid driving risks in real-world applications. Current technologies typically use hard-coding to generate obstacles on map coordinates within specific driving scenarios. However, this method requires writing code scripts and generates relatively homogeneous driving scenarios, making it difficult for autonomous driving models to learn how to accurately avoid collisions and other driving risks in real-world applications. Therefore, there is an urgent need for a simpler and more flexible method to generate driving scenario images to obtain diverse driving scenarios.

[0026] In view of this, embodiments of this disclosure provide a method for generating autonomous driving scene images. By extracting image features from an initial image and text features representing changes to the initial image, the method processes the image features and text features to obtain a target image after modifying the initial image according to the modified content. This method allows for changes to the content of the initial image based on the text representing the changes relative to the initial image, generating an image with modified content. This simplifies the operation of generating driving scene images and allows for the production of diverse driving scene images.

[0027] Figure 1The illustration schematically shows an exemplary system architecture of an autonomous driving scene image generation method or a deep learning model training method and apparatus that can be applied according to embodiments of the present disclosure.

[0028] It is important to note that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture for applying the autonomous driving scene image generation method or the deep learning model training method and apparatus may include a terminal device. However, the terminal device can implement the autonomous driving scene image generation method or the deep learning model training method and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide a communication link between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0031] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0032] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0033] It should be noted that the autonomous driving scene image generation method or deep learning model training method provided in this disclosure embodiment can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the autonomous driving scene image generation method or deep learning model training device provided in this disclosure embodiment can also be disposed in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0034] Alternatively, the autonomous driving scene image generation method or deep learning model training method provided in this disclosure embodiment can generally also be executed by server 105. Correspondingly, the autonomous driving scene image generation method or deep learning model training device provided in this disclosure embodiment can generally be located in server 105. The autonomous driving scene image generation method or deep learning model training method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the autonomous driving scene image generation method or deep learning model training device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0036] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0037] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0038] Figure 2 A flowchart illustrating an autonomous driving scene image generation method according to an embodiment of the present disclosure is shown schematically.

[0039] like Figure 2 As shown, the method includes operations S210 to S230.

[0040] In operation S210, image features of the initial image are extracted, wherein the image features represent features corresponding to the descriptive text of the initial image.

[0041] In operation S220, text features of the changed content relative to the initial image are extracted.

[0042] In operation S230, image features and text features are processed to generate a target image; wherein, the target image represents the image after the initial image has been modified according to the changed content.

[0043] According to embodiments of this disclosure, the initial image may include an image of the autonomous vehicle and a lane line image of the map area where the autonomous vehicle is located. It may also include other static and dynamic obstacles besides the autonomous vehicle. Static obstacles may include roadblocks, traffic signs, trees, etc. Dynamic obstacles may include other vehicles traveling in the same or different lanes as the autonomous vehicle.

[0044] According to embodiments of this disclosure, image features can characterize features corresponding to descriptive text of an initial image. The descriptive text of the initial image can be descriptive text describing the content of the initial image.

[0045] Figure 3a The illustration shows a schematic diagram of an initial image according to an embodiment of the present disclosure.

[0046] like Figure 3a As shown in the initial image 300a, the autonomous vehicle Ta301 is traveling in the lane between lane line L1 and lane line L2. Vehicles in the same lane as autonomous vehicle Ta301 include vehicle Tb302, which is following autonomous vehicle Ta301. Vehicle Tc303 is traveling in the lane between lane line L1 and the guardrail L0.

[0047] According to embodiments of this disclosure, the descriptive text of the initial image 300 may be "A vehicle is traveling in the same lane as the main vehicle, and another vehicle is traveling in the lane to the left of the main vehicle, following behind it." Depending on the needs of the actual application scenario, the descriptive text of the initial image may also include other driving status information such as vehicle speed and direction. "Main vehicle" may refer to an autonomous driving vehicle.

[0048] According to embodiments of this disclosure, the changes relative to the initial image can represent the intended text for changing the content of the initial image. The changes can be specific to a particular lane, such as: "Add two vehicles with a speed of 30 km / h to the right lane of the main vehicle," or "Delete vehicles from the left lane of the main vehicle." The changes can also be based on risk levels, such as: "Add two obstacle vehicles of level one risk," or "Delete all obstacle vehicles of level two risk."

[0049] According to embodiments of this disclosure, the changes categorized by risk level can be pre-configured. For example, a vehicle obstructing a vehicle at risk level 1 can be "a vehicle located in the same lane as the main vehicle and traveling in the opposite direction." A vehicle obstructing a vehicle at risk level 2 can be "a vehicle located in the adjacent lane as the main vehicle and traveling in the same direction."

[0050] It should be noted that changes relative to the initial image can include not only dynamic obstacles but also static obstacles. For example: "Add a roadblock 30 meters ahead of the main vehicle, etc."

[0051] According to embodiments of this disclosure, a target image is generated by processing image features and text features. The target image can represent an image after the initial image has been modified according to the changed content.

[0052] For example, the change could be: "Two new motor vehicles with a speed of 30 km / h have been added to the right lane of the main vehicle lane."

[0053] Figure 3b The illustration shows a target image after the initial image 300a has been modified according to the changes made according to an embodiment of the present disclosure.

[0054] like Figure 3b As shown, in target image 300b, the autonomous vehicle Ta301 is traveling in the lane between lane line L1 and lane line L2. Vehicles in the same lane as autonomous vehicle Ta301 include vehicle Tb302, which is following autonomous vehicle Ta301. Vehicle Tc303 is traveling in the lane between lane line L1 and the guardrail L0. Vehicle Td304 has been added to the lane between lane lines L2 and L3 to the right of autonomous vehicle Ta301. Vehicle Te305 has been added to the lane between lane lines L3 and L4 to the right of autonomous vehicle Ta301.

[0055] According to embodiments of this disclosure, by extracting image features of an initial image and text features representing changes to the initial image, and processing the image features and text features, a target image is obtained after modifying the initial image according to the modified content. The content of the initial image can be modified based on the text representing the changes to the initial image, generating an image with modified content. This simplifies the operation of generating driving scene images and allows for the generation of diverse driving scene images.

[0056] According to embodiments of this disclosure, operation S210 may include the following operations: encoding an initial image to obtain multiple image-encoded features; obtaining a first matching probability between the multiple image-encoded features and multiple text features describing the text based on a predetermined feature distribution map; and obtaining image features from the multiple image-encoded features based on the first matching probability.

[0057] According to embodiments of this disclosure, a predetermined feature distribution map can characterize the matching probability between image-coded features and textual features of the descriptive text of an initial image.

[0058] For example, the first predetermined feature distribution map may include: the matching probability of image coding feature IFa with text feature TFa of the description text of the initial image is 0.96, the matching probability of image coding feature IFb with text feature TFb of the description text of the initial image is 0.83, and the matching probability of image coding feature IFc with text feature TFc of the description text of the initial image is 0.92.

[0059] According to embodiments of this disclosure, image features can be obtained from multiple image coding features by setting a first predetermined matching probability threshold.

[0060] For example, the first predetermined matching probability threshold can be 0.9. Image features whose matching probability with the text features of the initial image's descriptive text is greater than the predetermined matching probability threshold, such as image-coded features IFa, are identified as image features. That is, the image features are image-coded features IFa and IFc.

[0061] According to embodiments of this disclosure, by selecting image features that match the textual features of the descriptive text of the initial image through a first predetermined feature distribution map, the accuracy of information in the fused features can be improved.

[0062] According to an embodiment of this disclosure, the above operation S220 may include the following operations: encoding the changed content to obtain multiple text encoding features; obtaining a second matching probability between the multiple text encoding features and multiple image features of the changed content according to a predetermined feature distribution map; and obtaining text features from the multiple text encoding features according to the second matching probability.

[0063] According to embodiments of this disclosure, a second predetermined feature distribution map can characterize the matching probability between text encoding features and image features of changed content.

[0064] For example, the second predetermined feature distribution map may include: a matching probability of 0.93 between the text encoding feature TFd and the image feature IFd of the changed content; a matching probability of 0.98 between the text encoding feature TFe and the image feature IFe of the changed content; and a matching probability of 0.95 between the text encoding feature TFf and the image feature IFf of the changed content.

[0065] According to embodiments of this disclosure, text features can be obtained from multiple text encoding features by setting a second predetermined matching probability threshold.

[0066] For example, the second predetermined matching probability threshold can be 0.95. Text-encoded features with a matching probability greater than or equal to 0.95 between the text-encoded features and the image features of the changed content can be identified as text features. That is, the text features are text-encoded features TFe and text-encoded features TFf.

[0067] It should be noted that the first predetermined matching probability threshold and the second predetermined matching probability threshold can be the same or different, and this is not limited here.

[0068] According to embodiments of this disclosure, by selecting text features that match the image features of the changed content through a second predetermined feature distribution map, the accuracy of information in the fused features can be improved.

[0069] According to an embodiment of this disclosure, the above operation S230 may include: fusing image features and text features to obtain fused features; and processing the fused features to generate a target image.

[0070] According to embodiments of this disclosure, image features and text features can be fused based on an attention mechanism to obtain fused features.

[0071] For example, image features are used as the query matrix, and text features are used as the key and value matrices. Based on the cross-attention mechanism, the image features and text features are fused to obtain the fused features.

[0072] According to embodiments of this disclosure, the fusion features may include content information in the target image after the initial image has been modified according to the changed content.

[0073] For example, the initial image can include the main vehicle and its position information on the road. The change could be "two new vehicles added to the right lane of the main vehicle". In this case, the fused features can include at least the main vehicle, the two newly added vehicles, the main vehicle's position information on the road, and the position information of the two newly added vehicles on the road.

[0074] Figure 4A schematic diagram of an image generation method according to an embodiment of the present disclosure is shown.

[0075] like Figure 4 As shown, in embodiment 400, an initial image 401 is encoded to obtain multiple image encoding features 402. Based on a first matching probability in a first predetermined feature distribution map, image features 403 are obtained from the multiple image encoding features 402. Changes 404 relative to the initial image are encoded to obtain multiple text encoding features 405. Based on a second matching probability in a second predetermined feature distribution map, text features 406 are obtained from the multiple text encoding features 405. Based on an attention mechanism, image features 403 and text features 406 are fused to obtain fused features 407. Processing fused features 407 yields a target image 408.

[0076] Figure 5 A schematic diagram of an image generation method according to another embodiment of the present disclosure is shown.

[0077] like Figure 5 As shown, in embodiment 500, the initial image 501 is input to the image feature extraction module 502, which outputs the image features of the initial image. The change content relative to the initial image, "two vehicles have been added to the right lane of the main vehicle" 503, is input to the text feature module 504, which outputs the text features of the change content. The image features of the initial image and the text features of the change content are input together to the attention module 505, which outputs the fused features. The fused features are input to the decoding module 506 to obtain the target image 507. Compared with the initial image 501, the target image 507 shows that two vehicles have been added to the right lane of the main vehicle (the vehicle marked T in the figure).

[0078] According to embodiments of this disclosure, based on an attention mechanism, changes to image content guided by textual features of the changed content can be achieved, enabling changes to the content of driving scene images based on language text, thereby generating diverse driving scene images.

[0079] Figure 6 A flowchart illustrating a method for training a deep learning model according to an embodiment of the present disclosure is shown.

[0080] like Figure 6 As shown, the method includes operations S610 to S650.

[0081] When operating S610, the feature extraction module is used to extract the sample image features of the initial sample image.

[0082] In operation S620, the feature extraction module is used to extract sample text features of sample changes relative to the initial sample image.

[0083] When operating the S630, the feature processing module is used to process the features of the sample image and the sample text to generate the target sample image.

[0084] In operation S640, based on the loss function, the first loss value is obtained according to the initial sample image, sample text features and target sample image.

[0085] When operating the S650, based on the first loss value, the parameters of the feature extraction module are fixed, and the parameters of the feature processing module are adjusted to obtain the trained deep learning model.

[0086] According to the embodiments of this disclosure, the initial sample image, the sample change content, and the target sample image have the same definition scope as the initial image, the change content, and the target image described above, and will not be repeated here.

[0087] According to embodiments of this disclosure, the sample image features represent features corresponding to the descriptive text of the initial sample image. The target sample image represents the image after the initial sample image has been modified according to the sample image modification content.

[0088] According to embodiments of this disclosure, the loss function may be a cross-entropy loss function or a cosine embedding loss function, and no specific limitation is made herein.

[0089] According to embodiments of this disclosure, the feature extraction module can be pre-trained. Therefore, during training, the parameters of the feature extraction module are not adjusted; only the parameters of the feature processing module are adjusted to obtain the trained deep learning model.

[0090] According to embodiments of this disclosure, since the features extracted by the pre-trained feature extraction module correspond to the descriptive text of the initial sample image, they can be effectively fused with the text features of the changed sample content to generate a target image that modifies the initial sample image according to the changed sample content. This model realizes the process of modifying image content based on the changed text, simplifying the operation of generating driving scene images. By stimulating the potential of the model, diverse driving scenarios can be generated.

[0091] According to embodiments of this disclosure, the feature extraction module may include an image encoding module and a text encoding module. The feature extraction module is pre-trained, and the training method for the feature extraction module may include: extracting sample image features from a sample image using the image encoding module; extracting sample text features from sample text used to describe the sample image using the text encoding module; obtaining a second loss value based on a loss function, according to the sample image features and sample text features; and adjusting the parameters of the text encoding module and the image encoding module based on the second loss value to obtain the trained feature extraction module and a sample feature distribution map.

[0092] According to embodiments of this disclosure, multiple sample images and multiple sample texts describing the sample images can be combined to form multiple sample pairs based on contrastive learning, and a sample matrix can be constructed. In the sample matrix, the elements on the diagonal represent positive sample pairs, and the other elements represent negative sample pairs. A positive sample pair indicates that the image content of the sample image in the sample pair corresponds to the text content of the sample text. A negative sample pair indicates that the image content of the sample image in the sample pair does not correspond to the text content of the sample text.

[0093] According to embodiments of this disclosure, by processing the above-mentioned positive sample pairs and negative sample pairs using an image encoding module and a text encoding module, the matching probability between the image features and text features of the positive sample pairs and the matching probability between the image features and text features of the negative sample pairs can be obtained.

[0094] According to embodiments of this disclosure, based on a loss function, a second loss value is calculated according to the matching probability between image features and text features of positive sample pairs and the matching probability between image features and text features of negative sample pairs. Based on this second loss value, the parameters of the text encoding module and the image encoding module are adjusted until a predetermined convergence condition is met, namely: the matching probability between image features and text features of positive sample pairs is much greater than the matching probability between image features and text features of negative sample pairs. This yields the trained feature extraction module.

[0095] According to embodiments of this disclosure, the predetermined convergence condition may be that the matching probability between image features and text features of a positive sample pair converges or is greater than or equal to a certain predetermined threshold, and the matching probability between image features and text features of a negative sample pair converges or is less than or equal to a certain predetermined threshold. The convergence condition can be set according to actual application requirements, and is not specifically limited herein.

[0096] According to embodiments of this disclosure, the sample feature distribution map can be a sample feature distribution map between multiple image features and multiple text features of the descriptive text of the image content.

[0097] According to embodiments of this disclosure, the feature extraction module obtained through comparative learning can extract image features corresponding to the image description text using the image encoding module, and extract image features corresponding to the image associated with the text using the text encoding module, so as to achieve the technical effect of generating diverse scene images based on changes to image content based on language text.

[0098] The feature extraction module can be used not only to extract image features and text features of the changed content, but also to extract image features of the target sample image, so as to verify whether the content of the target sample image is consistent with the content of the initial image after the changes are made according to the changed content.

[0099] According to embodiments of this disclosure, a first loss value is obtained based on a loss function, according to an initial sample image, sample text features, and a target sample image. This may include the following operations: extracting target sample image features from the target sample image and text features from the descriptive text of the initial image; concatenating the text features from the descriptive text and the sample text features from the sample changed content to obtain target sample text features; and obtaining a first loss value based on the loss function, according to the target sample image features and the target sample text features.

[0100] According to embodiments of this disclosure, target sample image features can be extracted using an image encoding module. Text features of the descriptive text of the initial sample image can be extracted using a text encoding module.

[0101] According to embodiments of this disclosure, target sample text features for describing a target sample image can be obtained by concatenating the text features of the descriptive text and the sample text features of the sample change content.

[0102] For example, the text features of the descriptive text of the initial sample image can represent the main vehicle and its location information on the road. The sample text features of the changed content can represent the changed vehicle and its location information on the road. By concatenating the text features of the descriptive text and the sample text features of the changed content, the resulting target sample text features can represent the main vehicle information, the main vehicle's location information on the road, the changed vehicle information, and the changed vehicle's location information on the road.

[0103] Since the feature extraction module is pre-trained, the first loss value can be calculated using the target sample image features and target sample text features extracted by the image encoding module, such as cross-entropy loss or cosine embedding loss. If the first loss value does not meet the predetermined convergence condition, the parameters of the feature processing module need to be adjusted until the first loss value converges, thus obtaining the trained deep learning model.

[0104] According to embodiments of this disclosure, the model loss is calculated based on the initial sample image, sample text features, and target sample image, making the image content in the target sample image closer to the content of the image after the initial image has been modified according to the modified content, thereby improving the accuracy of the model.

[0105] According to embodiments of this disclosure, extracting sample image features from an initial sample image may include the following operations: encoding the initial sample image to obtain multiple sample image encoded features; obtaining a third matching probability between the multiple sample image encoded features and multiple sample text features of the descriptive text based on a first sample feature distribution map; and obtaining sample image features from the multiple sample image encoded features based on the third matching probability.

[0106] For example, the first sample feature distribution map may include: the matching probability of sample image encoding feature IF1 with sample text feature TF1 of the description text of the initial sample image is 0.96, and the matching probability of sample image encoding feature IF2 with sample text feature TF2 of the description text of the initial sample image is 0.83. Sample image feature IF1 can be obtained from multiple sample image encoding features by setting a predetermined matching probability threshold, for example, 0.9.

[0107] According to embodiments of this disclosure, extracting sample text features of sample change content relative to an initial sample image may include the following operations: encoding the sample change content to obtain multiple sample text encoding features; obtaining a fourth matching probability between the multiple sample text encoding features and multiple sample image features of the sample change content based on a second sample feature distribution map; and obtaining sample text features from the multiple sample text encoding features based on the fourth matching probability.

[0108] For example, the second sample feature distribution map may include: the matching probability of sample text encoding feature TF3 with the image feature IF3 of the image of the changed content is 0.95. The matching probability of sample text encoding feature TF4 with the image feature IF4 of the image of the changed content is 0.85. Sample text feature TF3 can be obtained from multiple sample text encoding features by setting a predetermined matching probability threshold, for example, 0.9.

[0109] According to embodiments of this disclosure, the process of extracting sample image features of the initial sample image and extracting sample text features of sample changes relative to the initial sample image is the same as the process of extracting image features of the initial image and extracting text features of changes relative to the initial image described above, and will not be repeated here.

[0110] According to embodiments of this disclosure, the image features extracted from the initial sample image by the pre-trained feature extraction module have a high probability of matching with the text features of the descriptive text, and the extracted text features of the changed content have a high probability of matching with the image features of the changed content. Therefore, it is possible to achieve a high degree of matching between the image content in the generated target image and the changed content.

[0111] According to embodiments of this disclosure, processing sample image features and sample text features to generate a sample target image may include the following operations: fusing sample image features and sample text features to obtain sample fused features; and processing the sample fused features to obtain the sample target image.

[0112] According to embodiments of this disclosure, sample image features and sample text features can be fused based on an attention mechanism to obtain sample fused features.

[0113] For example, sample image features can be used as a sample query matrix, and sample text features can be used as a sample key matrix and a sample value matrix.

[0114] According to the embodiments of this disclosure, the sample fusion features, attention mechanism, sample query matrix, sample key matrix, and sample value matrix have the same definition range as the fusion features, attention mechanism, query matrix, key matrix, and value matrix described above, and will not be repeated here.

[0115] According to embodiments of this disclosure, by fusing sample image features and sample text features based on an attention mechanism, the content features of the changed image guided by the text features can be obtained, thereby obtaining the changed sample target image that modifies the initial sample image according to the changed sample content.

[0116] Figure 7 The diagram illustrates a method for training a deep learning model according to an embodiment of the present disclosure.

[0117] like Figure 7 As shown, in embodiment 700, the initial sample image 701 is input into the image encoding module 703 to obtain sample image features. The sample change content 702 relative to the initial sample image is input into the text encoding module 704 to obtain sample text features. The sample image features and sample text features are input into the feature processing module 705 to generate the target sample image 706.

[0118] Then, the target sample image 706 is input into the image encoding module 703, which outputs the image features 707 of the target sample image. The initial sample image 701 and the sample changes 702 relative to the initial sample image are input into the text encoding module 704, which outputs the text features 708 of the target sample. Based on the loss function, a first loss value 709 is obtained according to the image features 707 and the text features 708 of the target sample image. The parameters of the feature processing module 705 are then adjusted based on the first loss value 709 to obtain the trained deep learning model.

[0119] In this embodiment of the disclosure, the training process of the deep learning model also includes a pre-training process of the feature extraction module.

[0120] Figure 8 The diagram illustrates a training method for a feature extraction module of a deep learning model according to an embodiment of the present disclosure.

[0121] like Figure 8As shown, sample image 801 is input into image encoding module 803, which outputs sample image features 805. Sample text 802, used to describe the sample image, is input into text encoding module 804, which outputs sample text features 806. Based on the loss function, a second loss value 807 is obtained according to the sample image features 805 and the sample text features 806. Based on the second loss value 807, the parameters of image encoding module 803 and text encoding module 804 are adjusted until the second loss value converges, thus obtaining the pre-trained feature extraction module.

[0122] Figure 9 A block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0123] like Figure 9 As shown, the image generation device 900 may include: a first extraction module 910, a second extraction module 920, and a generation module 930.

[0124] The first extraction module 910 is used to extract image features from the initial image, wherein the image features represent features corresponding to the descriptive text of the initial image.

[0125] The second extraction module 920 is used to extract text features of the changed content relative to the initial image.

[0126] The generation module 930 is used to process image features and text features to generate a target image; wherein, the target image represents the image after the initial image has been modified according to the changed content.

[0127] According to embodiments of this disclosure, the first extraction module includes: a first image encoding submodule, a first acquisition submodule, and a second acquisition submodule.

[0128] The first image encoding submodule is used to encode the initial image to obtain multiple image encoding features. The first acquisition submodule is used to obtain a first matching probability between the multiple image encoding features and multiple text features describing the text, based on a predetermined feature distribution map. The second acquisition submodule is used to obtain image features from the multiple image encoding features based on the first matching probability.

[0129] According to embodiments of this disclosure, the second extraction module includes: a first text encoding submodule, a third obtaining submodule, and a fourth obtaining submodule. The first text encoding submodule is used to encode the changed content to obtain multiple text encoding features. The third obtaining submodule is used to obtain a second matching probability between the multiple text encoding features and multiple image features of the changed content based on a predetermined feature distribution map. The fourth obtaining submodule is used to obtain text features from the multiple text encoding features based on the second matching probability.

[0130] According to embodiments of this disclosure, the generation module includes a first feature fusion submodule and a first generation submodule. The first feature fusion submodule is used to fuse image features and text features to obtain fused features. The first generation submodule is used to process the fused features to generate a target image.

[0131] According to an embodiment of this disclosure, the first feature fusion submodule includes: a first feature fusion unit, used to fuse image features and text features based on an attention mechanism to obtain fused features, wherein the image features are used as a query matrix and the text features are used as a key matrix and a value matrix.

[0132] Figure 10 A block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0133] like Figure 10 As shown, the training device for the deep learning model may include: a feature extraction module 1010, a feature processing module 1020, a loss calculation module 1030, and an adjustment module 1040.

[0134] The feature extraction module 1010 is used to extract sample image features of the initial sample image; wherein, the sample image features represent the features corresponding to the descriptive text of the initial sample image; and extract sample text features of the sample changes relative to the initial sample image.

[0135] The feature processing module 1020 is used to process the features of the sample image and the features of the sample text to generate a target sample image, wherein the target sample image represents the image after the initial sample image has been modified according to the changes made to the sample image.

[0136] The loss calculation module 1030 is used to obtain the first loss value based on the loss function, the initial sample image, the sample text features, and the target sample image.

[0137] The adjustment module 1040 is used to fix the parameters of the feature extraction module and adjust the parameters of the feature processing module based on the first loss value to obtain the trained deep learning model.

[0138] According to embodiments of this disclosure, the loss calculation module includes a feature extraction submodule, a concatenation submodule, and a loss calculation submodule. The feature extraction submodule is used to extract target sample image features from the target sample image and text features from the descriptive text of the initial image. The concatenation submodule is used to concatenate the text features of the descriptive text and the sample text features of the changed sample content to obtain target sample text features. The loss calculation submodule is used to obtain a first loss value based on a loss function, according to the target sample image features and the target sample text features.

[0139] According to embodiments of this disclosure, the feature extraction module includes an image encoding module and a text encoding module. The apparatus also includes a training module for training the feature extraction module.

[0140] According to embodiments of this disclosure, the training module includes: an image feature extraction submodule, a text feature extraction submodule, a loss calculation submodule, and an adjustment submodule. The image feature extraction submodule is used to extract sample image features from the sample image using an image encoding module. The text feature extraction submodule is used to extract sample text features from the sample text describing the sample image using a text encoding module. The loss calculation submodule is used to obtain a second loss value based on a loss function, according to the sample image features and sample text features. The adjustment submodule is used to adjust the parameters of the text encoding module and the image encoding module based on the second loss value, to obtain a trained feature extraction module and a sample feature distribution map.

[0141] According to embodiments of this disclosure, the feature extraction module includes: a second image encoding submodule, a fifth acquisition submodule, and a sixth acquisition submodule. The second image encoding submodule is used to encode an initial sample image to obtain multiple sample image encoded features. The fifth acquisition submodule is used to obtain a third matching probability between the multiple sample image encoded features and multiple sample text features describing the text, based on a sample feature distribution map. The sixth acquisition submodule is used to obtain sample image features from the multiple sample image encoded features based on the third matching probability.

[0142] According to embodiments of this disclosure, the feature extraction module includes: a second text encoding submodule, a seventh obtaining submodule, and an eighth obtaining submodule. The second text encoding submodule is used to encode the sample change content to obtain multiple sample text encoding features. The seventh obtaining submodule is used to obtain a fourth matching probability between the multiple sample text encoding features and multiple sample image features of the sample change content based on a sample feature distribution map. The eighth obtaining submodule is used to obtain sample text features from the multiple sample text encoding features based on the fourth matching probability.

[0143] According to embodiments of this disclosure, the feature processing module includes a second feature fusion submodule and a feature processing submodule. The second feature fusion submodule is used to fuse sample image features and sample text features to obtain sample fused features. The feature processing submodule is used to process the sample fused features to obtain a sample target image.

[0144] According to an embodiment of this disclosure, the second feature fusion submodule includes: a second feature fusion unit, used to fuse sample image features and sample text features based on an attention mechanism to obtain sample fusion features, wherein the sample image features are used as a sample query matrix, and the sample text features are used as a sample key matrix and a sample value matrix.

[0145] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0146] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0147] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0148] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0149] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0151] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as image generation methods or deep learning model training methods. For example, in some embodiments, the image generation method or deep learning model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the image generation method or deep learning model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform an image generation method or a training method for a deep learning model.

[0153] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0159] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating images of an autonomous driving scene, comprising: The initial image is encoded to obtain multiple image encoding features; Based on the first predetermined feature distribution map, a first matching probability is obtained between the plurality of image coding features and the plurality of text features of the descriptive text of the initial image; the first predetermined feature distribution map represents the matching probability between the image coding features and the text features of the descriptive text of the initial image. Based on the first matching probability, image features of an initial image are obtained from the plurality of image coding features, wherein the image features represent features corresponding to the descriptive text of the initial image; The changes to the initial image are encoded to obtain multiple text encoding features; Based on the second predetermined feature distribution map, a second matching probability is obtained between multiple text encoding features and multiple image features of the changed content; wherein, the second predetermined feature distribution represents the matching probability between the text encoding features and the image features of the changed content; Based on the second matching probability, text features representing the changed content relative to the initial image are obtained from the plurality of text encoding features; and The image features and the text features are processed to generate a target image; wherein the target image represents the image after the initial image has been modified according to the modified content.

2. The method according to claim 1, wherein, The process of processing the image features and the text features to generate the target image includes: The image features and the text features are fused to obtain fused features; The fusion features are processed to generate the target image.

3. The method according to claim 2, wherein, The process of fusing the image features and the text features to obtain fused features includes: Based on the attention mechanism, the image features and the text features are fused to obtain the fused features, wherein the image features are used as a query matrix and the text features are used as a key matrix and a value matrix.

4. A training method for a deep learning model, the deep learning model including a feature extraction module and a feature processing module, the training method comprising: The initial sample image is encoded using the feature extraction module to obtain multiple sample image encoded features; Based on the first sample feature distribution map, a third matching probability is obtained between the multiple sample image encoding features and the multiple sample text features of the descriptive text of the initial sample image; based on the third matching probability, the sample image features are obtained from the multiple sample image encoding features; wherein, the sample image features represent features corresponding to the descriptive text of the initial sample image; the first sample feature distribution map represents the matching probability between the sample image encoding features and the sample text features of the descriptive text of the initial sample image; Using the feature extraction module, the sample changes in the initial image are encoded to obtain multiple sample text encoding features; based on the second sample feature distribution map, a fourth matching probability is obtained between the multiple sample text encoding features and multiple sample image features of the sample changes; based on the fourth matching probability, the sample text features are obtained from the multiple sample text encoding features; the sample feature distribution map includes a first sample feature distribution map and a second sample feature distribution map, the sample feature distribution map representing the sample feature distribution between multiple image features and multiple text features of the descriptive text of the image content; the second sample feature distribution map represents the matching probability between the sample text encoding features and the sample image features of the sample changes; The feature processing module is used to process the features of the sample image and the features of the sample text to generate a target sample image, wherein the target sample image represents the image after the initial sample image has been modified according to the changes made to the sample image. Based on the loss function, a first loss value is obtained according to the initial sample image, the sample text features, and the target sample image; and Based on the first loss value, the parameters of the feature extraction module are fixed, and the parameters of the feature processing module are adjusted to obtain a trained deep learning model.

5. The method according to claim 4, wherein, The first loss value is obtained based on the loss function, according to the initial sample image, the sample text features, and the target sample image, including: Extract the target sample image features from the target sample image and the text features from the descriptive text of the initial sample image; The text features of the descriptive text and the sample text features of the sample changed content are concatenated to obtain the target sample text features; and Based on the loss function, the first loss value is obtained according to the target sample image features and the target sample text features.

6. The method according to claim 5, wherein, The feature extraction module includes an image encoding module and a text encoding module, and the training method for the feature extraction module includes: The image encoding module is used to extract sample image features from the sample images; The text encoding module is used to extract sample text features used to describe the sample image. Based on the loss function, a second loss value is obtained according to the sample image features and the sample text features; and Based on the second loss value, the parameters of the text encoding module and the image encoding module are adjusted to obtain a trained feature extraction module and a sample feature distribution map.

7. The method according to claim 5, wherein, The process of processing the sample image features and the sample text features to generate the target sample image includes: The sample image features and the sample text features are fused to obtain sample fused features; and The sample fusion features are processed to obtain the target sample image.

8. The method according to claim 7, wherein, The process of fusing the sample image features and the sample text features to obtain sample fused features includes: Based on the attention mechanism, the sample image features and the sample text features are fused to obtain the sample fusion features, wherein the sample image features are used as the sample query matrix, and the sample text features are used as the sample key matrix and the sample value matrix.

9. An image generation apparatus, comprising: A first extraction module is used to encode an initial image to obtain multiple image encoding features; according to a first predetermined feature distribution map, to obtain a first matching probability between the multiple image encoding features and multiple text features of the descriptive text of the initial image; according to the first matching probability, to obtain image features of the initial image from the multiple image encoding features, wherein the image features represent features corresponding to the descriptive text of the initial image; the first predetermined feature distribution map represents the matching probability between the image encoding features and the text features of the descriptive text of the initial image. The second extraction module is used to encode the changed content of the initial image to obtain multiple text encoding features; according to a second predetermined feature distribution map, to obtain a second matching probability between the multiple text encoding features and multiple image features of the changed content; and according to the second matching probability, to obtain text features from the multiple text encoding features; the second predetermined feature distribution map represents the matching probability between the text encoding features and the image features of the changed content; and The generation module is used to process the image features and the text features to generate a target image; wherein the target image represents the image after the initial image has been modified according to the modified content.

10. The apparatus according to claim 9, wherein, The generation module includes: The first feature fusion submodule is used to fuse the image features and the text features to obtain fused features; and The first generation submodule is used to process the fused features to generate the target image.

11. The apparatus according to claim 10, wherein, The first feature fusion submodule includes: The first feature fusion unit is used to fuse the image features and the text features based on an attention mechanism to obtain the fused features, wherein the image features are used as a query matrix and the text features are used as a key matrix and a value matrix.

12. A training apparatus for a deep learning model, the deep learning model comprising a feature extraction module and a feature processing module, the apparatus comprising: A feature extraction module is used to encode an initial sample image to obtain multiple sample image encoded features; based on a first sample feature distribution map, to obtain a third matching probability between the multiple sample image encoded features and multiple sample text features of the descriptive text of the initial sample image; based on the third matching probability, to obtain the sample image feature from the multiple sample image encoded features; wherein, the sample image feature represents a feature corresponding to the descriptive text of the initial sample image; the first sample feature distribution map represents the matching probability between the sample image encoded features and the sample text features of the descriptive text of the initial sample image; and to encode sample changes to the initial image to obtain multiple sample text encoded features; based on a second sample feature distribution map, to obtain a fourth matching probability between the multiple sample text encoded features and multiple sample image features of the sample changes; based on the fourth matching probability, to obtain the sample text feature from the multiple sample text encoded features; the sample feature distribution map includes the first sample feature distribution map and the second sample feature distribution map, the sample feature distribution map representing the sample feature distribution between multiple image features and multiple text features of the descriptive text of the image content; the second sample feature distribution map representing the matching probability between the sample text encoded features and the sample image features of the sample changes; The feature processing module is used to process the sample image features and the sample text features to generate a target sample image, wherein the target sample image represents the image after the initial sample image has been modified according to the sample image modification content; The loss calculation module is used to obtain a first loss value based on the initial sample image, the sample text features, and the target sample image, according to a loss function; and An adjustment module is used to fix the parameters of the feature extraction module and adjust the parameters of the feature processing module based on the first loss value to obtain a trained deep learning model.

13. The apparatus according to claim 12, wherein, The loss calculation module includes: The feature extraction submodule is used to extract the target sample image features of the target sample image and the text features of the descriptive text of the initial sample image; The concatenation submodule is used to concatenate the text features of the descriptive text and the sample text features of the sample changed content to obtain the target sample text features; and The loss calculation submodule is used to obtain the first loss value based on the loss function, according to the target sample image features and the target sample text features.

14. The apparatus according to claim 12, wherein, The feature extraction module includes an image encoding module and a text encoding module. The deep learning model further includes a training module for training the feature extraction module, the training module including: The image feature extraction submodule is used to extract sample image features from the sample image using the image encoding module; The text feature extraction submodule is used to extract sample text features for describing the sample image using the text encoding module; A loss calculation submodule is used to obtain a second loss value based on the loss function, according to the sample image features and the sample text features; and An adjustment submodule is used to adjust the parameters of the text encoding module and the image encoding module based on the second loss value, so as to obtain a trained feature extraction module and a sample feature distribution map.

15. The apparatus according to claim 12, wherein, The feature processing module includes: The second feature fusion submodule is used to fuse the sample image features and the sample text features to obtain sample fused features; and The feature processing submodule is used to process the sample fusion features to obtain the target sample image.

16. The apparatus according to claim 15, wherein, The second feature fusion submodule includes: The second feature fusion unit is used to fuse the sample image features and the sample text features based on an attention mechanism to obtain the sample fusion features, wherein the sample image features are used as a sample query matrix, and the sample text features are used as a sample key matrix and a sample value matrix.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images