Virtual image generation method and device, electronic equipment and storage medium

Abstract

The invention provides a virtual image generation method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to AIGC content generation and other scenes based on artificial intelligence. According to the specific implementation scheme, the method comprises the steps of obtaining a mask image and three-dimensional modeling information for a target vehicle in response to received style description information for the target vehicle; each mask region in the mask image represents a position for adding a material image in the initial texture image for the target vehicle; extracting an initial texture image and a mapping relation between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image from the three-dimensional modeling information; generating a target texture image by processing the style description information, the initial texture image and the mask image; and based on the mapping relationship, generating a target virtual image by processing the target texture image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Specifically, it relates to a method, device, electronic device and storage medium for generating a virtual image. Background Art

[0002] AIGC (Artificial Intelligence Generated Content) is a technology based on artificial intelligence techniques such as generative adversarial networks and large pre-trained models. It generates relevant content with appropriate generalization capabilities through learning and identifying existing data. For example, it is used to generate personalized guidance signs. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for generating a virtual image.

[0004] According to one aspect of the present disclosure, a method for generating a virtual image is provided, comprising: in response to received style description information for a target vehicle, obtaining a mask image and three-dimensional modeling information for the target vehicle; each mask area in the mask image represents a position for adding a material image in an initial texture image for the target vehicle; the material image is matched with the style description information; the initial texture image and the mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image are extracted from the three-dimensional modeling information; a target texture image is generated by processing the style description information, the initial texture image and the mask image; and based on the mapping relationship, a target virtual image is generated by processing the target texture image.

[0005] According to another aspect of the present disclosure, a device for generating a virtual image is provided, comprising: an acquisition module, a first extraction module, a first generation module and a second generation module.

[0006] An acquisition module is used to obtain a mask image and three-dimensional modeling information for the target vehicle in response to the received style description information for the target vehicle; each mask area in the mask image represents a position for adding a material image in an initial texture image for the target vehicle; and the material image is matched with the style description information.

[0007] The first extraction module is used to extract the initial texture image and the mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image from the three-dimensional modeling information.

[0008] The first generation module is used to generate a target texture image by processing style description information, an initial texture image and a mask image.

[0009] The second generating module is used to generate a target virtual image by processing the target texture image based on the mapping relationship.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 An exemplary system architecture to which the method and device for generating a virtual image according to an embodiment of the present disclosure can be applied is schematically shown;

[0016] Figure 2 A flowchart of a method for generating a virtual image according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3A A schematic diagram schematically shows a method of generating a target virtual image at a predetermined position based on a single material image according to an embodiment of the present disclosure;

[0018] Figure 3B A schematic diagram schematically shows a method of generating a target virtual image based on matching map positions of multiple material images according to another embodiment of the present disclosure;

[0019] Figure 3C A schematic diagram schematically shows a method of generating a target virtual image based on matching mapping positions of a custom image and multiple material images according to an embodiment of the present disclosure;

[0020] Figure 4AA schematic diagram schematically shows a method of modifying an image uploaded by a user based on modification requirements to generate a custom image according to an embodiment of the present disclosure;

[0021] Figure 4B Schematically illustrates a schematic diagram of modifying an image uploaded by a user to generate a custom image based on modification requirements according to another embodiment of the present disclosure;

[0022] Figure 5 A block diagram schematically shows a virtual image generating device according to an embodiment of the present disclosure; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a method for generating a virtual image according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Electronic navigation applications make travel more convenient for users. As the number of users using electronic navigation increases, different users have also put forward various personalized requirements for guidance signs in the electronic navigation interface. At present, when users use electronic maps for navigation, the user's location is usually marked with dots or arrows on the navigation interface. This type of mark is too simple and similar in shape to other marks in the navigation interface, which can easily cause visual confusion and affect the user's perception. Therefore, there is an urgent need for guidance signs that can meet the personalized needs of different users to improve the user experience.

[0026] Therefore, an embodiment of the present invention provides a method for generating a virtual image for a navigation map. By generating a 3D virtual image, it not only meets the user's personalized needs for guidance signs in the navigation interface, but also improves the fun of electronic navigation applications and improves user experience.

[0027] Figure 1 An exemplary system architecture to which the method and apparatus for generating a virtual image according to an embodiment of the present disclosure can be applied is schematically shown.

[0028] It should be noted that Figure 1The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the method and apparatus for generating a virtual image can be applied may include a terminal device, but the terminal device may implement the method and apparatus for generating a virtual image provided by the embodiments of the present disclosure without interacting with a server.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as electronic navigation applications, knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0031] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0032] The server 103 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the terminal device 101. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0033] It should be noted that the method for generating a virtual image provided in the embodiment of the present disclosure can generally be executed by the terminal device 101. Accordingly, the apparatus for generating a virtual image provided in the embodiment of the present disclosure can also be set in the terminal device 101.

[0034] Alternatively, the method for generating a virtual image provided in the embodiment of the present disclosure may also be generally executed by the server 103. Accordingly, the device for generating a virtual image provided in the embodiment of the present disclosure may generally be arranged in the server 103. The method for generating a virtual image provided in the embodiment of the present disclosure may also be executed by a server or a server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103. Accordingly, the device for generating a virtual image provided in the embodiment of the present disclosure may also be arranged in a server or a server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103.

[0035] For example, the user may input style description information to the terminal device 101 through voice or text, for example: "Please generate a cute 3D car logo of the XX model" 110, and the terminal device 101 may send the style description information to the server 103. The server 103 may obtain the mask image and the material image based on the style description information, and generate the target virtual image 120 based on the method provided in the embodiment of the present invention, and send it to the terminal device 101.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0038] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information. The images used in the examples are all from public image data sources.

[0039] Figure 2 The flowchart of the method for generating a virtual image according to an embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0041] In operation S210 , in response to the received style description information for the target vehicle, a mask image and three-dimensional modeling information for the target vehicle are acquired.

[0042] In operation S220, an initial texture image and a mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image are extracted from the three-dimensional modeling information.

[0043] In operation S230, a target texture image is generated by processing the style description information, the initial texture image, and the mask image.

[0044] In operation S240, a target virtual image is generated by processing the target texture image based on the mapping relationship.

[0045] According to an embodiment of the present disclosure, the style description information may be voice information or text information. The content of the style description information includes but is not limited to the appearance color and overall style of the target virtual image of the target vehicle, such as simple style, cute style, black, etc.

[0046] According to an embodiment of the present disclosure, the target vehicle may be a vehicle that the user is currently driving, or may be a dream vehicle that the user wants to drive. The target virtual image may be used to be displayed in a navigation map.

[0047] According to the embodiments of the present disclosure, for material images, a material library including material images of various styles can be pre-configured, and material images matching the style description information can be determined from the material library. For example, the style description information can be cute style, and the matched material images can be various cartoon images.

[0048] In some embodiments, a plurality of material images matching the style description information may be recommended to the user first, and the final material image may be determined through interaction with the user.

[0049] In some embodiments, material images may be recommended to the user based on the user's historical preferences in combination with the style description information, thereby further improving the user's interactive experience.

[0050] According to the embodiments of the present disclosure, the initial texture image can be style modified through a simple style description, which can reduce the input pressure of the user and further improve the user experience.

[0051] According to an embodiment of the present disclosure, a target texture image can be generated by calling a generative model to process style description information, an initial texture image, and a mask image. For example, the generative model can be a generative adversarial network or a diffusion model. This embodiment of the present disclosure does not specifically limit this.

[0052] According to an embodiment of the present disclosure, each mask region in the mask image represents a position for adding a material image to an initial texture image for a target vehicle.

[0053] According to an embodiment of the present disclosure, the three-dimensional modeling information may include an initial texture image and three-dimensional mesh information for constructing a 3D virtual image of a target vehicle. The three-dimensional mesh information represents the mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image.

[0054] In related examples, 2D images of multiple perspectives are usually collected for the target vehicle, and then a 3D virtual image is reconstructed based on the 2D images of multiple perspectives. This method causes the 3D inconsistency of the texture of the 3D virtual image due to the deviation of the camera pose in the generation process of the 2D images of each perspective, for example, the camera pose of the same object corresponding to the front view and the left view is different.

[0055] However, in the embodiment of the present invention, since a position for adding a material image is reserved in the mask image, the two-dimensional coordinates in the target texture image generated after adding the material image to the initial texture image and the three-dimensional coordinates in the virtual image still satisfy the initial mapping relationship. Therefore, based on the mapping relationship, the texture in the target virtual image generated by processing the target texture image has 3D consistency.

[0056] At the same time, by adding material images that match the style description information to the initial texture image, it not only meets the user's personalized needs, but also reduces the user's input pressure, further improving the user experience.

[0057] Reference below Figure 3A to Figure 3C , Figure 4A~Figure 4B , combined with specific embodiments Figure 2 The method shown is further explained.

[0058] Figure 3A The schematic diagram schematically shows a method of generating a target virtual image at a predetermined position based on a single material image according to an embodiment of the present disclosure.

[0059] like Figure 3A As shown, in embodiment 300A, the style description information for the target vehicle input by the user may be "please generate a 3D car logo of a cute XX model" 301. "Style: cute style" 302 and "model: XX" 303 may be extracted from the style description information.

[0060] According to an embodiment of the present disclosure, generating a target texture image by processing style description information, an initial texture image and a mask image may include the following operations: determining a target material image according to the style description information; determining a target position for adding the target material image in the initial texture image based on a mask area in the mask image; and adding the target material image to the target position in the initial texture image by calling a generation model to generate a target texture image.

[0061] For example, first, based on “style: cute style” 302 , a material image 304A may be acquired.

[0062] According to an embodiment of the present disclosure, determining a target material image based on style description information may include the following operations: extracting a first keyword from the style description information; and determining a target material image from candidate material images based on a matching degree between the first keyword and a subject word of the candidate material image.

[0063] For example, the first keyword may be "cute", and the subject words of the candidate material images may be "cartoon", "cute animals", etc. The target material image may represent an image whose matching degree between the subject words of the candidate material images and the first keyword is greater than a predetermined matching degree threshold.

[0064] Next, based on "Car Type: XX" 303, the initial texture image 305A and the mask image 306A for the XX car type can be obtained. The initial texture image 305A may include the appearance texture images of various parts of the vehicle, such as the body, windows, wheels, etc., and may also include a hand image for generating a virtual driver. The row / column position corresponding to the identifier "TTTT" in the mask image 306A indicates the position where the material image 304A is added in the initial texture image 305A. For example, this position may correspond to the position of the dotted box identifier in the initial texture image.

[0065] Next, the diffusion model may be called to add the material image 304A to the initial texture image 305A according to the position indicated in the mask image 306A to generate a target texture image 307A.

[0066] Finally, based on the mapping relationship 308 between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image, the target texture image 307A is mapped to the initial three-dimensional grid to generate a target virtual image 309A.

[0067] According to the embodiments of the present disclosure, by screening material images based on the degree of matching between style keywords and subject words of the material images, the matching efficiency of the material images can be improved, thereby improving the generation efficiency of the target virtual image.

[0068] According to an embodiment of the present disclosure, a target texture image is generated under the guidance of a target position in a mask image by calling a generation model. Since the initial texture image reserves an adding position for the material image at the target position corresponding to the mask image, the initial mapping relationship between each two-dimensional coordinate in the target texture image and each three-dimensional coordinate in the virtual image still satisfies the initial mapping relationship, thereby improving the 3D texture consistency of the target texture image.

[0069] In some embodiments, the user may not only want to add a virtual character in the driving position, but may also want to decorate the exterior parts of the vehicle, such as the hood, windows, wheels, etc.

[0070] Therefore, for a scene with multiple material images and multiple mask areas, the correspondence between each target material image and each mask area can be generated according to the attributes of each target material image and the attributes of each mask area; and based on the correspondence, the target positions of each target material image can be determined.

[0071] According to an embodiment of the present disclosure, the attribute of each target material image may be the image size, and the attribute of each mask area may be the mask area size. For example, the region matching degree may be calculated based on the image size of each target material image and the size of each mask area. The region matching degree may be a percentage of the image area to the mask area area, or may be the difference between the image area and the mask area area. The embodiment of the present disclosure does not specifically limit the calculation method of the region matching degree, as long as the difference between the image area and the mask area area can be characterized. The size of the target material image that matches the mask area may be the material image that is closest to the size of the mask area.

[0072] According to an embodiment of the present disclosure, the attribute of each target material image may also be the location where the image is added. For example, for a material image of an animal, the location where the image is added may be preconfigured in the material library as the hood. Then, the mask area corresponding to the hood location in the mask image may be determined as corresponding to the material image of the animal.

[0073] Figure 3B The diagram schematically shows a schematic diagram of generating a target virtual image based on matching map positions of multiple material images according to another embodiment of the present disclosure.

[0074] like Figure 3B As shown, in embodiment 300B, the style description information input by the user is the same as that in embodiment 300A. The difference is that the candidate material images 310 obtained based on "Style: Cute" 302 may include 7 images, which can be displayed to the user through interaction with the user, for example, through a visual interface, and the user selects and determines the target material image 304B through voice input or click operation, which may include a tiger image, a unicorn image, and a baby elephant image.

[0075] The mask image 306B of the embodiment 300B indicates the positions of each image in the target material image 304B to be added to the initial texture image 305B. For example, the mask area marked with "MM" is used to add a tiger image, the mask area marked with "Y" is used to add a unicorn image, and the mask area marked with "P" is used to add a baby elephant image.

[0076] By calling the generation model, the target material image 304B, the initial texture image 305B and the mask image 306B are processed to generate a target texture image 307B in which each target material image has been added to each mask area. Finally, based on the mapping relationship 308, the target texture image 307B is mapped to the initial three-dimensional grid to generate a target virtual image 309B.

[0077] According to the embodiments of the present disclosure, by matching the attributes of the mask area with the attributes of each target material image, various virtual images can be generated flexibly and variably, and the fun can be increased through interaction with the user, further improving the user experience.

[0078] In addition, the embodiment of the present invention can also generate a target virtual image based on a user-defined material image. For example, according to the content of each target material image, the mask areas corresponding to each target material image are determined according to a predetermined correspondence relationship; and according to the attributes of each mask area, the attributes of each target material image are correspondingly adjusted.

[0079] Figure 3C The schematic diagram schematically shows a method of generating a target virtual image based on matching mapping positions of a custom image and multiple material images according to an embodiment of the present disclosure.

[0080] like Figure 3C As shown, in this embodiment 300C, the style description information is the same as that in the embodiment 300A and the embodiment 300B, and the obtained initial texture image 305B and the mask image 306B are the same as those in the embodiment 300B. The difference is that in addition to the pre-configured images in the material library, the candidate material image 320 also includes user-defined images. The user-defined image can be generated by the user after beautification and modification through any image processing software. The user-defined image can be stored in the material library, or it can be uploaded by the user during the interaction with the user.

[0081] According to an embodiment of the present disclosure, the content of each target material image may be the content of a user-defined image, such as a head image. The predetermined correspondence may represent the correspondence between the material content and the mask area, such as a head image corresponding to a mask area in the mask image for representing the driver's position. For example, the mask area marked "MM" in the mask image 306B.

[0082] Since the size or orientation of the user-defined image may not match the mask area, the properties of the user-defined image may be adjusted according to the properties of each mask area. For example, the size of the user-defined image may be adjusted to be slightly smaller than the size of the mask area used to characterize the driver's position. The orientation of the user-defined image may be adjusted to be the same as the orientation of the mask area used to characterize the driver's position. For example, the positions of key points such as hair, ears, and nose may be finely identified in the mask area, and the orientation and / or size of the user-defined image may be adjusted so that the key points of the user-defined image are in the same position as the key points of the mask area.

[0083] By calling the generation model, the target material image 304C, the initial texture image 305B and the mask image 306B are processed to generate a target texture image 307C in which each target material image is added to each mask area. Finally, based on the mapping relationship 308, the target texture image 307C is mapped to the initial three-dimensional grid to generate a target virtual image 309C.

[0084] According to an embodiment of the present disclosure, the properties of each target material image are adjusted based on the mask area properties, thereby improving the consistency of each key position in the target texture image and the reserved position in the initial texture image, thereby improving the consistency and aesthetics of the 3D texture of the target virtual image and enhancing the user viewing experience.

[0085] For application scenarios in which users want to generate 3D car logos through customized images, an embodiment of the present invention can perform semantic segmentation on the initial image to generate a first image for the target part; and in response to the received modification requirement information for the initial image, generate a material image by processing the first image based on the modification requirement information.

[0086] According to an embodiment of the present disclosure, the initial image may be a user's own photo, or a user's pet photo, etc. The target part may be user-defined, or may be determined based on the position that the user wants to add to the initial texture image. For example, the target part may be the head at the reserved position of the driver added to the initial texture image. The target part may be the foot of a pet at the reserved position of the wheel hub added to the initial texture image.

[0087] For example, the image segmentation model can be called to implement semantic segmentation of the initial image, or key points of the initial image can be detected before semantic segmentation to determine the positions of key points of the target parts, such as hair, ears, nose, etc. The image segmentation model can be any model that can implement image segmentation, and the present disclosure does not make specific limitations on this.

[0088] Figure 4AA schematic diagram schematically illustrates a method of modifying an image uploaded by a user based on modification requirements to generate a customized image according to an embodiment of the present disclosure.

[0089] like Figure 4A As shown, the image segmentation model is used to perform semantic segmentation on the initial image 401 to obtain a semantic segmentation map 402 , and the head image 403 is extracted from the initial image 401 according to the semantic segmentation map 402 .

[0090] According to an embodiment of the present disclosure, the modification requirement information may be a modification style, such as cool style, or may be specific to a modification part and a modification object, such as adding a hairpin in the shape of a cat ear to the hair. The modification requirement information "please add cartoon modification" 404 and the head image 403 are input into the diffusion model to generate a material image 405 decorated with cartoon accessories.

[0091] For example, for a target modification style, a first modification material image and a first position for adding the first modification material image in a first image may be determined according to the target modification style; and the first material image is generated by adding the first modification material image at the first position.

[0092] According to an embodiment of the present disclosure, the modification material image and the adding position corresponding to the target modification style may be pre-configured. The adding position may be a pixel coordinate. For example, for the modification style of "cute style", the modification material image may be a cartoon ornament such as "red cheek" or "little red flower", and the adding position may be, for example, "face", "hair" or "ear".

[0093] Finally, the diffusion model may be used to add the first modified material image to the first image at the first position, so as to generate the first material image with a "red cheek" and a "little red flower" on the hair.

[0094] According to the embodiments of the present disclosure, the initial image is modified based on the modification style, which reduces the input pressure of the user and improves the user experience for those who prefer efficiency.

[0095] For example: in response to the modification requirements of a part to be modified and a modification object for the part to be modified, the second position of the part to be modified can be extracted from the first image; according to the modification style, a second modification material image corresponding to the modification object can be determined; and a second material image can be generated by adding the second modification material image at the second position.

[0096] For example, the part to be modified may be "hair", and the modification object for the part to be modified may be "hat". The second position may be the pixel coordinates of the "hair" area in the first image. According to the modification style, for example, "cute style", the second modified material image may be "a furry hat image with rabbit ears".

[0097] Finally, the second modified material image may be added to the first image at the second position using the diffusion model to generate a second material image with a “furry hat with rabbit ears” on the hair.

[0098] According to the embodiments of the present disclosure, the initial image is modified based on the position to be modified and the specific modifying object, which increases user participation and improves the user experience for users who prefer to pursue fun and aesthetics.

[0099] For users who require both efficiency and fun, candidate materials can be provided to users based on modification styles first, and then the candidate materials can be modified again based on interaction with the users.

[0100] In actual applications, when a user is immersed in the interactive selection of various material images, a scenario may occur in which the modification style of the user-defined image conflicts with the modification style of the target virtual image.

[0101] Figure 4B A schematic diagram schematically illustrates modifying an image uploaded by a user based on modification requirements to generate a customized image according to another embodiment of the present disclosure.

[0102] like Figure 4B As shown, in embodiment 400B, the process of extracting head image 403 through semantic segmentation based on initial image 401 is the same as that in embodiment 400A. In this embodiment 400B, style conflict detection between modification style requirement "please add cartoon modification" 404 and style description information "please generate a cute 3D logo of XX model" is added.

[0103] By executing S410, it is determined whether there is a conflict between the modification style requirement and the style description information. For example, a first keyword may be extracted from the style description information; a second keyword may be extracted from the target modification style; and in response to the matching degree between the first keyword and the second keyword being less than a predetermined threshold, warning information for characterizing the style conflict is generated.

[0104] For example, the first keyword may be "cute style" and the second keyword may be "cartoon". Since cartoon images usually represent a cute style, it can be determined that there is no style conflict. The diffusion model can be called to process the modification requirement information "please add cartoon modification" 404 and the head image 403 to generate a material image 405 decorated with cartoon ornaments.

[0105] When it is determined by executing S410 that there is a conflict between the modification style requirement and the style description information, a style conflict warning 406 may be sent to the user.

[0106] For example, the first keyword in the style description information is "cute style", and the second keyword extracted from the modification requirement information is "bat ornaments". Generally, bat ornaments represent "dark style" and do not match "cute style". Therefore, a style conflict warning may be sent to the user to prompt the user to change the modification requirement information. The style conflict warning information may be presented through a visual interface or through voice broadcast, which is not specifically limited in the embodiments of the present disclosure.

[0107] According to the embodiments of the present disclosure, through style conflict verification, the interaction with the user can be increased during the process of generating the target virtual image. At the same time, the probability of generating virtual images with very different styles is reduced, thereby improving user satisfaction.

[0108] Figure 5 A block diagram of a device for generating a virtual image according to an embodiment of the present disclosure is schematically shown.

[0109] like Figure 5 As shown, the apparatus 500 may include an acquisition module 510 , a first extraction module 520 , a first generation module 530 , and a second generation module 540 .

[0110] The acquisition module 510 is used to obtain a mask image and three-dimensional modeling information for the target vehicle in response to the received style description information for the target vehicle; each mask area in the mask image represents a position for adding a material image in the initial texture image for the target vehicle; the material image is matched with the style description information.

[0111] The first extraction module 520 is used to extract the initial texture image and the mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image from the three-dimensional modeling information.

[0112] The first generating module 530 is used to generate a target texture image by processing the style description information, the initial texture image and the mask image.

[0113] The second generating module 540 is used to generate a target virtual image by processing the target texture image based on the mapping relationship.

[0114] According to an embodiment of the present disclosure, the first generating module includes: a first determining submodule, a second determining submodule and a first generating submodule.

[0115] The first determination submodule is used to determine the target material image according to the style description information. The second determination submodule is used to determine the target position of adding the target material image in the initial texture image based on the mask area in the mask image. The first generation submodule is used to add the target material image to the target position in the initial texture image by calling the generation model to generate the target texture image.

[0116] According to an embodiment of the present disclosure, the target material image includes a plurality of images; the mask area includes a plurality of images; and the second determination submodule includes: a first generation unit and a first determination unit.

[0117] The first generating unit is used to generate the corresponding relationship between each target material image and each mask area according to the attribute of each target material image and the attribute of each mask area. The first determining unit is used to determine each target position of each target material image based on the corresponding relationship.

[0118] According to an embodiment of the present disclosure, the target material image includes multiple ones; the mask area includes multiple ones; the first generating module further includes: a third determining submodule and an adjusting submodule.

[0119] The third determination submodule is used to determine the mask areas corresponding to the target material images according to the content of the target material images and the predetermined corresponding relationship, wherein the predetermined corresponding relationship represents the corresponding relationship between the material content and the mask area. The adjustment submodule is used to adjust the attributes of the target material images according to the attributes of the mask areas.

[0120] According to an embodiment of the present disclosure, the first determination submodule includes: an extraction unit and a second determination unit. The extraction unit is used to extract the first keyword from the style description information. The second determination unit is used to determine the target material image from the candidate material images based on the matching degree between the first keyword and the subject word of the candidate material image.

[0121] According to an embodiment of the present disclosure, the device further includes: a segmentation module and a processing module. The segmentation module is used to perform semantic segmentation on the initial image to generate a first image for the target part. The processing module is used to generate a material image by processing the first image based on the modification requirement information received for the initial image.

[0122] According to an embodiment of the present disclosure, the modification requirement information includes a target modification style; the processing module includes: a fourth determination submodule and a second generation submodule. The fourth determination submodule is used to determine a first modification material image and a first position for adding the first modification material image in the first image according to the target modification style. The second generation submodule is used to generate the first material image by adding the first modification material image at the first position.

[0123] According to an embodiment of the present disclosure, the demand information also includes a part to be modified and a modification object for the part to be modified; the processing module also includes: an extraction submodule and a fifth determination submodule. The extraction submodule is used to extract the second position of the part to be modified from the first image. The fifth determination submodule is used to determine the second modification material image corresponding to the modification object according to the modification style. The third generation submodule is used to generate the second material image by adding the second modification material image at the second position.

[0124] According to an embodiment of the present disclosure, the device further includes: a second extraction module, a third extraction module and a third generation module.

[0125] The second extraction module is used to extract the first keyword from the style description information. The third extraction module is used to extract the second keyword from the target modification style. The third generation module is used to generate warning information for characterizing style conflict in response to the matching degree between the first keyword and the second keyword being less than a predetermined threshold.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0127] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0128] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0129] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0130] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0131] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0132] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the method for generating a virtual image. For example, in some embodiments, the method for generating a virtual image may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for generating a virtual image described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method for generating a virtual image in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0139] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a virtual image, comprising: In response to the received style description information for the target vehicle, obtaining a mask image and three-dimensional modeling information for the target vehicle; Each mask region in the mask image represents a location for adding a material image to an initial texture image for a target vehicle; the material image matches the style description information; Extracting the initial texture image and the mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image from the three-dimensional modeling information; Generate a target texture image by processing the style description information, the initial texture image and the mask image; as well as Based on the mapping relationship, a target virtual image is generated by processing the target texture image.

2. The method according to claim 1, wherein: The step of generating a target texture image by processing the style description information, the initial texture image and the mask image comprises: Determining a target material image according to the style description information; Determining a target position for adding the target material image in the initial texture image based on a mask area in the mask image; and The target texture image is generated by calling the generation model and adding the target material image to the target position in the initial texture image.

3. The method according to claim 2, wherein: The target material images include multiple ones; the mask areas include multiple ones; The step of determining a target position for adding the target material image to the initial texture image based on the mask area in the mask image includes: generating a correspondence between each of the target material images and each of the mask regions according to the attributes of each of the target material images and the attributes of each of the mask regions; and Based on the corresponding relationship, each target position of each target material image is determined.

4. The method according to claim 2, wherein: The target material images include a plurality of images; the mask areas include a plurality of areas; and the method further includes: According to the content of each target material image, determine each mask area corresponding to each target material image according to a predetermined corresponding relationship; wherein the predetermined corresponding relationship represents the corresponding relationship between the material content and the mask area; According to the attributes of each of the mask regions, the attributes of each of the target material images are adjusted accordingly.

5. The method according to any one of claims 2 to 4, wherein: The step of determining the target material image according to the style description information includes: Extracting a first keyword from the style description information; and The target material image is determined from the candidate material images based on the matching degree between the first keyword and the subject word of the candidate material image.

6. The method according to claim 1, wherein: The method further comprises: Performing semantic segmentation on the initial image to generate a first image for the target part; and In response to the received modification requirement information for the initial image, the material image is generated by processing the first image based on the modification requirement information.

7. The method according to claim 6, wherein: The modification requirement information includes a target modification style; The generating the material image by processing the first image based on the modification requirement information in response to the received modification requirement information for the initial image includes: determining, according to the target modification style, a first modification material image and a first position for adding the first modification material image in the first image; and A first material image is generated by adding the first modified material image at the first position.

8. The method according to claim 7, wherein: The modification requirement information also includes a part to be modified and a modification object for the part to be modified; The generating the material image by processing the first image based on the modification requirement information in response to the received modification requirement information for the initial image includes: Extracting a second position of the part to be modified from the first image; Determining a second modified material image corresponding to the modified object according to the modified style; A second material image is generated by adding the second modified material image at the second position.

9. The method according to claim 7, further comprising: extracting the first keyword from the style description information; extracting a second keyword from the target modification style; In response to the matching degree between the first keyword and the second keyword being less than a predetermined threshold, warning information for characterizing a style conflict is generated.

10. The method according to any one of claims 1 to 9, wherein: The target virtual image is used for display in the navigation map.

11. A device for generating a virtual image, comprising: An acquisition module, configured to acquire a mask image and three-dimensional modeling information for the target vehicle in response to the received style description information for the target vehicle; Each mask region in the mask image represents a location for adding a material image to an initial texture image for a target vehicle; the material image matches the style description information; A first extraction module, configured to extract the initial texture image and a mapping relationship between each two-dimensional coordinate in the initial texture image and each three-dimensional coordinate in the virtual image from the three-dimensional modeling information; A first generating module, configured to generate a target texture image by processing the style description information, the initial texture image and the mask image; as well as The second generating module is used to generate a target virtual image by processing the target texture image based on the mapping relationship.

12. The device according to claim 11, wherein The first generation module includes: A first determination submodule, configured to determine a target material image according to the style description information; A second determining submodule is configured to determine a target position for adding the target material image in the initial texture image based on a mask area in the mask image; and The first generation submodule is used to add the target material image to the target position in the initial texture image by calling the generation model to generate the target texture image.

13. The device according to claim 12, wherein: The target material images include multiple ones; the mask areas include multiple ones; the second determination submodule includes: A first generating unit, configured to generate a correspondence between each of the target material images and each of the mask regions according to an attribute of each of the target material images and an attribute of each of the mask regions; and The first determining unit determines each target position of each target material image based on the corresponding relationship.

14. The device according to claim 12, wherein: The target material images include multiple ones; the mask areas include multiple ones; the first generation module also includes: A third determination submodule is used to determine, according to the content of each target material image, each mask area corresponding to each target material image according to a predetermined corresponding relationship; wherein the predetermined corresponding relationship represents the corresponding relationship between the material content and the mask area; The adjustment submodule is used to adjust the attributes of each of the target material images according to the attributes of each of the mask regions.

15. The device according to any one of claims 12 to 14, wherein: The first determining submodule includes: an extraction unit, configured to extract a first keyword from the style description information; and The second determining unit is configured to determine the target material image from the candidate material images based on a matching degree between the first keyword and a subject word of the candidate material images.

16. The device according to claim 11, wherein The device also includes: A segmentation module, used to perform semantic segmentation on the initial image to generate a first image for the target part; and The processing module is used for generating the material image by processing the first image based on the received modification requirement information for the initial image.

17. The device according to claim 16, wherein: The modification requirement information includes a target modification style; The processing module comprises: a fourth determining submodule, configured to determine, according to the target modification style, a first modification material image and a first position for adding the first modification material image into the first image; as well as The second generating submodule is used to generate a first material image by adding the first modified material image at the first position.

18. The device according to claim 17, wherein: The demand information also includes the part to be modified and the modification object for the part to be modified; The processing module further includes: An extraction submodule, used for extracting a second position of the part to be modified from the first image; a fifth determining submodule, configured to determine a second modified material image corresponding to the modified object according to the modification style; The third generating submodule is used to generate a second material image by adding the second modified material image at the second position.

19. The apparatus according to claim 17, further comprising: A second extraction module, used to extract the first keyword from the style description information; A third extraction module, configured to extract a second keyword from the target modification style; The third generating module is configured to generate warning information for characterizing style conflict in response to the matching degree between the first keyword and the second keyword being less than a predetermined threshold.

20. The device according to any one of claims 11 to 19, wherein: The target virtual image is used for display in the navigation map.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images