Image processing methods, apparatus, devices and storage media

Abstract

This application discloses an image processing method, apparatus, device, and storage medium, belonging to the field of artificial intelligence technology. The method is executed by a computer device and includes: acquiring a first image and guiding text, the guiding text indicating the correlation between image grayscale and depth information; inputting the first image into a structure prediction model to obtain structural features; performing encoding prediction on the guiding text using a first generation sub-model in an image generation model to obtain hidden layer features corresponding to the correlation; and performing image depth prediction based on the structural features and the hidden layer features using a second generation sub-model in the image generation model to obtain a depth image corresponding to the first image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, the demand for image information processing is becoming increasingly diverse.

[0003] In related technologies, depth images are image information used to describe three-dimensional spatial structures. They can indicate the surface depth of a three-dimensional structure based on grayscale values. For example, the higher the grayscale value of a point, the higher the height of that point in three-dimensional space.

[0004] However, how to efficiently generate depth images is a problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and storage medium, the technical solution of which is as follows:

[0006] According to one aspect of this application, an image processing method is provided, the method being performed by a computer device, the method comprising:

[0007] Acquire a first image and guiding text, wherein the guiding text is used to indicate the correlation between image grayscale and depth information;

[0008] The first image is input into the structure prediction model to obtain structural features;

[0009] The first generation sub-model in the image generation model is used to perform encoding prediction on the guiding text to obtain the hidden layer features corresponding to the association relationship;

[0010] The second generation sub-model in the image generation model is used to perform image depth prediction based on the structural features and the hidden layer features to obtain the depth image corresponding to the first image.

[0011] According to another aspect of this application, an image processing apparatus is provided, the apparatus comprising:

[0012] An acquisition module is used to acquire a first image and guiding text, wherein the guiding text is used to indicate the correlation between image grayscale and depth information;

[0013] The processing module is used to input the first image into the structure prediction model to obtain structural features;

[0014] The processing module is further configured to perform encoding prediction on the guiding text using the first generation sub-model in the image generation model to obtain the hidden layer features corresponding to the association relationship;

[0015] The processing module is further configured to use the second generation sub-model in the image generation model to perform image depth prediction based on the structural features and the hidden layer features, so as to obtain a depth image corresponding to the first image.

[0016] In an optional design of this application, the structural prediction model includes at least two prediction network layers, and the processing module is further configured to:

[0017] The first image is input into the structure prediction model to obtain at least two structural sub-features, wherein the at least two structural sub-features are the output information corresponding to at least two prediction network layers in the structure prediction model;

[0018] The second generation sub-model in the image generation model includes at least two sub-networks, and the processing module is further configured to:

[0019] At least two sub-networks in the second generative sub-model are used to perform predictions based on the hidden layer features and one of the at least two structural sub-features, respectively, to obtain at least two structural prediction tensors, wherein the at least two structural prediction tensors are the output information corresponding one-to-one with the at least two sub-networks in the second generative sub-model;

[0020] The depth image corresponding to the first image is determined based on at least one structural prediction tensor.

[0021] In an optional design of this application, at least two sub-networks in the second generated sub-model include a first sub-network and a second sub-network cascaded sequentially.

[0022] The processing module is also used for:

[0023] The first sub-network is used to perform prediction based on the hidden layer features and the first sub-feature among the at least two structural sub-features to obtain the first structural prediction tensor;

[0024] The second sub-network is used to perform prediction based on the first structure prediction tensor and the second sub-feature among the at least two structure sub-features to obtain the second structure prediction tensor;

[0025] The depth image is determined based on the second structure prediction tensor.

[0026] In an optional design of this application, the processing module is further configured to:

[0027] A weighted summation is performed on the structural features and the hidden layer features to obtain the fused features;

[0028] The second generation sub-model in the image generation model is used to perform image depth prediction based on the fusion features to obtain the depth image corresponding to the first image.

[0029] In an optional design of this application, the processing module is further configured to:

[0030] Using an image style model to perform prediction based on the guiding text, the style features corresponding to the correlation between the image grayscale and the depth information are obtained;

[0031] The depth image is obtained by the second generation sub-model performing image prediction based on the structural features, the hidden layer features, and the style features.

[0032] In an optional design of this application, the processing module is further configured to be used for any one of the following:

[0033] Using the third sub-network in the second generative sub-model, image prediction is performed based on the structural features, the hidden layer features, and the style features to obtain the depth image;

[0034] Using the fourth sub-network in the second generative sub-model, a prediction is performed based on the structural features and the hidden layer features to obtain a structural prediction tensor; and using the fifth sub-network in the second generative sub-model, a prediction is performed based on the structural prediction tensor and the style features to obtain the depth image;

[0035] Using the sixth sub-network in the second generative sub-model, a prediction is performed based on the style features and the hidden layer features to obtain a style prediction tensor; and using the seventh sub-network in the second generative sub-model, a prediction is performed based on the style prediction tensor and the structural features to obtain the depth image.

[0036] In an optional design of this application, the image style model includes at least two style network layers, and the processing module is further configured to:

[0037] Using the image style model to perform prediction based on the guiding text, at least two style sub-features are obtained, and the at least two style sub-features are the output information corresponding to at least two style network layers in the image style model;

[0038] In the second generative sub-model, at least two generative network layers perform predictions based on the hidden layer features and one of the at least two style sub-features to obtain the depth image; the at least two generative network layers belong to the third, fifth, or sixth sub-network.

[0039] In an optional design of this application, the acquisition module is further configured to:

[0040] Obtain sample depth images and labeled text;

[0041] The processing module is also used for:

[0042] The initial image prediction model is used to perform prediction based on the labeled text to obtain a predicted depth image;

[0043] The device further includes:

[0044] The training module is used to train the initial image prediction model based on the difference between the predicted depth image and the sample depth image to obtain the image style model in the image prediction model.

[0045] In an optional design of this application, the processing module is further configured to:

[0046] Edge extraction is performed on the first image to obtain a second image, which is a line drawing image corresponding to the first image;

[0047] The structural features are obtained by the structural prediction model based on the second image.

[0048] In an optional design of this application, the processing module is further configured to:

[0049] Perform image enhancement on the first image to obtain the enhanced image corresponding to the first image;

[0050] Edge extraction is performed on the enhanced image to obtain the second image;

[0051] The image enhancement is used to instruct the first image to undergo grayscale conversion and / or resolution correction.

[0052] In an optional design of this application, the processing module is further configured to:

[0053] Based on the depth image, a three-dimensional structural model of the image object in the first image is constructed.

[0054] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the image processing method as described above.

[0055] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image processing method described above.

[0056] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer instructions to implement the image processing method described above.

[0057] The beneficial effects of the technical solution provided in this application include at least the following:

[0058] The structural features of the first image are extracted by the structural prediction model to describe the image content of the first image. The image generation model performs image prediction based on the structural features and the hidden features of the guiding text. The structural features of the first image provide a reference for the content of the depth image, and the guiding text indicates the relationship between image grayscale and depth information, providing a reference for the image style of the depth image based on the image grayscale representation of depth changes in the predicted depth image. The depth image is generated without the need to label the image content of the first image. This avoids the need to label the image content of the sample images with natural language when training the image generation model. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application;

[0061] Figure 2 This is a schematic diagram of an image processing method provided in an exemplary embodiment of this application;

[0062] Figure 3 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0063] Figure 4 This is a schematic diagram of a depth image provided in an exemplary embodiment of this application;

[0064] Figure 5 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0065] Figure 6 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0066] Figure 7This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0067] Figure 8 This is a schematic diagram of a depth image provided in an exemplary embodiment of this application;

[0068] Figure 9 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0069] Figure 10 This is a schematic diagram illustrating image style model training provided in an exemplary embodiment of this application;

[0070] Figure 11 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0071] Figure 12 This is a structural block diagram of an image processing apparatus provided in an exemplary embodiment of this application;

[0072] Figure 13 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0073] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0075] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0076] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the images, guidance text and other information involved in this application were obtained with full authorization.

[0078] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0079] Figure 1 A schematic diagram of a computer system provided in one embodiment of this application is shown. This computer system can implement a system architecture for an image processing method. The computer system may include a terminal 100 and a server 200.

[0080] Terminal 100 can be an electronic device such as a mobile phone, tablet computer, or PC (Personal Computer). A client application for the target application can be installed and run on terminal 100. This target application can be an image processing application, a model building application, or other applications that provide data processing functions; this application does not limit the specific application. Furthermore, this application does not limit the form of the target application, including but not limited to apps, mini-programs, etc., installed on terminal 100, and it can also be in web page form.

[0081] Server 200 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 200 can be the backend server for the aforementioned target application, used to provide backend services to the clients of the target application.

[0082] The image processing method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1Taking the implementation environment of the scheme shown as an example, the image processing method can be executed by the terminal 100 (such as by the client of the target application installed and running in the terminal 100), or by the server 200, or by the terminal 100 and the server 200 interacting and cooperating to execute it. This application does not limit this.

[0083] Furthermore, the technical solution of this application can be combined with blockchain technology. For example, in the image processing method disclosed in this application, some data involved (such as the first image, depth image, three-dimensional structural model, etc.) can be stored on the blockchain. The terminal 100 and the server 200 can communicate through a network, such as a wired or wireless network.

[0084] Figure 2 A schematic diagram of an image processing method provided in an exemplary embodiment of this application is shown.

[0085] A color image 200 and guiding text 210 are acquired; the color image 200 is the image from which depth information is to be extracted, and the guiding text 210 is used to indicate the relationship between image grayscale and depth information; for example, as depth information increases, the distance between pixels and the imaging plane (or camera) in three-dimensional space increases, and the grayscale of the desired generated image also increases. The line drawing image 204 carries the geometric feature information contained in the color image 200;

[0086] Perform grayscale conversion 310 on the color image 200 to obtain the grayscale image 202 corresponding to the color image 200; perform edge extraction 230 on the grayscale image 202 to obtain the line drawing image 204;

[0087] The structure prediction model 410 performs predictions based on the line drawing image 204 to obtain structural features 204a; structural features 204a are used to indicate at least one of the features such as shape, layout, and texture of objects in the color image 200. For example, the structure prediction model 410 is implemented as a Linear-Based ControlNet.

[0088] The image generation model 420 includes a first generation sub-model 422 and a second generation sub-model 424, which are described below. For example, the image generation model 420 is implemented as a Text2Image model.

[0089] The first generation sub-model 422 in the image generation model 420 performs encoding prediction based on the guiding text 210 to obtain the hidden layer features 210a corresponding to the association relationship; the first generation sub-model 422 is the model in the image generation model 420 that performs encoding on the guiding text 210 to convert natural text information into feature information in the hidden layer space.

[0090] The second generation sub-model 424 in the image generation model 420 performs image depth prediction based on structural features 204a, hidden layer features 210a and style features 210b to obtain a depth image 220; the second generation sub-model 424 is a model in the image generation model 420 that performs at least one of the following methods on the feature information of the hidden layer space: convolution, pooling, fully connected, decoding, etc., to obtain image information.

[0091] For example, the second generative sub-model 424 may simultaneously perform data processing on structural features 204a, hidden layer features 210a, and style features 210b; it may also include multiple sub-networks, each performing data processing on at least one of structural features 204a, hidden layer features 210a, and style features 210b.

[0092] Regarding style feature 210b, style feature 210b is obtained by performing predictions on the guiding text 210 using the image style model 430. For example, the image style model 430 is implemented as a gray-level-based low-rank adaptation (Lora) model. The image style model 430 has the ability to predict text information as hidden features of image style that are correlated with gray-level and depth information, providing a supplementary description of the correlation between image gray-level and depth information from the dimension of hidden features.

[0093] For example, a three-dimensional structure model 230 in three-dimensional space is determined based on the grayscale of each pixel in the depth image 220.

[0094] The image processing method will now be described through the following examples.

[0095] Figure 3 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. The method can be executed by a computer device. The method includes:

[0096] Step 510: Obtain the first image and guiding text;

[0097] For example, the first image is an image from which depth information is to be extracted. The depth information describes the distance between pixels in the first image and the imaging plane of the first image. For example, the depth information is used to describe the positional information of pixels in the first image in three-dimensional space. For example, the first image is typically a color image, but a grayscale image is also possible.

[0098] For example, guiding text is used to indicate the relationship between image grayscale and depth information; for instance, as depth information increases, the distance between a pixel and the imaging plane (or camera) in three-dimensional space increases, and the desired grayscale of the generated image also increases. For example, guiding text, also called style guiding text or image style guiding text, is used to constrain the desired image style to be one where grayscale and depth information are correlated; for example, guiding text indicates that the desired generated image has the relationship between depth information and image grayscale as described above. For example, guiding text describes the relationship between image grayscale and depth information in natural language.

[0099] Step 520: Input the first image into the structure prediction model to obtain structural features;

[0100] For example, the structure prediction model is used to predict structural features in a first image. These structural features indicate at least one of the shape, layout, and texture of objects in the first image, and also indicate the style and arrangement of items within the first image. For example, the structure prediction model can be implemented as a statistical model, such as a model that statistically identifies pixels in the first image that have significant visual differences from their neighboring pixels, in order to extract the main subject of the image. The structure prediction model can also be implemented as an Artificial Neural Network (ANN) model, predicting structural features through at least one of the operations such as encoding, pooling, decoding, and convolution.

[0101] Step 530: Use the first generation sub-model in the image generation model to perform encoding prediction on the guiding text to obtain the hidden features corresponding to the association;

[0102] For example, the first generation sub-model is used to convert the guiding text from text format into at least one data format among feature values, feature vectors, and feature matrices, to describe the feature information of the guiding text in the hidden layer space. For example, the hidden layer space is used to indicate the semantics in the guiding text in natural language form in a quantitative data description manner using at least one data format among feature values, feature vectors, and feature matrices. The associations indicated in the guiding text are encoded into hidden layer features by the first generation sub-model, obtaining the hidden layer features corresponding to the associations. For example, the hidden layer features can be feature information predicted at the word-level by encoding each word in the guiding text separately; or they can be feature information predicted at the overall sentence-level by encoding all words in the guiding text. This application does not limit the prediction method of the guiding text.

[0103] Step 540: Use the second generation sub-model in the image generation model to perform image depth prediction based on structural features and hidden layer features to obtain the depth image corresponding to the first image;

[0104] For example, the second generating sub-model is used to perform image depth prediction to obtain a depth image by taking structural features and hidden layer features as input information.

[0105] For example, hidden layer features provide image style constraint information for the second generation sub-model, instructing it to generate a depth image according to the correlation between image grayscale and depth information indicated by the guiding text. The depth image has an image style where there is a correspondence between depth information and image grayscale. For example, structural features provide image content constraint information for the second generation sub-model, instructing it to generate a depth image according to the structural features of the first image.

[0106] In summary, the method provided in this embodiment extracts structural features from the first image through a structural prediction model, describing the image content in the first image. The image generation model performs image prediction based on the structural features and the hidden layer features of the guiding text. The structural features of the first image provide a reference for the content of the depth image, and the guiding text indicates the correlation between image grayscale and depth information, providing a reference for the image style of the predicted depth image based on the image grayscale representation of depth changes. The method generates a depth image without needing to label the image content of the first image, and avoids labeling the image content of sample images with natural language when training the image generation model.

[0107] Figure 4 A schematic diagram of a depth image provided in an exemplary embodiment of this application is shown.

[0108] A camera 650 captures images of three-dimensional objects in three-dimensional space, resulting in a three-dimensional view, i.e., an image containing the three-dimensional objects. Figure 4 In this context, the three-dimensional objects include spheres 662, pyramids 664, and cylinders 666; however, it is not excluded that in other examples, the three-dimensional space may include more or fewer three-dimensional objects, and / or other three-dimensional objects with delayed appearances may be included in the three-dimensional space.

[0109] For example, the imaging plane 670 is perpendicular to the shooting direction 650a of the camera 650; Figure 4The imaging plane 670 in the text is an exemplary description; further, in some examples, the imaging plane is the plane where the lens of the camera 650 is located, or the plane where the light rays are focused by the lens of the camera 650. Exemplarily, the distance between the sphere 662 and the imaging plane 670 is a first distance 672, the distance between the pyramid 664 and the imaging plane 670 is a second distance 674, and the distance between the cylinder 666 and the imaging plane 670 is a third distance 676. Wherein, the first distance 672 is less than the second distance 674, and the second distance 674 is less than the third distance 676.

[0110] For example, the depth image 680 includes projections of the sphere 662, the pyramid 664, and the cylinder 666 onto a plane;

[0111] The fill color of the projection area of ​​sphere 662 in depth image 680 is the first gray value 682; the fill color of the projection area of ​​pyramid 664 in depth image 680 is the second gray value 684; and the fill color of the projection area of ​​cylinder 666 in depth image 680 is the third gray value 686.

[0112] For example, the gray values ​​filled in the depth image 680 increase as the distance between the object and the imaging plane 670 increases; therefore, the first gray value 682 is less than the second gray value 684, and the second gray value 684 is less than the third gray value 686.

[0113] Figure 5 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 3 In the illustrated embodiment, step 520 can be implemented as step 522; step 540 can be implemented as steps 542 and 544.

[0114] Step 522: Input the first image into the structure prediction model to obtain at least two structural sub-features;

[0115] For example, the structural prediction model includes at least two prediction network layers, which include two prediction network layers connected in parallel, and / or two prediction network layers cascaded one after the other; the structural prediction model may include more prediction network layers in different implementations.

[0116] For example, two parallel prediction network layers are used to perform predictions on different sub-regions in the first image to obtain structural features of different sub-regions. The two cascaded prediction network layers are used to perform predictions on the first image at different granularities. For example, one prediction network layer performs predictions on pixel blocks containing multiple pixels with a large field of view, while the other prediction network layer performs predictions on individual pixels with a small field of view.

[0117] For example, at least two structural sub-features are the output information corresponding one-to-one between at least two prediction network layers in the structural prediction model. Structural sub-features are information describing the style and arrangement of items within a first image from one dimension, such as pixel blocks or individual pixels, or information describing the style and arrangement of items in a portion of the first image. As described above, two structural sub-features can be used to indicate the structural features of different regions in the first image; two structural sub-features can also be used to indicate the structural features of the same region or the first image as a whole in different dimensions, such as indicating the structural features of pixel blocks of different sizes as a whole, or two features among the structural features of individual pixels.

[0118] Step 542: Use at least two sub-networks in the second generative sub-model to perform predictions based on hidden layer features and one of the sub-features of at least two structural sub-features, respectively, to obtain at least two structural prediction tensors;

[0119] For example, the second generation sub-model in the image generation model includes at least two sub-networks. Similar to step 522, this step does not restrict the connection relationship between the two sub-networks. The at least two sub-networks include two sub-networks connected in parallel, and / or two sub-networks cascaded one after the other. For example, the input sub-features are usually different when different sub-networks perform predictions, but it does not exclude the case that different sub-networks perform predictions based on the same sub-features.

[0120] For example, at least two structure prediction tensors are the output information corresponding one-to-one with at least two sub-networks in the second generative sub-model. The structure prediction tensor is the information predicted based on the constraints of the structure sub-features during the prediction process of the second generative sub-model. For example, this application does not limit the order of the tensor; for instance, a tensor implemented as order 0 is a scalar, a tensor implemented as order 1 is a vector, and a tensor implemented as order 2 is a matrix. It is understood that tensors of higher orders can also be called high-dimensional matrices.

[0121] For example, the subnetwork can be implemented as at least one of fully connected layers, convolutional layers, pooling layers, etc. This application does not limit the specific implementation type and number of subnetworks.

[0122] In one alternative implementation, at least two sub-networks in the second generating sub-model include a first sub-network and a second sub-network that are cascaded sequentially; correspondingly, step 542 can be implemented as follows:

[0123] • Perform predictions using the first sub-network based on hidden layer features and the first sub-feature from at least two structural sub-features to obtain the first structural prediction tensor;

[0124] For example, the first sub-network is used to predict the first structure prediction tensor using hidden layer features and first sub-features as input information.

[0125] • Perform predictions using a second sub-network based on the first structure prediction tensor and the second sub-features from at least two structure sub-features to obtain the second structure prediction tensor;

[0126] For example, the second sub-network is used to predict the second structure prediction tensor by taking the first structure prediction tensor and the second sub-feature as input information.

[0127] Optionally, in this implementation, the structural prediction model includes at least two prediction network layers, including a first prediction network layer and a second prediction network layer cascaded one after the other; the first sub-feature is predicted by the first prediction network layer, and the second sub-feature is predicted by the second prediction network layer.

[0128] For example, cascaded prediction network layers are used to predict structural features of the first image in different dimensions. Two cascaded sub-networks in the second generative sub-model perform predictions based on structural features in different dimensions, resulting in tensor information with reference to these structural features.

[0129] For example, in the case where at least two subnetworks include a first subnetwork and a second subnetwork cascaded sequentially, the depth image is determined based on a second structure prediction tensor. The second structure prediction tensor is predicted based on the first structure prediction tensor and the second sub-features, directly referencing the structural information of the first image described by the second sub-features, and indirectly referencing the structural information of the first image described by the first sub-features and the correlation between image grayscale and depth information indicated by the guiding text through the first structure prediction tensor.

[0130] Step 544: Based on at least one structure prediction tensor, determine the depth image corresponding to the first image;

[0131] For example, the structure prediction tensor can be implemented as two-dimensional matrix information or as higher-order matrix information; this application does not impose any limitation on this. For example, the structure prediction tensor is only used to limit the tensor information to be generated based on structural features, and does not limit the type of information carried in the structure prediction tensor.

[0132] For example, the structure prediction tensor can be information stored in the form of red-green-blue (RGB) channels and / or hue-saturation-value (HSV) channels and / or grayscale channels, which is one way to store image information. A depth image in image form is obtained through the conversion of the above channel information.

[0133] For example, structural prediction information can also be a feature description information in a hidden layer space. For example, the hidden layer space in this step is different from the hidden layer space used for transcoding natural language information in step 530. For example, based on at least one structural prediction tensor, a depth image corresponding to the first image is obtained by decoding. This embodiment does not limit the decoding method of the structural prediction tensor. For example, a decoder (or decoding sub-model / decoding sub-network) in the image generation model is used to perform decoding prediction on at least one structural prediction tensor to obtain the depth image corresponding to the first image.

[0134] In summary, the method provided in this embodiment includes a structural prediction model comprising at least two prediction network layers, which respectively predict different content in different parts of the image or describe the image content in the first image from different dimensions; a second generation sub-model comprising at least two sub-networks performing image prediction based on one of at least two structural sub-features and the hidden layer features of the guiding text; the hidden layer features corresponding to the correlation between image grayscale and depth information provide a reference for the image style of the depth image based on the image grayscale representation of depth changes; the method achieves the generation of depth images without needing to label the image content of the first image; and avoids labeling the image content of sample images with natural language when training the image generation model.

[0135] Figure 6 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 3 In the illustrated embodiment, step 540 can be implemented as steps 546 and 548:

[0136] Step 546: Perform a weighted summation on the structural features and hidden layer features to obtain the fused features;

[0137] For example, a weighted summation is performed on the depth image based on the importance of structural features and hidden layer features to obtain fused features. For example, the structural features and hidden layer features have the same matrix dimension and / or matrix size.

[0138] For example, during the weighted summation process, the weights of structural features and hidden features can be pre-set empirical values ​​or parameter values ​​trained during the training process that demonstrate good depth image generation capabilities. In one example, the weighted weight of structural features is less than the weighted weight of hidden features. Since structural features are extracted from the first image, they can quantitatively describe the content of the first image in a data-driven manner. The guiding text is a conversational introduction to the relationship between image grayscale and depth information. By increasing the weight of the hidden features encoded by the guiding text, the focus on the relationship between image grayscale and depth information during image prediction is increased, ensuring that the generated depth image closely reflects the relationship.

[0139] Step 548: Use the second generation sub-model in the image generation model to perform image depth prediction based on the fusion features to obtain the depth image corresponding to the first image;

[0140] For example, in this step, step 548 performs image prediction based on the fused features to obtain a depth image. The second generative sub-model obtains the fused features with single input information and obtains the depth image through image prediction. For example, in this application, the second generative sub-model can be implemented as at least one of fully connected layers, convolutional layers, pooling layers, etc., and this application does not limit the specific implementation type and number of the second generative sub-model.

[0141] In summary, the method provided in this embodiment extracts structural features from the first image through a structural prediction model, describing the image content in the first image. The image generation model performs image prediction based on the structural features and the hidden layer features of the guiding text. The structural features of the first image provide a reference for the content of the depth image, and the guiding text indicates the correlation between image grayscale and depth information, providing a reference for the image style of the predicted depth image based on the image grayscale representation of depth changes. The method generates a depth image without needing to label the image content of the first image, and avoids labeling the image content of sample images with natural language when training the image generation model.

[0142] The image style model will be introduced below.

[0143] Figure 7 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 3 In the illustrated embodiment, step 525 is also included, and step 540 can be implemented as any one of steps 540a to 540c:

[0144] Step 525: Use an image style model to perform prediction based on the guiding text to obtain the style features corresponding to the correlation between image grayscale and depth information;

[0145] For example, as described above, the guiding text is used to indicate the relationship between image grayscale and depth information; for instance, as depth information increases, the distance between pixels and the imaging plane (or camera) in three-dimensional space increases, and the grayscale of the desired generated image also increases. This is also called the image style of the relationship between image grayscale and depth information. Image style refers to the overall representation of an image. For the same object in an image, different colors or textures can present different styles. For example, for a person in an image, using large solid color blocks to represent the person is an anime style presentation; while using pencil stroke textures to represent the person is a sketch style presentation. For example, the image style model has the ability to predict guiding text as feature information in the hidden layer space. In various embodiments of this application, the guiding text includes: a black background, sharp edges, and smoothness; background and line features used to constrain the desired generated depth image. For example, the image style model is a sub-part of an artificial neural network with text-to-image generation capabilities. For example, in the text-to-image generation process, the feature information of the guiding text is used as the style feature of the guiding text. Furthermore, the artificial neural network with text-to-image generation capabilities is trained by fine-tuning based on manually co-constructed deep images. The training process for the image style model will be described separately below.

[0146] For example, style features provide the desired image style for the prediction process of depth images in the form of at least one of eigenvalues, eigenvectors, and eigenmatrices.

[0147] It should be noted that step 520 can be executed before, after, or simultaneously with step 525. This application does not restrict the execution order between step 520 and step 525.

[0148] It should be noted that step 525 can be related to... Figure 3 The various steps in the process can be combined to form new embodiments for individual implementation, and this application does not impose any limitations on this; correspondingly, the depth image is obtained by the second generation sub-model performing image prediction based on structural features, hidden layer features, and style features. Style features and hidden layer features are the prediction results of different models for the guiding text, providing the desired image style for the second generation sub-model from multiple perspectives, and providing more basis for image style for the image prediction of the second generation sub-model.

[0149] Step 540a: Using the third sub-network in the second generative sub-model, perform image prediction based on structural features, hidden layer features, and style features to obtain a depth image;

[0150] In this step, the second generative sub-model contains a sub-network that performs predictions on structural features, hidden layer features, and style features to obtain a depth image.

[0151] For example, the third sub-network can directly predict depth images based on structural features, hidden layer features, and style features; the third sub-network can also be combined with... Figure 5 Similar to the illustrated embodiment, prediction is performed based on structural features, hidden layer features, and style features to obtain a prediction tensor (also known as a fused prediction tensor); the depth image corresponding to the first image is determined based on the prediction tensor. This embodiment does not limit the prediction method of the third sub-network.

[0152] In one alternative implementation, the image style model includes at least two style network layers, and the third sub-network includes at least two generative network layers. Accordingly, step 525 above is implemented as follows:

[0153] Using an image style model to perform predictions based on guiding text, at least two style sub-features are obtained. The at least two style sub-features are the output information of at least two style network layers in the image style model.

[0154] For example, at least two style network layers include two style network layers connected in parallel, and / or two style network layers cascaded one after the other; for example, the two style network layers connected in parallel are used to perform predictions on different characters in the guiding text, and the two style network layers cascaded one after the other are used to perform predictions on the guiding text at two different granularities (such as single character dimension, phrase dimension, sentence dimension, etc.).

[0155] Accordingly, in the third sub-network, the generator network layer performs predictions based on hidden layer features, one sub-feature from at least two style sub-features, and one sub-feature from at least two structural sub-features to obtain the depth image. For an explanation of the structural sub-features, please refer to step 522 above; it will not be repeated here. The third sub-network, through at least two generator network layers, simultaneously utilizes the prediction results of the image style model and the structural prediction model, providing a basis for depth image prediction and improving the accuracy of the depth image.

[0156] Step 540b: Using the fourth sub-network in the second generative sub-model, perform prediction based on structural features and hidden layer features to obtain a structural prediction tensor; and using the fifth sub-network in the second generative sub-model, perform prediction based on the structural prediction tensor and style features to obtain a depth image;

[0157] The second generative sub-model contains two sub-networks. First, it performs predictions on structural features and hidden layer features, and then it performs predictions on style features and the structural prediction tensor obtained from the previous predictions to obtain a depth image.

[0158] Similar to the third sub-network, the fifth sub-network can directly predict the depth image; alternatively, it can predict the prediction tensor and then determine the depth image based on the prediction tensor. This embodiment does not limit the prediction method of the fifth sub-network. For example, the prediction tensor is information predicted based on constraints of structural and style features during the prediction process of the fifth sub-network. For example, similar to step 544 above, the depth image can be obtained by converting the prediction tensor into an image format, or the depth image corresponding to the first image can be obtained by decoding the prediction tensor.

[0159] In one alternative implementation, the image style model includes at least two style network layers, and the fifth sub-network includes at least two generative network layers. For a description of the at least two style sub-features and the at least two style network layers, please refer to the description of the alternative implementation in step 540a, which will not be repeated here.

[0160] Accordingly, the generative network layer performs predictions based on the structure prediction tensor and one of the at least two style sub-features to obtain a depth image. For an introduction to the structure prediction tensor and the fourth sub-network, please refer to step 522 above. The fourth sub-network can be implemented as a set of at least two sub-networks from step 542. For example, the structure prediction tensor used by the generative network layer during prediction is predicted by the fourth sub-network based on the structure features and hidden layer features; the structure prediction tensor indirectly carries the hidden layer features.

[0161] Step 540c: Using the sixth sub-network in the second generative sub-model, perform prediction based on style features and hidden layer features to obtain the style prediction tensor; and using the seventh sub-network in the second generative sub-model, perform prediction based on the style prediction tensor and structural features to obtain the depth image;

[0162] The second generative sub-model contains two sub-networks. First, it performs prediction on style features and hidden layer features, and then it performs prediction on structural features and the style prediction tensor obtained from the previous prediction to obtain a depth image.

[0163] Similar to the third sub-network, the seventh sub-network can directly predict the depth image; alternatively, it can predict the prediction tensor and then determine the depth image based on the prediction tensor. This embodiment does not limit the prediction method of the fifth sub-network. For example, the prediction tensor is the information predicted during the prediction process of the seventh sub-network based on constraints of structural and style features. For example, similar to step 544 above, the depth image can be obtained by converting the prediction tensor into an image format, or the depth image corresponding to the first image can be obtained by decoding the prediction tensor.

[0164] In one alternative implementation, the image style model includes at least two style network layers, and the sixth sub-network includes at least two generative network layers. For a description of the at least two style sub-features and at least two style network layers, please refer to the description of the alternative implementation in step 540a, which will not be repeated here.

[0165] Accordingly, the generative network layer performs predictions based on hidden layer features and one of at least two style sub-features to obtain a style prediction tensor; wherein, the seventh sub-network can be implemented as a set of at least two sub-networks in step 542. For example, the generative network layer directly uses hidden layer features during prediction, and the depth image is obtained after prediction by the seventh sub-network.

[0166] In steps 540a to 540c, at least two generative network layers in the second generative sub-model perform predictions based on hidden layer features and one of the sub-features of at least two style sub-features to obtain a depth image; at least two generative network layers belong to the third sub-network, the fifth sub-network, or the sixth sub-network.

[0167] In summary, the method provided in this embodiment extracts structural features from the first image through a structural prediction model, describing the image content in the first image; it performs prediction based on guiding text through an image style model, realizing the acquisition of feature representations in the hidden layer space of the desired image style from a model with depth image prediction capabilities; the image generation model performs image prediction based on the hidden layer features of structural features, guiding text, and style features. The structural features of the first image provide a reference for the content of the depth image, and the guiding text indicates the correlation between image grayscale and depth information, providing a reference for predicting the image style of the depth image based on the image grayscale representation of depth changes; it achieves the generation of depth images without needing to label the image content of the first image; and it avoids labeling the image content of sample images with natural language when training the image generation model.

[0168] In one implementation, the prediction process for a depth image is as follows:

[0169] R cn =M cn (I lineart )

[0170] R t2i =M t2i (T prompt )

[0171] R g =M g (T prompt )

[0172] I out =Pipe(R) cn ,Rt2i ,R g params)

[0173] The structural prediction model is denoted as M. cn R cn The structural prediction model is applied to the first image I. lineart The predicted structural features; for example, the structural prediction model is implemented as a Linear-Based ControlNet.

[0174] In one example:

[0175]

[0176] Where N control =22, indicating the number of sub-features included in the structural feature.

[0177] Image style model denoted as M g R g It is an image style model for guiding text T prompt The predicted style features; in one example:

[0178]

[0179] Where N lora =350 indicates the number of sub-features included in the style feature.

[0180] The image generation model is denoted as M. t2i R t2i It is an image generation model for guiding text T prompt The predicted hidden layer features, for example, M t2i It is the first generative sub-model in the image generation model, R t2i These are the predicted hidden layer features.

[0181] In one example:

[0182]

[0183] Where, N control =22, N lora =350, the specific meaning of which will be explained below.

[0184] For example, a Pipe is a standard computation pipeline consisting of a single ControlNet model, a single LoRa model, and a single Text2Image model, and performs the following main operations:

[0185]

[0186] `params` represents hyperparameters used to control the speed and quality of image generation, including the aforementioned α and β, with default values ​​of α = 0.7 and β = 0.85, but other empirical values ​​are also possible. For example, the pipeline is also called the second generative sub-model in the image generation model; one sub-network in the second generative sub-model performs prediction on the weighted sum of structural features and hidden layer features, while another sub-network in the second generative sub-model performs prediction on the weighted sum of style features and hidden layer features. This is to achieve the prediction of a depth image I. out For example, the two sub-networks in the second generative sub-model predict the predicted tensor and the depth image I, respectively. out It is determined based on the prediction tensors obtained from the predictions of the two subnetworks respectively. Figure 8 This illustration shows a schematic diagram of depth images provided in an exemplary embodiment of this application. The first depth image 602 is a depth image of the virtual poet's character image, the second depth image 604 is a depth image of the appearance of a virtual flower, and the third depth image 606 is a depth image of the appearance of a virtual ornament.

[0187] In various embodiments of this application, a depth image, also called a height map or displacement map, is an image file that stores information about the depth (or height) of a three-dimensional object's surface in grayscale values. The primary function of a depth image is to represent the levels of detail on a three-dimensional model. In a depth image, changes in depth (or height) are converted into changes in grayscale values; a higher grayscale value indicates a higher height of that pixel in three-dimensional space, and a lower grayscale value indicates a lower height.

[0188] The following describes the training process of image style models. For example, in... Figure 7 In addition to, it also includes, Figure 9 Steps 502, 504, and 506 are shown below:

[0189] Step 502: Obtain the sample depth image and labeled text;

[0190] For example, a sample depth image is an artificially constructed image with grayscale and depth information correlated; the annotation text is used to indicate the image content in the depth image, such as the depth image including at least one of the following: character appearance, natural landscape, architectural landscape, artistic form, etc.

[0191] Figure 10 This illustration shows a schematic diagram of image style model training provided in an exemplary embodiment of this application.

[0192] For example, a sample depth map 702 is collected, and image annotation 705 is performed on the sample depth map 702 to obtain annotated text 704; based on the sample depth map 702 and the annotated text 704, a sample information pair 708 is obtained. The sample depth map 702 can be collected from publicly available information or drawn manually. The annotated text 704 can be annotated using a large language model or manually annotated and corrected to ensure the accuracy of the annotated text 704.

[0193] Step 504: Use the initial image prediction model to perform prediction based on the labeled text to obtain the predicted depth image;

[0194] For example, the initial image prediction model has the ability to generate images based on textual information. For instance, if the labeled text does not contain information about the image style of the sample depth image, the initial image prediction model uses the image content as input to predict the depth image based on the description of the image content.

[0195] Step 506: Based on the difference between the predicted depth image and the sample depth image, train the initial image prediction model to obtain the image style model in the image prediction model;

[0196] For example, by training the initial image prediction model by predicting the difference between the depth image and the sample depth image, the model parameters of the initial image prediction model are adjusted to obtain the image prediction model. For example, since the training process uses the difference between the depth image and the sample depth image as the loss function, the trained image prediction model can predict images with grayscale and depth information that are correlated by the constraints of the model parameters.

[0197] For example, an image style model is part or all of an image prediction model; for example, the model parameters of an image style model have hidden features that predict natural language as an image style with a correlation between grayscale and depth information.

[0198] For example, Figure 9 The execution timing of steps 502, 504, and 506 shown is in Figure 9 Before step 525. For example, in step 525, the guiding text itself indicates an image style that is associated with grayscale and depth information, and the image style model has hidden layer features that predict text information as an image style that is associated with grayscale and depth information, thereby achieving further enhancement of the desired image style.

[0199] like Figure 10As shown, based on sample information, model training 720 is performed on the initial image prediction model 708 to obtain the image style model 730. For example, the kohya_ss tool is used to train the initial image prediction model (e.g., the LoRa model), which provides the runtime environment for the initial image prediction model. A grid search method is used to find the optimal training hyperparameters. The trained model is then tested, and the image style model 730 with the highest image generation quality is selected.

[0200] In summary, the method provided in this embodiment achieves the acquisition of feature representations in the hidden layer space of the desired image style from a model with depth image prediction capabilities by performing prediction based on guided text using an image style model. The image style model is obtained by training an initial image prediction model, which improves the accuracy of acquiring style features in the hidden layer space of the desired image style and provides a reference for predicting the image style based on the image grayscale representation of depth changes in the depth image. It achieves the generation of depth images without needing to label the image content of the first image and avoids labeling the image content of sample images with natural language when training the image generation model.

[0201] Figure 11 A flowchart illustrating an exemplary embodiment of the image processing method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 3 The illustrated embodiment further includes steps 512, 514, and 550:

[0202] Step 512: Perform image enhancement on the first image to obtain the enhanced image corresponding to the first image;

[0203] For example, image enhancement is used to instruct the first image to undergo grayscale conversion and / or resolution correction.

[0204] For grayscale conversion; first convert the first image I ref Convert to a standard grayscale image and perform grayscale histogram equalization to enhance image contrast and highlight image details, for example:

[0205] I g =f hist (f grayscale (I ref ))

[0206] Among them, f grayscale This represents the operation of converting an RGB image to a standard grayscale image, f. hist This represents the operation of adaptive grayscale histogram equalization on a grayscale image. g This represents the image after grayscale conversion.

[0207] For resolution correction; in one example, resolution correction is performed after grayscale conversion.

[0208] First Image I ref The original resolution is (w raw ,h raw The resolution after resolution correction is (w e ,h e Resolution correction is implemented as follows:

[0209]

[0210] w offset =f round (0.5*(w e -w raw ))

[0211] h offset =f round (0.5*(h e -h raw ))

[0212] I pre [w offset :w raw +w offset ,h offset :h raw +h offset ] = I g

[0213] Among them, I pre [w offset :w raw +w offset ,h offset :h raw +h offset ] = I g I represents the central region of the expanded image (i.e., the enhanced image). g The gray levels of other edge regions are direct copies and extensions of the gray levels of the outermost edges of the central region image, f ceil The floor function f round This indicates rounding operations. Resolution correction ensures that the enhanced image's resolution meets the resolution requirements of the model's prediction model, avoiding the loss of image details.

[0214] Step 514: Perform edge extraction on the enhanced image to obtain the second image;

[0215] For example, the second image is a line drawing image corresponding to the first image, and the line drawing image carries the geometric feature information contained in the first image; the second image can be regarded as a maximized representation of the three-dimensional structural information corresponding to the first image on a two-dimensional plane. For example, edge extraction is implemented as follows:

[0216] I lineart =F lineart (I pre )

[0217] Where F lineart This refers to the line art extraction operation. For example, the line art extraction operation can be implemented as either Realistic Lineart or Standard Lineart. By performing the line art extraction operation, edge extraction is achieved for the enhanced image.

[0218] Accordingly, in this embodiment, the structural features are predicted by the structural prediction model based on the second image.

[0219] In one example, edge extraction is performed only on the first image to obtain the second image. Accordingly, the structural features are predicted by the structural prediction model based on the second image.

[0220] It should be noted that steps 512 and 514 above can be combined with... Figure 3 The various steps in the process can be combined to form new embodiments and implemented separately, and this application does not limit this.

[0221] Step 550: Based on the depth image, construct a 3D structural model of the image object in the first image;

[0222] For example, grayscale values ​​in a depth image are correlated with depth information. Based on the grayscale values ​​of each pixel in the depth image, a three-dimensional structural model in three-dimensional space is determined. It should be noted that since the depth image carries depth information about the object at a single angle, the constructed three-dimensional structural model may be incomplete and unable to represent spatial information at partial angles. Alternatively, it can be a three-dimensional model predicted from the depth image, supplementing the incomplete spatial structure predictions; this embodiment does not impose any limitations on this.

[0223] It should be noted that step 550 can be related to... Figure 3 The various steps in the process can be combined to form new embodiments and implemented separately, and this application does not limit this.

[0224] In summary, the method provided in this embodiment performs image enhancement and edge extraction on the first image to obtain a line drawing image corresponding to the first image. The line drawing image can more clearly present the structure of the image content through visual information, laying the foundation for the structure prediction model to extract structural features from the first image and describe the image content in the first image, thus providing accuracy in describing structural features. The image generation model performs image prediction based on the structural features and the hidden layer features of the guiding text. The structural features of the first image provide a reference for the content of the depth image, and the guiding text indicates the correlation between image grayscale and depth information, providing a reference for predicting the image style of the depth image based on the image grayscale representation of depth changes. The method achieves the generation of depth images without needing to label the image content of the first image, thus avoiding the need to label the image content of sample images with natural language when training the image generation model.

[0225] In one application scenario of this application, a computer device is equipped with a virtual environment development program. The first image is an image of the appearance of a virtual object in the environment, also referred to as a virtual object image, provided to the virtual environment development program. The virtual environment development program is used to construct a three-dimensional virtual environment. Accordingly,

[0226] • Obtain images and guidance text for virtual items;

[0227] For example, the virtual object image is an image from which depth information is to be extracted. The depth information is information describing the distance between pixels in the virtual object image and the imaging plane of the virtual object image.

[0228] For example, guiding text is used to indicate the relationship between image grayscale and depth information; for example, guiding text is also called style guiding text or image style guiding text, which is used to constrain the desired image style to be an image style in which grayscale and depth information are correlated; for example, guiding text describes the relationship between image grayscale and depth information in natural language.

[0229] For example, guiding text is used to indicate the relationship between image grayscale and depth information; for instance, as depth information increases, the distance between a pixel and the imaging plane (or camera) in three-dimensional space increases, and the grayscale of the resulting image is expected to increase as well.

[0230] • Input the virtual object image into the structure prediction model to obtain structural features;

[0231] For example, the structure prediction model is used to predict structural features in a virtual object image, where the structural features indicate at least one of the features of the object in the virtual object image, such as shape, layout, and texture. For example, the structure prediction model can be implemented as an artificial neural network model, predicting structural features through at least one of the operations such as encoding, pooling, decoding, and convolution.

[0232] • The first generation sub-model in the image generation model is used to perform encoding prediction on the guiding text to obtain the hidden layer features corresponding to the association relationship;

[0233] For example, the first generating sub-model is used to convert the guiding text from text format into at least one data format among feature values, feature vectors, and feature matrices, to describe the feature information of the guiding text in the hidden layer space.

[0234] • The second generation sub-model in the image generation model is used to perform image depth prediction based on structural features and hidden layer features to obtain the depth image corresponding to the virtual object image;

[0235] For example, the second generative sub-model is used to predict a depth image using structural features and hidden layer features as input information.

[0236] For example, hidden layer features provide image style constraint information for the second generation sub-model, instructing it to generate a depth image according to the desired image style indicated by the guiding text. The depth image has an image style where there is a correspondence between depth information and image grayscale. For example, structural features provide image content constraint information for the second generation sub-model, instructing it to generate a depth image according to the structural features of the virtual object image.

[0237] Optionally, the structural prediction model includes at least two prediction network layers, and the second generative sub-model in the image generation model includes at least two sub-networks; correspondingly,

[0238] • Use a structural prediction model to perform predictions based on virtual object images to obtain at least two structural sub-features, where each of the at least two structural sub-features is the output information corresponding to at least two prediction network layers in the structural prediction model.

[0239] • Use at least two sub-networks in the second generative sub-model to perform predictions based on hidden layer features and one of the at least two structural sub-features, respectively, to obtain at least two structural prediction tensors. The at least two structural prediction tensors are the output information of at least two sub-networks in the second generative sub-model.

[0240] The second generative sub-model includes at least two sub-networks, including a first sub-network and a second sub-network that are cascaded together.

[0241] The first sub-network is used to perform predictions based on hidden layer features and the first sub-feature from at least two structural sub-features to obtain the first structural prediction tensor;

[0242] The second subnetwork is used to perform prediction based on the first structure prediction tensor and the second sub-features from at least two structure sub-features to obtain the second structure prediction tensor;

[0243] The depth image is determined based on the second structure prediction tensor.

[0244] • Determine the depth image corresponding to the virtual object image based on at least one structure prediction tensor.

[0245] Optionally, the method also includes:

[0246] • Using an image style model to perform predictions based on guiding text, the style features of the desired image style are obtained. The desired image style is an image style in which grayscale and depth information are correlated.

[0247] The depth image is obtained by the second generative sub-model through image prediction based on structural features, hidden layer features, and style features.

[0248] The training process for image style models also includes:

[0249] Obtain sample depth images and labeled text;

[0250] The initial image prediction model is used to perform predictions based on the labeled text to obtain the predicted depth image;

[0251] Based on the difference between the predicted depth image and the sample depth image, the initial image prediction model is trained to obtain the image style model in the image prediction model.

[0252] Optionally, the method also includes:

[0253] Perform image enhancement on the virtual item image to obtain the enhanced image corresponding to the virtual item image;

[0254] Edge extraction is performed on the enhanced image to obtain the second image;

[0255] Image enhancement is used to instruct the virtual item image to undergo grayscale conversion and / or resolution correction.

[0256] Optionally, the method further includes: constructing a three-dimensional structural model of the image object in the virtual item image based on the depth image. In this embodiment, the three-dimensional structural model is used to construct the virtual environment and to indicate the three-dimensional structure of virtual items in the virtual environment. For example, as described above, since the depth image carries depth information of the image object at one angle, a classifier model is needed during the construction of the three-dimensional structural model of the virtual item to determine what kind of item the depth information or the image appearance in the virtual item image represents, i.e., to obtain the item label. For example, the description of the virtual item is supplemented based on the descriptive information of the item label from the large language model. The three-dimensional structural model is constructed jointly based on depth information and descriptive information. The depth information directly provides the structural information of the virtual item at one angle; the structure of the virtual item at other angles is predicted jointly based on the depth information and descriptive information.

[0257] In another application scenario of this application, the computer device is equipped with an image processing program, such as an image editing software. The first image is the image to be edited provided to the image processing program, also known as the initial image. Correspondingly,

[0258] • Obtain the initial image and guide text;

[0259] For example, the initial image is the image from which depth information is to be extracted. The depth information is information describing the distance between pixels in the initial image and the imaging plane of the initial image.

[0260] For example, guiding text is used to indicate the relationship between image grayscale and depth information; for example, guiding text is also called style guiding text or image style guiding text, which is used to constrain the desired image style to be an image style in which grayscale and depth information are correlated; for example, guiding text describes the relationship between image grayscale and depth information in natural language.

[0261] For example, guiding text is used to indicate the relationship between image grayscale and depth information; for instance, as depth information increases, the distance between a pixel and the imaging plane (or camera) in three-dimensional space increases, and the grayscale of the resulting image is expected to increase as well.

[0262] • Input the initial image into the structure prediction model to obtain structural features;

[0263] For example, the structure prediction model is used to predict structural features in an initial image, whereby the structural features indicate at least one of the features of an object in the initial image, such as shape, layout, and texture. For example, the structure prediction model can be implemented as an artificial neural network model that predicts structural features through at least one of the operations such as encoding, pooling, decoding, and convolution.

[0264] • The first generation sub-model in the image generation model is used to perform encoding prediction on the guiding text to obtain the hidden layer features corresponding to the association relationship;

[0265] For example, the first generating sub-model is used to convert the guiding text from text format into at least one data format among feature values, feature vectors, and feature matrices, to describe the feature information of the guiding text in the hidden layer space.

[0266] • The second generation sub-model in the image generation model performs image depth prediction based on structural features and hidden layer features to obtain the depth image corresponding to the initial image;

[0267] For example, the second generative sub-model is used to predict a depth image using structural features and hidden layer features as input information.

[0268] For example, hidden layer features provide image style constraint information for the second generation sub-model, instructing it to generate a depth image according to the desired image style indicated by the guiding text. The depth image has an image style where there is a correspondence between depth information and image grayscale. For example, structural features provide image content constraint information for the second generation sub-model, instructing it to generate a depth image according to the structural features of the initial image.

[0269] Optionally, the structural prediction model includes at least two prediction network layers, and the second generative sub-model in the image generation model includes at least two sub-networks; correspondingly,

[0270] • Use the structural prediction model to perform predictions based on the initial image to obtain at least two structural sub-features, where the at least two structural sub-features are the output information corresponding to at least two prediction network layers in the structural prediction model;

[0271] • Use at least two sub-networks in the second generative sub-model to perform predictions based on hidden layer features and one of the at least two structural sub-features, respectively, to obtain at least two structural prediction tensors. The at least two structural prediction tensors are the output information of at least two sub-networks in the second generative sub-model.

[0272] The second generative sub-model includes at least two sub-networks, including a first sub-network and a second sub-network that are cascaded together.

[0273] The first sub-network is used to perform predictions based on hidden layer features and the first sub-feature from at least two structural sub-features to obtain the first structural prediction tensor;

[0274] The second subnetwork is used to perform prediction based on the first structure prediction tensor and the second sub-features from at least two structure sub-features to obtain the second structure prediction tensor;

[0275] The depth image is determined based on the second structure prediction tensor.

[0276] • Determine the depth image corresponding to the initial image based on at least one structure prediction tensor.

[0277] Optionally, the method also includes:

[0278] • Using an image style model to perform predictions based on guiding text, the style features of the desired image style are obtained. The desired image style is an image style in which grayscale and depth information are correlated.

[0279] The depth image is obtained by the second generative sub-model through image prediction based on structural features, hidden layer features, and style features.

[0280] The training process for image style models also includes:

[0281] Obtain sample depth images and labeled text;

[0282] The initial image prediction model is used to perform predictions based on the labeled text to obtain the predicted depth image;

[0283] Based on the difference between the predicted depth image and the sample depth image, the initial image prediction model is trained to obtain the image style model in the image prediction model.

[0284] Optionally, the method also includes:

[0285] Perform image enhancement on the initial image to obtain the enhanced image corresponding to the initial image;

[0286] Edge extraction is performed on the enhanced image to obtain the second image;

[0287] Image enhancement is used to instruct the initial image to undergo grayscale conversion and / or resolution correction.

[0288] Optionally, the method also includes: constructing a three-dimensional structural model of the image objects in the initial image based on the depth image.

[0289] Furthermore, it also includes: constructing a color map of the three-dimensional structure model based on the color information of each pixel in the original image to obtain a shading model;

[0290] A virtual camera is used to capture the view of the shaded model from a corrected angle, resulting in a corrected image.

[0291] In this embodiment, the image processing program, such as an image editing software, provides a function to convert the angle of an object in an image. This function can change the angle at which the object in the first image is facing to obtain a corrected image. The corrected angle when the virtual camera captures the corrected image is different from the initial angle at which the virtual camera can capture the initial image. This provides a new image processing function. The first image carries certain perspective relationships and other information through color information on a two-dimensional plane; that is, spatial structure information is contained within the color information. The image processing program provides a function to process this spatial structure information, thereby converting the angle of the object in the image.

[0292] Those skilled in the art will understand that the above embodiments can be implemented independently, or the above embodiments can be freely combined to create new embodiments to implement the image processing method of this application.

[0293] Figure 12 A structural block diagram of an image processing apparatus provided in an exemplary embodiment of this application is shown.

[0294] The device includes:

[0295] The acquisition module 810 is used to acquire a first image and guiding text, wherein the guiding text is used to indicate the correlation between image grayscale and depth information;

[0296] Processing module 820 is used to input the first image into the structure prediction model to obtain structural features;

[0297] The processing module 820 is further configured to use the first generation sub-model in the image generation model to perform encoding prediction on the guiding text to obtain the hidden layer features corresponding to the association relationship;

[0298] The processing module 820 is further configured to use the second generation sub-model in the image generation model to perform image depth prediction based on the structural features and the hidden layer features, so as to obtain a depth image corresponding to the first image.

[0299] In an optional implementation of this embodiment, the structural prediction model includes at least two prediction network layers, and the processing module 820 is further configured to:

[0300] The first image is input into the structure prediction model to obtain at least two structural sub-features, wherein the at least two structural sub-features are the output information corresponding to at least two prediction network layers in the structure prediction model;

[0301] The second generation sub-model in the image generation model includes at least two sub-networks, and the processing module 820 is further configured to:

[0302] At least two sub-networks in the second generative sub-model are used to perform predictions based on the hidden layer features and one of the at least two structural sub-features, respectively, to obtain at least two structural prediction tensors, wherein the at least two structural prediction tensors are the output information corresponding one-to-one with the at least two sub-networks in the second generative sub-model;

[0303] The depth image corresponding to the first image is determined based on at least one structural prediction tensor.

[0304] In an optional implementation of this embodiment, at least two sub-networks in the second generated sub-model include a first sub-network and a second sub-network that are cascaded sequentially.

[0305] The processing module 820 is further configured to:

[0306] The first sub-network is used to perform prediction based on the hidden layer features and the first sub-feature among the at least two structural sub-features to obtain the first structural prediction tensor;

[0307] The second sub-network is used to perform prediction based on the first structure prediction tensor and the second sub-feature among the at least two structure sub-features to obtain the second structure prediction tensor;

[0308] The depth image is determined based on the second structure prediction tensor.

[0309] In an optional implementation of this embodiment, the processing module 820 is further configured to:

[0310] A weighted summation is performed on the structural features and the hidden layer features to obtain the fused features;

[0311] The second generation sub-model in the image generation model is used to perform image depth prediction based on the fusion features to obtain the depth image corresponding to the first image.

[0312] In an optional implementation of this embodiment, the processing module 820 is further configured to:

[0313] Using an image style model to perform prediction based on the guiding text, the style features corresponding to the correlation between the image grayscale and the depth information are obtained;

[0314] The depth image is obtained by the second generation sub-model performing image prediction based on the structural features, the hidden layer features, and the style features.

[0315] In an optional implementation of this embodiment, the processing module 820 is further configured to perform any one of the following:

[0316] Using the third sub-network in the second generative sub-model, image prediction is performed based on the structural features, the hidden layer features, and the style features to obtain the depth image;

[0317] Using the fourth sub-network in the second generative sub-model, a prediction is performed based on the structural features and the hidden layer features to obtain a structural prediction tensor; and using the fifth sub-network in the second generative sub-model, a prediction is performed based on the structural prediction tensor and the style features to obtain the depth image;

[0318] Using the sixth sub-network in the second generative sub-model, a prediction is performed based on the style features and the hidden layer features to obtain a style prediction tensor; and using the seventh sub-network in the second generative sub-model, a prediction is performed based on the style prediction tensor and the structural features to obtain the depth image.

[0319] In an optional implementation of this embodiment, the image style model includes at least two style network layers, and the processing module 820 is further configured to:

[0320] Using the image style model to perform prediction based on the guiding text, at least two style sub-features are obtained, and the at least two style sub-features are the output information corresponding to at least two style network layers in the image style model;

[0321] In the second generative sub-model, at least two generative network layers perform predictions based on the hidden layer features and one of the at least two style sub-features to obtain the depth image; the at least two generative network layers belong to the third, fifth, or sixth sub-network.

[0322] In an optional implementation of this embodiment, the acquisition module 810 is further configured to:

[0323] Obtain sample depth images and labeled text;

[0324] The processing module 820 is further configured to:

[0325] The initial image prediction model is used to perform prediction based on the labeled text to obtain a predicted depth image;

[0326] The device further includes:

[0327] Training module 830 is used to train the initial image prediction model based on the difference between the predicted depth image and the sample depth image to obtain the image style model in the image prediction model.

[0328] In an optional implementation of this embodiment, the processing module 820 is further configured to:

[0329] Edge extraction is performed on the first image to obtain a second image, which is a line drawing image corresponding to the first image;

[0330] The structural features are obtained by the structural prediction model based on the second image.

[0331] In an optional implementation of this embodiment, the processing module 820 is further configured to:

[0332] Perform image enhancement on the first image to obtain the enhanced image corresponding to the first image;

[0333] Edge extraction is performed on the enhanced image to obtain the second image;

[0334] The image enhancement is used to instruct the first image to undergo grayscale conversion and / or resolution correction.

[0335] In an optional implementation of this embodiment, the processing module 820 is further configured to:

[0336] Based on the depth image, a three-dimensional structural model of the image object in the first image is constructed.

[0337] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0338] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant method; the technical effects achieved by each module performing its operation are the same as the technical effects in the embodiments of the relevant method, and will not be elaborated here.

[0339] This application also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the image processing methods provided in the above-described method embodiments.

[0340] Alternatively, the computer device is a server. For example, Figure 13 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0341] Typically, server 2300 includes a processor 2301 and memory 2302.

[0342] Processor 2301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2301 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 2301 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2301 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2301 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0343] The memory 2302 may include one or more computer-readable storage media, which may be non-transitory. The memory 2302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2302 are used to store at least one instruction, which is executed by the processor 2301 to implement the image processing method provided in the method embodiments of this application.

[0344] In some embodiments, the server 2300 may optionally include an input interface 2303 and an output interface 2304. The processor 2301, memory 2302, and input interfaces 2303 and 2304 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interfaces 2303 and 2304 via a bus, signal lines, or a circuit board. The input interfaces 2303 and 2304 can be used to connect at least one input / output (I / O) related peripheral device to the processor 2301 and memory 2302. In some embodiments, the processor 2301, memory 2302, and input interfaces 2303 and 2304 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2301, memory 2302, and input interfaces 2303 and 2304 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0345] Those skilled in the art will understand that the structure shown above does not constitute a limitation on server 2300, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0346] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuitry and / or program instructions, which, when the chip is run on a computer device, are used to implement the image processing method described above.

[0347] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the image processing methods provided in the above-described method embodiments.

[0348] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program that is loaded and executed by a processor to implement the image processing methods provided in the above-described method embodiments.

[0349] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0350] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0351] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method is performed by a computer device, and the method includes: Acquire a first image and guiding text, wherein the guiding text is used to indicate the correlation between image grayscale and depth information; The first image is input into the structure prediction model to obtain structural features; The first generation sub-model in the image generation model is used to perform encoding prediction on the guiding text to obtain the hidden layer features corresponding to the association relationship; The second generation sub-model in the image generation model is used to perform image depth prediction based on the structural features and the hidden layer features to obtain the depth image corresponding to the first image.

2. The method according to claim 1, characterized in that, The structure prediction model includes at least two prediction network layers. The step of inputting the first image into the structure prediction model to obtain structural features includes: The first image is input into the structure prediction model to obtain at least two structural sub-features, wherein the at least two structural sub-features are the output information corresponding to at least two prediction network layers in the structure prediction model; The second generation sub-model in the image generation model includes at least two sub-networks. The step of using the second generation sub-model in the image generation model to perform image depth prediction based on the structural features and the hidden layer features to obtain a depth image corresponding to the first image includes: At least two sub-networks in the second generative sub-model are used to perform predictions based on the hidden layer features and one of the at least two structural sub-features, respectively, to obtain at least two structural prediction tensors, wherein the at least two structural prediction tensors are the output information corresponding one-to-one with the at least two sub-networks in the second generative sub-model; The depth image corresponding to the first image is determined based on at least one structural prediction tensor.

3. The method according to claim 2, characterized in that, The second generative sub-model includes at least two sub-networks, including a first sub-network and a second sub-network that are cascaded together. The step of using at least two sub-networks in the second generative sub-model to perform predictions based on the hidden layer features and one of the at least two structural sub-features, respectively, to obtain at least two structural prediction tensors, includes: The first sub-network is used to perform prediction based on the hidden layer features and the first sub-feature among the at least two structural sub-features to obtain the first structural prediction tensor; The second sub-network is used to perform prediction based on the first structure prediction tensor and the second sub-feature among the at least two structure sub-features to obtain the second structure prediction tensor; The depth image is determined based on the second structure prediction tensor.

4. The method according to claim 1, characterized in that, The step of using the second generation sub-model in the image generation model to perform image depth prediction based on the structural features and the hidden layer features to obtain the depth image corresponding to the first image includes: A weighted summation is performed on the structural features and the hidden layer features to obtain the fused features; The second generation sub-model in the image generation model is used to perform image depth prediction based on the fusion features to obtain the depth image corresponding to the first image.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Using an image style model to perform prediction based on the guiding text, the style features corresponding to the correlation between the image grayscale and the depth information are obtained; The depth image is obtained by the second generation sub-model performing image prediction based on the structural features, the hidden layer features, and the style features.

6. The method according to claim 5, characterized in that, The step of using the second generation sub-model in the image generation model to perform image depth prediction based on the structural features and the hidden layer features to obtain the depth image corresponding to the first image includes any one of the following: Using the third sub-network in the second generative sub-model, image prediction is performed based on the structural features, the hidden layer features, and the style features to obtain the depth image; Using the fourth sub-network in the second generative sub-model, a prediction is performed based on the structural features and the hidden layer features to obtain a structural prediction tensor; and using the fifth sub-network in the second generative sub-model, a prediction is performed based on the structural prediction tensor and the style features to obtain the depth image; Using the sixth sub-network in the second generative sub-model, a prediction is performed based on the style features and the hidden layer features to obtain a style prediction tensor; and using the seventh sub-network in the second generative sub-model, a prediction is performed based on the style prediction tensor and the structural features to obtain the depth image.

7. The method according to claim 5, characterized in that, The method further includes: Obtain sample depth images and labeled text; The initial image prediction model is used to perform prediction based on the labeled text to obtain a predicted depth image; Based on the difference between the predicted depth image and the sample depth image, the initial image prediction model is trained to obtain the image style model in the image prediction model.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Edge extraction is performed on the first image to obtain a second image, which is a line drawing image corresponding to the first image; The structural features are obtained by the structural prediction model based on the second image.

9. The method according to claim 8, characterized in that, The step of performing edge extraction on the first image to obtain the second image includes: Perform image enhancement on the first image to obtain the enhanced image corresponding to the first image; Edge extraction is performed on the enhanced image to obtain the second image; The image enhancement is used to instruct the first image to undergo grayscale conversion and / or resolution correction.

10. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on the depth image, a three-dimensional structural model of the image object in the first image is constructed.

11. An image processing apparatus, characterized in that, The device includes: An acquisition module is used to acquire a first image and guiding text, wherein the guiding text is used to indicate the correlation between image grayscale and depth information; The processing module is used to input the first image into the structure prediction model to obtain structural features; The processing module is further configured to perform encoding prediction on the guiding text using the first generation sub-model in the image generation model to obtain the hidden layer features corresponding to the association relationship; The processing module is further configured to use the second generation sub-model in the image generation model to perform image depth prediction based on the structural features and the hidden layer features, so as to obtain a depth image corresponding to the first image.

12. A computer device, characterized in that, The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is configured to execute the at least one program in the memory to implement the image processing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the image processing method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, and a processor reads from and executes the computer instructions to implement the image processing method as described in any one of claims 1 to 10.

Images