Image processing method and device, electronic equipment, storage medium and product

Abstract

The invention provides an image processing method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: for a first feature point of a first object in a first image, determining first position information of the first feature point in a first coordinate system, and for a second feature point corresponding to the first feature point in a second image, determining second position information of the second feature point in the first coordinate system, the resolution of the first image is greater than the resolution of the second image, determining a first offset of the second feature point relative to the first feature point in the first coordinate system based on the first position information and the second position information, extracting a first feature of the first object from the first image, and extracting a second feature of the first object from the second image based on the first offset and the first feature. And constructing a second feature of the first object corresponding to the second image, and generating a third image including the first object based on the second feature. According to the invention, the definition and the display effect of the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, storage medium and product. Background Art

[0002] In the related art, frequency domain methods can be used for image processing. For example, an image can be converted from space to frequency domain through Fourier transform, and then the frequency components of the image can be analyzed. A filter can be used to filter out noise in the image with a frequency greater than a frequency threshold. However, the use of frequency domain methods for image processing will result in filtering out the edges and details of the image, resulting in image blur and loss of details. In other words, the use of frequency domain methods for image processing will result in a decrease in the display effect of the image. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, electronic device, storage medium and product, which can improve the clarity and display effect of an image.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides an image processing method, the method comprising:

[0006] For a first feature point of a first object in a first image, determine first position information of the first feature point in a first coordinate system, and for a second feature point in a second image corresponding to the first feature point, determine second position information of the second feature point in the first coordinate system;

[0007] wherein the resolution of the first image is greater than the resolution of the second image;

[0008] Determine, in the first coordinate system, a first offset of the second feature point relative to the first feature point based on the first position information and the second position information;

[0009] Extracting a first feature of the first object from the first image, and constructing a second feature of the first object corresponding to the second image based on the first offset and the first feature;

[0010] Based on the second feature, a third image including the first object is generated.

[0011] The present application provides an image processing device, the device comprising:

[0012] a determination module, configured to determine, for a first feature point of a first object in a first image, first position information of the first feature point in a first coordinate system, and, for a second feature point in a second image corresponding to the first feature point, determine second position information of the second feature point in the first coordinate system;

[0013] wherein the resolution of the first image is greater than the resolution of the second image;

[0014] a determination module, further configured to determine, in the first coordinate system, a first offset of the second feature point relative to the first feature point based on the first position information and the second position information;

[0015] a construction module, configured to extract a first feature of the first object from the first image, and construct a second feature of the first object corresponding to the second image based on the first offset and the first feature;

[0016] A generating module is used to generate a third image including the first object based on the second feature.

[0017] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0018] A memory for storing computer executable instructions or computer programs;

[0019] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the image processing method provided by the embodiment of the present application is implemented.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] The image processing method provided in the embodiment of the present application can determine the first position information of the first feature point of the first object in the first image, and determine the second position information of the second feature point in the second image, wherein the resolution of the first image is greater than the resolution of the second image, and thus the first offset of the second feature point relative to the first feature point can be determined, and based on the first offset and the first feature of the first object extracted from the first image, the second feature of the first object corresponding to the second image is constructed, so that based on the second feature, a third image including the first object can be generated.

[0024] In the present application, the first feature is extracted from a first image with higher resolution, and the first feature has relatively rich details of the first object. The first offset is the offset of the first object in the second image relative to the first object in the first image, and the first offset can characterize the characteristics of the first object in the second image. The second feature constructed based on the first feature and the first offset can not only characterize the characteristics of the first object in the second image, but also has relatively rich details of the first object. Accordingly, the third image generated based on the second feature can include both the characteristics of the first object in the second image and relatively rich details of the first object. Compared with the related art, there is no image blurring and detail loss, and the image clarity and display effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 1 ;

[0028] Figure 4 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 2 ;

[0029] Figure 5 is a schematic diagram of third location information provided in an embodiment of the present application;

[0030] Figure 6 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 3 ;

[0031] Figure 7 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 4 ;

[0032] Figure 8is a structural diagram of an image processing model provided in an embodiment of the present application;

[0033] Fig. 9 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 5 ;

[0034] Fig.10 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 6 . DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0039] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0040] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0041] The embodiments of the present application provide an image processing method, an image processing device, an electronic device, a computer-readable storage medium, and a computer program product. The generated third image can include the characteristics of the first object in the second image and has relatively rich details of the first object, thereby improving the clarity and display effect of the image.

[0042] See also Figure 1 , Figure 1 is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application, Figure 1 The image processing system 100 shown can implement an image processing application. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0043] When the user needs to process an image whose resolution is less than a preset resolution threshold, that is, when the user needs to process the second image, the terminal 400 can generate an image processing request for processing the second image in response to an image processing instruction triggered by the graphical interface 410-1 of the terminal 400, and the terminal 400 can send the image processing request to the server 200.

[0044] After receiving the image processing request, the server 200 can determine the first position information of the first feature point in the first coordinate system for the first feature point of the first object in the first image, and determine the second position information of the second feature point in the first coordinate system for the second feature point corresponding to the first feature point in the second image, wherein the resolution of the first image is greater than the resolution of the second image.

[0045] After the server 200 determines the first position information and the second position information, it can determine the first offset of the second feature point relative to the first feature point in the first coordinate system based on the first position information and the second position information, extract the first feature of the first object from the first image, and construct the second feature of the first object corresponding to the second image based on the first offset and the first feature, and generate a third image including the first object based on the second feature.

[0046] In the present application, the second feature constructed based on the first feature and the first offset can not only characterize the characteristics of the first object in the second image, but also has relatively rich details of the first object. Correspondingly, the third image generated based on the second feature can include both the characteristics of the first object in the second image and relatively rich details of the first object. Compared with the related art, there is no image blur and detail loss, which can improve the image clarity and display effect.

[0047] The electronic device provided in the embodiment of the present application is introduced below. The electronic device that implements the image processing method in the embodiment of the present application can be a terminal, a server, or a combination of the two. The terminal can be various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a car terminal.

[0048] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0049] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0050] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0052] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0053] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0054] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0055] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0056] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0057] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0058] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0059] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 An image processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: a determination module 4551, a construction module 4552, and a generation module 4553. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0060] In other embodiments, the image processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the image processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0061] In some embodiments, the terminal or server can implement the image processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as an APP with a face recognition function, an APP that can process images; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0062] See also Figure 3 , Figure 3 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 1As mentioned above, the electronic device implementing the image processing method of the embodiment of the present application may be a terminal, a server, or a combination of the two. Figure 3 The steps shown take the electronic device being a server as an example to introduce the image processing method provided in the embodiment of the present application.

[0063] In step 101, for a first feature point of a first object in a first image, first position information of the first feature point in a first coordinate system is determined, and for a second feature point corresponding to the first feature point in a second image, second position information of the second feature point in the first coordinate system is determined.

[0064] In actual application, when a user needs to process an image whose resolution is lower than a preset resolution threshold, the first image and the second image can be acquired in response to an image processing instruction triggered by a display interface based on the server.

[0065] Alternatively, when a user needs to process an image whose resolution is lower than a preset resolution threshold, the terminal can send an image processing request to the server in response to an image processing instruction issued based on the display interface of the terminal. After receiving the image processing request, the server can obtain the first image and the second image.

[0066] The first image and the second image include the same object, and the same object may be one or more of the same creature, the same object, and the same building. When the first image and the second image include only one object, the same object may be the first object, that is, the first image and the second image both include the first object.

[0067] When the first image and the second image both include multiple objects, that is, when the number of identical objects is multiple, the first image and the second image both include the first object, the second object, the third object, etc., and the number of identical objects can be determined based on actual usage.

[0068] In the present application, the first image and the second image may include one or more identical objects. The identical object may be any entity that can be captured by a camera, for example, the identical object may be a building such as a house, a bridge, a tower, or a living thing such as a person, an animal, or a plant, or an object such as a vehicle, a table, or a water cup, and the specific setting may be based on actual usage requirements.

[0069] The processing method for each identical object is the same. The following takes the first object as an example. Figure 3Step 101 is shown to illustrate that in some embodiments, the first image is an image with a preset resolution greater than a resolution threshold, wherein the first image is equivalent to a reference image for improving the image resolution, and the first object is in a first posture in the first image. For other objects, the process of processing the first object can be equivalently replaced.

[0070] When the first object is a building, the first posture may represent the structural layout of the building, the direction of the building, the relationship between the building and the environmental elements in the image, etc. When the first object is a creature, the first posture may be the body posture of the creature, the facial expression of the creature, etc. When the first object is an object, the first posture may be the shape of the object, the size of the object, the direction of the object, the position of the object, etc. The specific setting may be based on actual usage.

[0071] The number of the second images may be one or more. In the second images, the first object is in a second posture different from the first posture. For the second posture, please refer to the above description of the first posture, which will not be repeated here. For different second images, the second posture of the first object in the second image is different.

[0072] For example, the number of second images is 3, specifically second image 1, second image 2, and second image 3. In second image 1, the target object is in second posture 1, in the second image, the target object is in second posture 2, and in second image 1, the target object is in second posture 3. Second posture 1, second posture 2, and second posture 3 are all different.

[0073] The second image is an image with a resolution less than a preset resolution threshold. In some embodiments, the resolution threshold includes a first resolution threshold and a second resolution threshold. The first image is an image with a resolution greater than the first resolution threshold, and the second image is an image with a resolution less than the second resolution threshold, wherein the second resolution threshold is less than the first resolution threshold.

[0074] For ease of understanding, the following describes the method of acquiring the second image in combination with different usage scenarios. For face recognition scenarios, the second image may be an image acquired by an image acquisition device with a lower resolution. When the second image needs to be acquired, one or more images acquired by an image acquisition device with a lower resolution may be acquired as the second image, or a video acquired by an image acquisition device with a lower resolution may be acquired, thereby extracting one or more image frames from the video as the second image.

[0075] For the image restoration scenario, the second image can be acquired in advance. For the video generation scenario, a first video including the first object can be generated based on the first image and the pre-set video prompt information of the first image, and an image frame with a resolution less than a resolution threshold is selected from the first video, and the selected image frame is used as the second image.

[0076] The video prompt information of the first image may include one or more of information characterizing the scene where the first object is located, information describing the posture of the first object, and information describing the acquisition angle. The video prompt information may be audio and / or text.

[0077] The following takes the first object being a human face as an example to illustrate the method of obtaining the second image in the image generation scene. The video prompt information is text, and the video prompt information may include information representing the expression of the human face (corresponding to the information describing the posture of the first object) and information representing the scene in which the human face is located (corresponding to the information representing the scene in which the first object is located).

[0078] The first feature point of the face in the first image (i.e., the first facial key point) can be extracted, the text features of the video prompt information can be extracted, and a video for the face can be generated based on the text features and the facial key points, wherein the video includes multiple image frames of the face, i.e., multiple image frames of the first object.

[0079] After generating a video including the first object, in some embodiments, the resolutions of the image frames in the video are all less than a resolution threshold, and therefore, one or more second images may be randomly selected from the multiple image frames including the first object.

[0080] In some embodiments, image frames in a video may be divided based on objects to obtain image frames of different objects. For each object, an image frame having a resolution less than a resolution threshold may be selected from the image frames of the object, and the selected image frame may be used as a second image corresponding to the object.

[0081] For example, if a video includes a first object and a second object, the image frames in the video can be divided to obtain image frames including the first object and image frames including the second object. For the first object, image frames with a resolution less than a resolution threshold can be selected from the image frames of the first object, and the selected image frames are used as the second images corresponding to the first object. For the second object, image frames with a resolution less than a resolution threshold can be selected from the image frames of the second object, and the selected image frames are used as the second images corresponding to the second object.

[0082] In some embodiments, according to the requirement for the frame rate of the video after the resolution is increased, a first number of second images can be extracted from the multiple images including the first object, wherein the first number is determined according to the frame rate and the number of frames of the image in the video. The specific setting can be based on actual usage requirements.

[0083] After acquiring the first image and the second image, the first feature point of the first object in the first image can be identified, and for the first feature point of the first object in the first image, the first position information of the first feature point in the first coordinate system can be determined. The first coordinate system is a three-dimensional rectangular coordinate system, that is, a world coordinate system.

[0084] The second feature point of the first object in the second image can also be identified. The second feature point corresponds to the first feature point. For the second feature corresponding to the first feature point in the second image, the second position information of the second feature point in the first coordinate system is determined. That is, for the second feature point in the second image, the second position information of the second feature point in the first coordinate system is determined.

[0085] The number of the first feature points and the number of the second feature points are both multiple, and the number of the first feature points may be equal to or different from the number of the second feature points. One first feature point corresponds to one first position information, and one second feature point corresponds to one second position information. The number of the first position information may be multiple, and the number of the second position information may also be multiple.

[0086] In some embodiments, a first feature point of a first object in a first image can be identified, and the first two-dimensional position information of the first feature point of the first object in the first image can be determined. Then, based on a three-dimensional deformable model (3D MorphableModel) and the first image, a three-dimensional face mesh can be reconstructed, and the three-dimensional face mesh can be mapped to a two-dimensional image space. The correspondence between the points on the three-dimensional face mesh and the first image is determined, and the first position information corresponding to the first two-dimensional position information is determined, that is, the first position information of the first feature point in the first coordinate system is determined.

[0087] Accordingly, the second feature point of the first object in the second image can be identified, the second feature point corresponds to the first feature point, and the second two-dimensional position information of the second feature point of the first object in the second image is determined, and then based on the three-dimensional deformation model and the second image, the three-dimensional face mesh is reconstructed, the three-dimensional face mesh is mapped to the two-dimensional image space, the correspondence between the points on the three-dimensional face mesh and the second image is determined, and the second position information corresponding to the second two-dimensional position information is determined, that is, the second position information of the second feature point in the first coordinate system is determined. In this way, accurate first position information and second position information can be determined, and the first position information and second position information are used to subsequently generate a third image, thereby improving the accuracy of generating the third image.

[0088] In step 102 , based on the first position information and the second position information, a first offset of the second feature point relative to the first feature point is determined in the first coordinate system.

[0089] In some embodiments, step 102 can be implemented in the following manner: the first position information and the second position information can both be in the form of three-dimensional coordinates, there is a correspondence between the first feature point and the second feature point, and there is a correspondence between the first position information of the first feature point and the second position information of the second feature point, that is, there is a correspondence between the first position information and the second position information.

[0090] The correspondence between the first position information and the second information means that the part of the object represented by the first feature point of the first object corresponding to the first position information is consistent with the part of the object represented by the second feature point of the first object corresponding to the second position information.

[0091] For example, when the first object is a human face, the first feature point of the face corresponding to the first position information can represent the right corner of the eye of the face, that is, the first key point of the face corresponding to the first position information can represent the right corner of the eye of the face, and there is a corresponding relationship between the first position information and the second position information, and the second feature point of the face corresponding to the second position information can represent the right corner of the eye of the face, that is, the second key point of the face corresponding to the second position information can represent the right corner of the eye of the face.

[0092] The first coordinate system has three coordinate dimensions. For each dimension, the difference between the value of the first position information in this dimension and the value of the second position information in this dimension can be determined. By combining the differences in different dimensions, the first offset of the second feature point relative to the first feature point in the first coordinate system can be obtained.

[0093] For example, the first position information can be represented by (x1, y1, z1), and the second position information can be represented by (x2, y2, z2). The dimensions of the coordinates can include the x-axis, the y-axis, and the z-axis. For the x-axis, the difference between the value of the first position information on the x-axis and the value of the second position information on the x-axis can be determined, that is, the difference between x1 and x2 can be determined.

[0094] Similarly, for the y-axis, the difference between the value of the first position information on the y-axis and the value of the second position information on the y-axis can be determined, that is, the difference between y1 and y2 can be determined. For the z-axis, the difference between the value of the first position information on the z-axis and the value of the second position information on the z-axis can be determined, and the difference between z1 and z2 can be determined. By combining the difference between x1 and x2, the difference between y1 and y2, and the difference between z1 and z2, the first offset of the second feature point relative to the first feature point (the difference between x1 and x2, the difference between y1 and y2, the difference between z1 and z2) can be obtained.

[0095] That is, the first offset is composed of the position difference between the first position information and the second position information in different dimensions in the three-dimensional coordinate system, and the first offset can be represented in the form of three-dimensional coordinates. In this way, an accurate first offset can be obtained, and the first offset can accurately represent the characteristics of the first object in the second image, thereby improving the accuracy of the subsequently generated third image.

[0096] With respect to step 103 , a first feature of the first object is extracted from the first image, and a second feature of the first object corresponding to the second image is constructed based on the first offset and the first feature.

[0097] In some embodiments, the first feature of the first object can be extracted from the first image, and the first feature of the first object can be represented in the form of three planes. For example, the first image can be input into a three-plane generator, and a feature map of 256 times 256 times 96 can be output. The feature map is divided into three groups, one group includes 32 feature maps, and the 32 feature maps are used as a feature plane, that is, three feature planes corresponding to the first image can be obtained, that is, the first feature includes three feature planes, that is, the first feature of the first object is represented in the form of three planes. The feature plane is a three-dimensional data structure containing multiple channels, where each channel is a feature map.

[0098] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 2 , the following combination Figure 4 right Figure 3 The step 103 of "constructing a second feature of the first object corresponding to the second image based on the first offset and the first feature" is illustrated for explanation.

[0099] In step 1031, the preset third position information is superimposed with the first offset to obtain fourth position information.

[0100] For ease of understanding, the preset third position information is first explained below. A starting point is set in the first coordinate system, and a ray is created based on the starting point along a preset direction, and the third position information is sampled and set on the ray. The preset direction corresponds to the field of view of the camera, and the preset direction can be determined based on the position of the camera in the first coordinate system and the orientation of the camera. The preset direction can be represented by an angle with the three-dimensional coordinate system.

[0101] For example, in the first coordinate system (three-dimensional coordinate system), it is assumed that the camera is located at the origin of the first coordinate system, that is, the dot of the first coordinate system is set as the starting point, the camera faces the positive direction of the z-axis, the horizontal field of view angle of the camera is 90 degrees, and the vertical field of view angle is also 90 degrees. The preset direction can be represented as the direction corresponding to the angle between -45 degrees and 45 degrees with the x-axis, and the angle between -45 degrees and 45 degrees with the y-axis.

[0102] The following describes a method for sampling on a ray and setting the third position information. Starting from the starting point, points are sampled on the ray at intervals of a preset distance, the position information of the sampled points in the first coordinate system is determined, and the position information corresponding to the sampled points is set as the third position information. One sampled point corresponds to one piece of third position information. The number of sampled points is multiple, and the number of third position information is multiple. The third position information can be represented in the form of three-dimensional coordinates.

[0103] For ease of understanding, the following Figure 5 The third location information provided in the embodiment of the present application is described in Figure 5 , Figure 5 is a schematic diagram of the third position information provided in the embodiment of the present application, in the first coordinate system ( Figure 5 As shown in FIG. 5 ), a starting point 501 is set, and a ray is created along a preset direction based on the starting point. Figure 5 Only rays 503 are shown, and each ray corresponds to a preset direction.

[0104] Starting from the starting point, sample points on the ray at every preset distance, determine the position information of the sampled points in the first coordinate system, set the position information corresponding to the sampled points as the third position information, and Figure 5 In the figure, the sampled points include point 502 and point 506. Point 502 is a point sampled at a first distance from the starting point, and point 506 is a point sampled at a second distance from the starting point. The second distance is greater than the first distance, and the second distance is equal to the first distance multiplied by 2.

[0105] exist Figure 5 In the figure, the dotted box 504 includes a plurality of sampled points (including point 502), and the distances between the plurality of sampled points included in the dotted box 504 and the starting point 501 are all the first distances, and the dotted box 505 includes a plurality of sampled points (including point 506), and the distances between the plurality of sampled points included in the dotted box 505 and the starting point 501 are all the second distances.

[0106] The number of the third position information can be multiple, and the number of the third position information can be determined based on the sampling number and the number of rays. The third position information can be the product of the sampling number and the number of rays, and the sampling number is the number of sampling points on the ray at a preset distance.

[0107] After setting the third position information, the pre-set third position information may be superimposed with the first offset to obtain fourth position information. In some embodiments, there is third position information with a corresponding first offset, and there is third position information without a corresponding first offset.

[0108] The third position information is used to construct a second feature of the first object of the second image. The second feature is used to generate a third image including the first object. The number of rays created is consistent with the number of pixels in the third image, and one ray corresponds to one pixel.

[0109] The third image is the second image with improved resolution. The posture of the first object in the third image is consistent with the posture of the first object in the second image. The first offset corresponds to the second feature point in the second image. The first offset corresponds to the second two-dimensional position information of the second feature point in the second image. The ray corresponding to the first offset can be determined based on the second two-dimensional position information, that is, the third position information corresponding to the first offset can be determined. The third position information corresponding to the first offset can be one or more.

[0110] For the third position information with the corresponding first offset, the third position information can be superimposed with the corresponding first offset to obtain the fourth position information. For example, the third position information is (x3, y3, z3), the first offset corresponding to the third position information is (x4, y4, z4), and the fourth position information is the sum of the values ​​of each dimension of the first offset corresponding to the third position information, that is, the fourth position information is (x3 plus x4, y3 plus y4, z3 plus z4), that is, the fourth position information can be represented in the form of three-dimensional coordinates.

[0111] For the third position information that does not have a corresponding first offset, the second offset corresponding to the third position information can be determined based on multiple first offsets by using trilinear interpolation. After determining the second offset corresponding to the third position information, the third position information can be superimposed with the second offset determined based on the first offset to obtain the fourth position information. In this way, accurate fourth position information can be obtained, and the number of fourth position information is consistent with the number of third position information.

[0112] In some embodiments, there are multiple third position information, and the third position information may be the product of the number of sampling times and the number of rays. For each ray, it may be determined whether the third position information corresponding to the ray has a corresponding first offset.

[0113] When the third position information corresponding to the ray has a first offset, each third position information can be superimposed with the first offset to obtain a superposition result corresponding to each third position information, and then the superposition results corresponding to each third position information can be fused to obtain the fourth position information.

[0114] In some embodiments, the superposition results corresponding to the third position information may be added to obtain the fourth position information. In some embodiments, the pre-set first weight may be multiplied by the superposition results corresponding to the third position information to obtain the product corresponding to the third position information, and the products corresponding to the third position information may be added to obtain the fourth position information, wherein the first weights corresponding to different third position information may be the same or different, and may be set according to actual usage requirements.

[0115] For example, see Figure 5 , for ray 503, ray 503 has a corresponding first offset, the third position information of point 502 can be superimposed with the first offset to obtain the superimposed result corresponding to the third position information of point 502, and the third position information of point 506 can be superimposed with the first offset to obtain the superimposed result corresponding to the third position information of point 506, and then the superimposed result corresponding to the third position information of point 502 and the superimposed result corresponding to the third position information of point 506 can be fused, that is, the superimposed result corresponding to the third position information of point 502 and the superimposed result corresponding to the third position information of point 506 are added to obtain the fourth position information. In this way, the number of fourth position information is consistent with the number of rays.

[0116] In step 1032, based on the fourth position information, a second feature of the first object corresponding to the second image is extracted from the first feature of the first object.

[0117] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 3 , the following combination Figure 6 The steps shown are Figure 4 Step 1032 is shown for illustration.

[0118] In step 10321, the first feature of the first object is mapped to each coordinate plane respectively to obtain the first sub-feature corresponding to the coordinate plane.

[0119] In some embodiments, the first coordinate system includes multiple coordinate planes. The first coordinate system is a three-dimensional coordinate system, including an x-axis, a y-axis, and a z-axis. The coordinate planes may include multiple coordinate planes, including a first coordinate plane, a second coordinate plane, and a third coordinate plane. The first coordinate plane may be a coordinate plane composed of an x-axis and a y-axis, the second coordinate plane may be a coordinate plane composed of an x-axis and a z-axis, and the third coordinate plane may be a coordinate plane composed of a y-axis and a z-axis.

[0120] As mentioned above, the first feature includes three feature planes, and the first feature of the first object can be mapped to each coordinate plane respectively to obtain the first sub-feature corresponding to the coordinate plane. The first sub-feature is the feature plane. For example, the first feature includes feature plane 1, feature plane 2, and feature plane 3. Feature plane 1 can be mapped to the first coordinate plane, and feature plane 1 is the first sub-feature corresponding to the first coordinate plane. Feature plane 2 can be mapped to the second coordinate plane, and feature plane 2 is the first sub-feature corresponding to the second coordinate plane. Feature plane 3 should be set to a third coordinate plane, and feature plane 3 is the first sub-feature corresponding to the third coordinate plane.

[0121] In step 10322, based on the fourth position information, feature extraction is performed on each first sub-feature to obtain a second sub-feature corresponding to each first sub-feature.

[0122] In some embodiments, the fourth position information can be represented in the form of three-dimensional coordinates, and the position sub-information corresponding to the coordinate plane in the fourth position information can be determined, and the position sub-information includes the position sub-information corresponding to the first coordinate plane, the position sub-information corresponding to the second coordinate plane, and the position sub-information corresponding to the third coordinate plane.

[0123] For example, the fourth position information is (x5, y5, z5), and the position sub-information may include position sub-information a(x5, y5) corresponding to the coordinate plane formed by the x-axis and the y-axis, position sub-information b(x5, z5) corresponding to the coordinate plane formed by the x-axis and the z-axis, and position sub-information c(y5, z5) corresponding to the coordinate plane formed by the y-axis and the z-axis.

[0124] After obtaining a plurality of position sub-information, based on the corresponding relationship between the position sub-information and the coordinate plane, feature extraction can be performed on each first sub-feature to obtain a second sub-feature corresponding to each first sub-feature.

[0125] There is a corresponding relationship between the position sub-information and the coordinate plane, there is a corresponding relationship between the coordinate plane and the first sub-feature, there is a corresponding relationship between the position sub-information and the first sub-feature, based on the position sub-information, feature extraction is performed on the first sub-feature corresponding to the position sub-information, and the feature corresponding to the extracted position sub-information is used as the second sub-feature corresponding to the first sub-feature, wherein the number of the first sub-features is consistent with the number of the second sub-features.

[0126] After obtaining the second sub-features corresponding to each first sub-feature, the second sub-features can be fused. In some embodiments, the second sub-features can be added to obtain fused features of the second sub-features. The number of fused features of the second sub-features is consistent with the number of fourth position information, and then the fused features can be spliced ​​to obtain the second feature of the first object of the second image.

[0127] In some embodiments, a pre-set second weight can be multiplied by each second sub-feature to obtain a product corresponding to the second sub-feature, and the products corresponding to the second sub-features are added to obtain a fused feature of the second sub-feature. The number of fused features of the second sub-feature is consistent with the number of fourth position information, and then the fused features can be spliced ​​to obtain the second feature of the first object of the second image.

[0128] The second weights corresponding to different second sub-features may be the same or different, and may be set according to actual usage requirements. The fourth position information may be consistent with the number of the third position information, and may also be consistent with the number of rays. Accordingly, the number of fused features of the second sub-feature may be consistent with the number of the third position information, and may also be consistent with the number of rays.

[0129] Continue to see Figure 3 In step 104, a third image including the first object is generated based on the second feature.

[0130] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 4 , the following combination Figure 7 The steps shown are Figure 3 Step 104 is shown for illustration.

[0131] In step 1041, the second feature is decoded to obtain a voxel grid corresponding to the first object.

[0132] In some embodiments, after constructing the second feature, the second feature can be decoded to obtain a voxel grid corresponding to the first object. The voxel grid is a three-dimensional grid composed of voxels, that is, the voxel grid includes multiple voxels. A voxel is a unit in a three-dimensional space. Each voxel represents a small cube and can store color information and density information.

[0133] In step 1042, color information and density information of each voxel is sampled from the voxel grid.

[0134] In some embodiments, after obtaining the voxel grid, each voxel in the voxel grid can be traversed to extract color information and density information of each voxel, wherein the color information refers to the color attribute of each voxel in the voxel grid, and the color information can be used to render the appearance of the object. The density information refers to the density attribute of each voxel in the voxel grid, and the density information can be used for volume rendering.

[0135] In step 1043 , volume rendering is performed based on the color information and density information of each voxel to obtain a third image including the first object.

[0136] In some embodiments, volume rendering is a three-dimensional graphics rendering technology that can render voxel data in three-dimensional space. Volume rendering is performed based on color information and density information of each voxel to obtain a third image including the first object, wherein the resolution of the third image is greater than the resolution of the second image.

[0137] The following is a detailed description of the process of obtaining a third image including a first object in an embodiment of the present application. In some embodiments, for the case where the number of fusion features of the second sub-features is consistent with the number of third position information, each ray corresponds to multiple fusion features of the second sub-features. For a fusion feature of a second sub-feature of a ray, the fusion feature of the second sub-feature can be decoded to obtain a voxel corresponding to the second sub-feature, and the color information and density information of the voxel can be determined. The first pixel value can be determined based on the color information and density information of the voxel. That is, multiple first pixel values ​​can be determined for a ray, and by fusing the multiple first pixel values, the second pixel value corresponding to the ray can be obtained, that is, the second pixel value at the position corresponding to the ray in the third image.

[0138] In some embodiments, a plurality of first pixel values ​​may be added to obtain a second pixel value. In some embodiments, a third weight may be pre-set, and the third weight may be multiplied by the first pixel value to obtain a product corresponding to the first pixel value, and then the products corresponding to the first pixel values ​​may be added to obtain the second pixel value.

[0139] In some embodiments, for the case where the fusion feature of the second sub-feature is consistent with the number of rays, each ray corresponds to a fusion feature of the second sub-feature, and for each fusion feature of the second sub-feature, the fusion feature of the second sub-feature can be decoded to obtain the voxel corresponding to the second sub-feature, and the color information and density information of the voxel can be determined. Based on the color information and density information of the voxel, the third pixel value can be determined, that is, a third pixel value can be determined for a ray, that is, the third pixel value of the position corresponding to the ray in the third image.

[0140] In the present application, the first feature is extracted from a first image with higher resolution, and the first feature has relatively rich details of the first object. The first offset is the offset of the first object in the second image relative to the first object in the first image, and the first offset can characterize the characteristics of the first object in the second image. The second feature constructed based on the first feature and the first offset can not only characterize the characteristics of the first object in the second image, but also has relatively rich details of the first object. Accordingly, the third image generated based on the second feature can include both the characteristics of the first object in the second image and relatively rich details of the first object. Compared with the related art, there is no image blurring and detail loss, and the image clarity and display effect can be improved.

[0141] In some embodiments, the image processing method provided in the embodiments of the present application can be implemented through an image processing model. The first image and the second image can be input into a pre-trained image processing model, and then the pre-image processing model can output a third image including the first object.

[0142] In some embodiments, the image processing model may include an input layer, a feature extraction layer, a position information acquisition layer, an offset acquisition layer, a feature construction layer, and a rendering layer. The input layer is connected to the feature extraction layer and the position information acquisition layer, the position information acquisition layer is connected to the offset acquisition layer, the feature extraction layer and the offset acquisition layer are connected to the feature construction layer, and the feature construction layer is connected to the rendering layer.

[0143] See also Figure 8 , Figure 8 Schematic diagram of the structure of the image processing model provided in the embodiment of the present application. Figure 8 In the figure, the image processing model 810 includes an input layer 801, a feature extraction layer 802, a position information acquisition layer 803, an offset acquisition layer 804, a feature construction layer 805 and a rendering layer 806.

[0144] Among them, the input layer 801 is connected to the feature extraction layer 802 and the position information acquisition layer 803, the position information acquisition layer 803 is connected to the offset acquisition layer 804, the feature extraction layer 802 and the offset acquisition layer 804 are connected to the feature construction layer 805, and the feature construction layer 805 is connected to the rendering layer 806.

[0145] The input layer is used to receive an input image, which may include a first image and a second image, wherein the resolution of the first image is greater than the resolution of the second image. The feature extraction layer is used to extract a first feature of a first object from the first image. The feature extraction layer is a pre-trained generator (Generator). The generator extracts features from the first image to obtain a feature map of the first image. The generator is composed of multiple convolutions, which is equivalent to performing multiple convolutions on the first image to obtain a feature map of the first image. The feature map of the first image can then be divided into three groups to obtain three feature planes corresponding to the first image, that is, the first feature includes three feature planes.

[0146] For example, the first image can be input into a pre-trained generator to obtain a feature map of 256 times 256 times 96 (i.e., the feature map of the first image), where 256 times 256 is the resolution of the feature map, and 96 is the number of channels of the feature map, which is equivalent to having 96 feature maps with a resolution of 256 times 256. A group includes 32 feature maps, and the 32 feature maps are regarded as a feature plane.

[0147] In some embodiments, the feature map has a pre-set number, and multiple feature maps can be sorted in ascending order of the number, and then the feature map of the first image is divided into three groups to obtain three feature planes corresponding to the first image, wherein there is a corresponding relationship between the coordinate plane and the feature plane.

[0148] For example, the number of feature maps of the first image is 96, and the corresponding numbers of the multiple feature maps are 0-95. The feature maps numbered 0-31 can be used as feature plane 1, the feature maps numbered 32-63 can be used as feature plane 2, and the feature maps numbered 64-95 can be used as feature plane 3.

[0149] The location information acquisition layer is used to perform Figure 3 Step 101, i.e., the position information acquisition layer is used to determine, for a first feature point of a first object in a first image, first position information of the first feature point in a first coordinate system, and for a second feature point in a second image corresponding to the first feature point, determine second position information of the second feature point in the first coordinate system.

[0150] The offset acquisition layer is used to perform Figure 3In step 102, the offset acquisition layer is used to determine a first offset of the second feature point relative to the first feature point in the first coordinate system based on the first position information and the second position information. The feature construction layer is used to construct a second feature of the first object corresponding to the second image based on the first offset and the first feature. The rendering layer is used to generate a third image including the first object based on the second feature.

[0151] The training process of the image processing model is described below. In some embodiments, a first sample image and a second sample image may be obtained and input into the image processing model. The resolution of the first sample image is greater than that of the second sample image. The first sample image and the second sample image both include the first sample object, and may also include the second sample object and the third sample object, etc., which may be determined according to actual conditions.

[0152] In some embodiments, the input layer may receive a first sample image and a second sample image. The input layer inputs the first sample image to the feature extraction layer. The feature extraction layer may extract features from the first sample image to obtain first sample features of the first sample image.

[0153] In some embodiments, the feature extraction layer can obtain a feature map of the first sample image, and can divide the feature map of the first image into three groups to obtain three sample feature planes corresponding to the first sample image, that is, the first sample feature includes three sample feature planes. After obtaining the three sample feature planes, a correspondence between the sample feature planes and the coordinate planes in the first coordinate system can be constructed.

[0154] The input layer can also input the second sample image to the position information acquisition layer, and the position information acquisition layer can identify the first sample feature point of the sample object in the first sample image, and determine the first sample position information of the first sample feature point in the first coordinate system for the first sample feature point of the first sample object in the first sample image.

[0155] The position information acquisition layer can identify the second sample feature point of the sample object in the second sample image, and determine the second sample position information of the second sample feature point in the first coordinate system for the second sample feature point of the first sample object in the second sample image.

[0156] The position information acquisition layer inputs the first sample position information and the second sample position information into the offset acquisition layer, and the offset acquisition layer can determine the sample offset of the second sample feature point relative to the first sample feature point in the first coordinate system based on the sample position information and the second sample position information. After obtaining the sample offset, the preset third position information can be superimposed with the sample offset to obtain the third sample position information.

[0157] The feature construction layer can obtain the third sample position information input by the position information acquisition layer, and the first sample features of the first sample image, and map the first sample features of the first sample object to each coordinate plane respectively to obtain the first sample sub-features corresponding to the coordinate planes, and based on the third sample position information, perform feature extraction on each first sample sub-feature to obtain the second sub-sample features corresponding to each first sample sub-feature, and fuse the second sample sub-features to obtain the second sample features of the first sample object corresponding to the second sample image.

[0158] Then, the rendering layer can obtain the second sample feature output by the feature construction layer, and based on the second sample feature, generate a predicted image including the first sample object, and obtain a third sample image corresponding to the second sample image. The difference between the second sample image and the third sample image is only that the resolution of the third sample image is greater than the resolution of the second sample image. Then, based on the difference between the predicted image and the second sample image, the parameters of the image processing model can be updated.

[0159] After the image processing model is trained, the image processing method provided in the embodiment of the present application can be executed based on the image processing model. The first feature is extracted from the first image with higher resolution. The first feature has relatively rich details of the first object. The first offset is the offset of the first object in the second image relative to the first object in the first image. The first offset can characterize the characteristics of the first object in the second image. The second feature constructed based on the first feature and the first offset can not only characterize the characteristics of the first object in the second image, but also has relatively rich details of the first object. Accordingly, the third image generated based on the second feature can include both the characteristics of the first object in the second image and relatively rich details of the first object. Compared with the related art, there is no image blurring and detail loss, which can improve the clarity and display effect of the image.

[0160] In some embodiments, for a video generation scenario, the second image is an image frame selected from the first video with a resolution less than a resolution threshold, the third image corresponds to the second image, and the resolution of the third image is greater than the resolution of the second image. After generating the third image including the first object, the second image in the first video can be replaced with the third image corresponding to the second image to obtain a second video, wherein the resolution of the image frames included in the second video is greater than the resolution of the image frames included in the first video, which can improve the viewing experience of users watching the video.

[0161] In the image processing method provided in the embodiment of the present application, the third image generated based on the second feature can include the characteristics of the first object in the second image, and also has relatively rich details of the first object. There is a correspondence between the second image and the third image. There is no problem of timing inconsistency between the multiple third images generated by the embodiment of the present application, and there is no need for timing alignment. Only by the correspondence between the second image and the third image, a second video with a higher resolution of the image frame can be generated.

[0162] In some embodiments, for the first image, the first feature point of the first image and the first position information corresponding to the first feature point can be determined in advance, so that there is no need to determine the first position information corresponding to the first image once for each second image, thereby saving the computing resources consumed in determining the first position information corresponding to the first image.

[0163] Combine the following Figure 9-10 , and taking the first object as a human face as an example, the image processing method provided in the embodiment of the present application is illustrated. Fig. 9 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 5 , Fig.10 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 6 .

[0164] See also Fig. 9 In step 901, a first image is acquired, a second image is generated based on the first image and a preset audio, and for a first feature point of a first object in the first image, first position information of the first feature point in a first coordinate system is determined; and for a second feature point in the second image corresponding to the first feature point, second position information of the second feature point in the first coordinate system is determined.

[0165] With respect to step 901, the resolution of the first image is greater than the resolution of the second image, and a video including a face can be generated based on the first image and the preset audio, and then the second image can be extracted from the video including the face.

[0166] In some embodiments, a convolutional neural network and a regression network can be pre-trained. The convolutional neural network and the regression network belong to the position information acquisition layer. The mapping of two-dimensional position information to three-dimensional position information can be learned through the convolutional neural network and the regression network. Therefore, when determining the first two-dimensional position information of the first feature point and determining the second two-dimensional position information of the second feature point, the first two-dimensional position information can be mapped to the first position information, and the second two-dimensional position information can be mapped to the second position information.

[0167] The first feature point and the second feature point are both facial key points. Facial key points refer to a series of coordinate positions with specific meanings defined in facial image processing. These points usually mark the characteristic parts of the face, such as the inner and outer corners of the eyes, the tip of the nose, the corners of the mouth, the center of the chin, and the facial contour. By detecting and marking these key points, the system can accurately understand the orientation, expression changes, and subtle deformations of the face, and then apply them to various application scenarios such as face recognition, face alignment (making the face postures in different photos consistent), beauty filters, virtual makeup trials, and animated expression generation.

[0168] In step 902 , based on the first position information and the second position information, a first offset of the second feature point relative to the first feature point is determined in the first coordinate system.

[0169] Step 902 and Figure 3 The same as step 102 shown in FIG. Figure 3 The description of step 102 is not repeated here.

[0170] In step 903, a starting point is set in the first coordinate system, and a ray is created along a preset direction based on the starting point, third position information is sampled and set on the ray, and the preset third position information is superimposed with the first offset to obtain fourth position information.

[0171] For step 903, the third position information may be (X, Y, Z), the first offset may be (ΔX, ΔY, ΔZ), and the fourth position information may be (Xˋ, Yˋ, Zˋ), where Xˋ is the sum of X and ΔX, Yˋ is the sum of Y and ΔY, and Zˋ is the sum of Z and ΔZ.

[0172] In step 904, a first feature of the first object is extracted from the first image, and based on the fourth position information, a second feature of the first object corresponding to the second image is extracted from the first feature of the first object.

[0173] For step 904, the first feature of the first object may include three feature planes, and the feature planes may include P_xy, P_xz and P_yz. The fourth position information may be split, that is, (Xˋ, Yˋ, Zˋ) may be split into (Xˋ, Yˋ), (Xˋ, Zˋ) and (Yˋ, Zˋ) respectively.

[0174] The first feature of the first object is mapped to each coordinate plane to obtain the first sub-features corresponding to the coordinate plane, and the first sub-features are P_xy, P_xz and P_yz. Based on the fourth position information, feature extraction is performed on each first sub-feature to obtain the second sub-features corresponding to each first sub-feature.

[0175] For example, based on (Xˋ, Yˋ), feature extraction can be performed on P_xy to obtain the second sub-feature F_xy corresponding to P_xy. Based on (Xˋ, Zˋ), feature extraction can be performed on P_xz to obtain the second sub-feature F_xz corresponding to P_xz. Based on (Yˋ, Zˋ), feature extraction can be performed on P_yz to obtain the second sub-feature F_yz corresponding to P_yz.

[0176] After obtaining multiple second sub-features, the second sub-features can be fused. Specifically, for each fourth position information, multiple second sub-features corresponding to the fourth position information can be fused to obtain a fused feature of the second sub-feature. For example, the fused feature of the second sub-feature can be F_xy plus F_xz plus F_yz. By fusing the fused features of the second sub-features of different fourth position information, the second feature of the first object corresponding to the second image can be obtained.

[0177] In step 905, the second feature is decoded to obtain a voxel grid corresponding to the first object, the voxel grid includes multiple voxels, color information and density information of each voxel are sampled from the voxel grid, and volume rendering is performed based on the color information and density information of each voxel to obtain a third image including the first object.

[0178] See also Fig.10 ,exist Fig.10 In the embodiment, a first image 1010 including a face can be input into a generator 1001, wherein the generator 1001 has the same function as the feature extraction layer, and the generator 1001 can perform feature extraction on the first image 1010 to obtain a feature map of the first image 1010, and split the feature map of the first image 1010 into three feature planes of the first image 1010, that is, extract the first feature of the first object from the first image 1010.

[0179] exist Fig.10 In the process, a first facial key point 1040 (i.e., a first feature point) of a first image 1010 including a face can be extracted, and based on the first facial key point 1040, first position information of the first facial key point 1040 in a first coordinate system can be determined, and a second facial key point 1030 (i.e., a second feature point) of a second image 1020 including a face can be extracted, and based on the second facial key point 1030, second position information of the second facial key point 1030 in the first coordinate system can be determined.

[0180] Obtain the difference 1004 between the first facial key point 1040 and the second facial key point 1030, that is, based on the first position information and the second position information, determine the first offset (ie, the difference 1004) of the second facial key point 1030 relative to the first facial key point 1040 in the first coordinate system.

[0181] The sampling position 1005 (ie, the third position information) when the image is captured is determined, and the sampling position 1005 and the difference 1004 (the first offset) between the first face key point 1040 and the second face key point 1030 are superimposed to obtain the fourth position information 1006.

[0182] After the fourth position information 1006 and the three feature planes are determined, the second sub-features can be determined from the three feature planes respectively based on the fourth position information 1006, and the second sub-features are fused to obtain the second feature 1002 of the face corresponding to the second image 1020.

[0183] The second feature 1002 is decoded to obtain a voxel grid corresponding to the face, the voxel grid includes multiple voxels, the color information and density information of each voxel are sampled from the voxel grid, and volume rendering is performed based on the color information and density information of each voxel through the voxel renderer 1003 to obtain a third image 1050 including the face.

[0184] In the present application, the first feature is extracted from a first image with higher resolution, and the first feature has relatively rich details of the first object. The first offset is the offset of the first object in the second image relative to the first object in the first image, and the first offset can characterize the characteristics of the first object in the second image. The second feature constructed based on the first feature and the first offset can not only characterize the characteristics of the first object in the second image, but also has relatively rich details of the first object. Accordingly, the third image generated based on the second feature can include both the characteristics of the first object in the second image and relatively rich details of the first object. Compared with the related art, there is no image blurring and detail loss, and the image clarity and display effect can be improved.

[0185] In the image processing method provided in the embodiment of the present application, the feature extraction layer is equivalent to an implicit 3D head model, which is equivalent to that the image processing method provided in the embodiment of the present application can sample features (corresponding to the second feature) from the implicit 3D head model, and then perform volume rendering based on the second feature to obtain a third graphic.

[0186] The following is a description of an exemplary structure of the image processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the image processing device 455 of the memory 450 may include:

[0187] a determination module 4551, configured to determine, for a first feature point of a first object in a first image, first position information of the first feature point in a first coordinate system, and for a second feature point in a second image corresponding to the first feature point, determine second position information of the second feature point in the first coordinate system; wherein a resolution of the first image is greater than a resolution of the second image;

[0188] The determination module 4551 is further configured to determine, in the first coordinate system, a first offset of the second feature point relative to the first feature point based on the first position information and the second position information;

[0189] A construction module 4552, configured to extract a first feature of the first object from the first image, and construct a second feature of the first object corresponding to the second image based on the first offset and the first feature;

[0190] The generating module 4553 is used to generate a third image including the first object based on the second feature.

[0191] In some embodiments, the construction module 4552 is further used to superimpose the preset third position information with the first offset to obtain fourth position information; based on the fourth position information, extract the second feature of the first object corresponding to the second image from the first feature of the first object.

[0192] In some embodiments, the first coordinate system includes multiple coordinate planes; the construction module 4552 is also used to map the first feature of the first object to each of the coordinate planes, respectively, to obtain the first sub-feature corresponding to the coordinate plane; based on the fourth position information, perform feature extraction on each of the first sub-features, respectively, to obtain the second sub-feature corresponding to each of the first sub-features; and fuse the second sub-features to obtain the second feature of the first object corresponding to the second image.

[0193] In some embodiments, the determination module 4551 is also used to set a starting point in the first coordinate system and create a ray along a preset direction based on the starting point before superimposing the preset third position information with the first offset to obtain the fourth position information; sample and set the third position information on the ray.

[0194] In some embodiments, there are multiple third position information; the construction module 4552 is also used to superimpose each of the third position information with the first offset to obtain the superposition results corresponding to each of the third position information; and to fuse the superposition results corresponding to each of the third position information to obtain the fourth position information.

[0195] In some embodiments, the generation module 4553 is also used to decode the second feature to obtain a voxel grid corresponding to the first object, wherein the voxel grid includes multiple voxels; sample the color information and density information of each voxel from the voxel grid; and perform volume rendering based on the color information and density information of each voxel to obtain a third image including the first object.

[0196] In some embodiments, the determination module 4551 is further configured to generate a first video based on the first image and the video prompt information for the first image before determining the second position information of the second feature point in the first coordinate system; select an image frame having a resolution less than a resolution threshold from the first video, and use the selected image frame as the second image;

[0197] The determination module 4551 is further configured to, after generating a third image including the first object, replace the second image in the first video with the third image to obtain a second video.

[0198] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the above-mentioned image processing method of the embodiment of the present application.

[0199] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the image processing method provided by the present application embodiment, for example, Figure 3 The image processing method is shown.

[0200] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0201] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0202] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0203] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0204] The embodiments of the present application have the following advantages:

[0205] In the present application, the first feature is extracted from a first image with higher resolution, and the first feature has relatively rich details of the first object. The first offset is the offset of the first object in the second image relative to the first object in the first image, and the first offset can characterize the characteristics of the first object in the second image. The second feature constructed based on the first feature and the first offset can not only characterize the characteristics of the first object in the second image, but also has relatively rich details of the first object. Accordingly, the third image generated based on the second feature can include both the characteristics of the first object in the second image and relatively rich details of the first object. Compared with the related art, there is no image blurring and detail loss, and the image clarity and display effect can be improved.

[0206] In the image processing method provided in the embodiment of the present application, the third image generated based on the second feature can include the characteristics of the first object in the second image, and also has relatively rich details of the first object. There is a correspondence between the second image and the third image. There is no problem of timing inconsistency between the multiple third images generated by the embodiment of the present application, and there is no need for timing alignment. Only by the correspondence between the second image and the third image, a second video with a higher resolution of the image frame can be generated.

[0207] In the present application, for the first image, the first feature point of the first image and the first position information corresponding to the first feature point can be determined in advance, so that there is no need to determine the first position information corresponding to the first image once for each second image, which can save the computing resources consumed by determining the first position information corresponding to the first image.

[0208] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: For a first feature point of a first object in a first image, determine first position information of the first feature point in a first coordinate system, and for a second feature point in a second image corresponding to the first feature point, determine second position information of the second feature point in the first coordinate system; wherein the resolution of the first image is greater than the resolution of the second image; Determine, in the first coordinate system, a first offset of the second feature point relative to the first feature point based on the first position information and the second position information; Extracting a first feature of the first object from the first image, and constructing a second feature of the first object corresponding to the second image based on the first offset and the first feature; Based on the second feature, a third image including the first object is generated.

2. The method according to claim 1, characterized in that The constructing a second feature of the first object corresponding to the second image based on the first offset and the first feature includes: Superimposing the preset third position information with the first offset to obtain fourth position information; Based on the fourth position information, a second feature of the first object corresponding to the second image is extracted from the first feature of the first object.

3. The method according to claim 2, characterized in that The first coordinate system includes a plurality of coordinate planes; and extracting, based on the fourth position information, from the first feature of the first object, a second feature of the first object corresponding to the second image, comprises: Mapping the first feature of the first object to each of the coordinate planes respectively to obtain a first sub-feature corresponding to the coordinate plane; Based on the fourth position information, feature extraction is performed on each of the first sub-features to obtain a second sub-feature corresponding to each of the first sub-features; The second sub-features are fused to obtain a second feature of the first object corresponding to the second image.

4. The method according to claim 2, characterized in that: Before superimposing the preset third position information with the first offset to obtain the fourth position information, the method further includes: Setting a starting point in the first coordinate system, and creating a ray along a preset direction based on the starting point; The third position information is sampled and set on the ray.

5. The method according to claim 4, characterized in that The number of the third position information is multiple; the pre-set third position information is superimposed with the first offset to obtain the fourth position information, including: Superimposing each of the third position information with the first offset respectively to obtain a superimposition result corresponding to each of the third position information; The superposition results corresponding to the third position information are fused to obtain the fourth position information.

6. The method according to claim 1, characterized in that The step of generating a third image including the first object based on the second feature comprises: Decoding the second feature to obtain a voxel grid corresponding to the first object, wherein the voxel grid includes a plurality of voxels; Sampling color information and density information of each of the voxels from the voxel grid; Volume rendering is performed based on the color information and density information of each voxel to obtain a third image including the first object.

7. The method according to claim 1, characterized in that Before determining the second position information of the second feature point in the first coordinate system, the method further includes: Generate a first video based on the first image and video prompt information for the first image; Selecting an image frame whose resolution is less than a resolution threshold from the first video, and using the selected image frame as the second image; After generating the third image including the first object, the method further includes: The second image in the first video is replaced by the third image to obtain a second video.

8. An image processing device, characterized in that: The device comprises: a determination module, configured to determine, for a first feature point of a first object in a first image, first position information of the first feature point in a first coordinate system, and, for a second feature point in a second image corresponding to the first feature point, determine second position information of the second feature point in the first coordinate system; wherein the resolution of the first image is greater than the resolution of the second image; a determination module, further configured to determine, in the first coordinate system, a first offset of the second feature point relative to the first feature point based on the first position information and the second position information; a construction module, configured to extract a first feature of the first object from the first image, and construct a second feature of the first object corresponding to the second image based on the first offset and the first feature; A generating module is used to generate a third image including the first object based on the second feature.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the image processing method according to any one of claims 1 to 7 when executing the computer executable instructions or computer programs stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images