Image Processing Method, Apparatus, Medium and Electronic Device

Abstract

The present application belongs to the field of artificial intelligence technology and specifically relates to an image processing method, an image processing device, a computer-readable medium, and an electronic device. The image processing method includes: receiving image encoding data of a video clip transmitted by an encoding end; decoding the image encoding data to obtain a first video frame, key points in the first video frame, and key points in a second video frame, wherein the key points are used to represent the motion posture of a target object in the video frame; generating transformation information of the motion posture of the target object based on the key points in the first video frame and the key points in the second video frame; and generating a reconstructed image of the second video frame based on the first video frame and the transformation information.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing method, an image processing device, a computer-readable medium, and an electronic device. Background Art

[0002] When encoding and transmitting video, pixel-by-pixel precision is typically used, using predictive coding techniques to transmit residual frame information at the pixel level. While the continuous evolution of traditional video coding standards has improved video compression performance, the high density of transmitted data and information associated with two-dimensional pixel-level signal compression makes it difficult to achieve significant bitrate savings while maintaining high-definition video quality. Summary of the Invention

[0003] The purpose of the present application is to provide an image processing method, an image processing device, a computer-readable medium, and an electronic device, which can at least to some extent overcome the technical problems existing in the related art such as large data transmission volume and high transmission bit rate.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, including: receiving image coding data of a video clip transmitted by an encoding end; decoding the image coding data to obtain a first video frame, key points in the first video frame, and key points in a second video frame, wherein the key points are used to represent the motion posture of the target object in the video frame; generating transformation information of the motion posture of the target object based on the key points in the first video frame and the key points in the second video frame; and generating a reconstructed image of the second video frame based on the first video frame and the transformation information.

[0006] According to one aspect of an embodiment of the present application, an image processing device is provided, including: a receiving module, configured to receive image coding data of a video clip transmitted by an encoding end; a decoding module, configured to decode the image coding data to obtain a first video frame, key points in the first video frame, and key points in the second video frame, wherein the key points are used to represent the motion posture of the target object in the video frame; a generation module, configured to generate transformation information of the motion posture of the target object based on the key points in the first video frame and the key points in the second video frame; and a reconstruction module, configured to generate a reconstructed image of the second video frame based on the first video frame and the transformation information.

[0007] In some embodiments of the present application, based on the above technical solution, the transformation information of the motion posture of the target object includes the motion field between the first video frame and the second video frame; the generation module is configured to: establish a one-to-one corresponding key point mapping relationship between the key points in the first video frame and the key points in the second video frame; generate affine transformation parameters of each key point neighborhood according to the key point mapping relationship; perform optical flow estimation on the first video frame according to the affine transformation parameters, the key points in the first video frame and the key points in the second video frame to obtain the motion field between the first video frame and the second video frame.

[0008] In some embodiments of the present application, based on the above technical solution, the reconstruction module is configured to: perform feature extraction on the first video frame to obtain multiple first image features corresponding to different feature depths; perform transformation processing on the first image features of each feature depth according to the transformation information to obtain deformation features corresponding to different feature depths; perform feature restoration on the first image features to obtain second image features corresponding to different feature depths; perform feature fusion on the deformation features and the second image features corresponding to the same feature depth, and perform mapping processing on the fused features to obtain a reconstructed image of the second video frame.

[0009] In some embodiments of the present application, based on the above technical solution, the reconstruction module is also configured to: map the fused features separately through multiple output networks with different network scales to obtain multiple reconstructed images of the second video frame, and the multiple reconstructed images have different image resolutions.

[0010] In some embodiments of the present application, based on the above technical solution, the device also includes: an audio acquisition module, configured to acquire audio data associated with the image encoding data; an image detection module, configured to perform key point detection on the reconstructed image of the second video frame to obtain the key points of the target object in the reconstructed image; a key point determination module, configured to determine the key points of the target area located inside the target object based on the distribution positions of each key point in the reconstructed image; a position correction module, configured to perform position correction on the key points of the target area according to the audio data to obtain corrected key points; and an image repair module, configured to perform local image modification on the reconstructed image according to the corrected key points to obtain a repaired reconstructed image.

[0011] In some embodiments of the present application, based on the above technical solution, the apparatus further includes: a first sample acquisition module, configured to acquire a first image sample including a target object and a second image sample having content relevance to the first image sample; a first sample detection module, configured to perform key point detection on the first image sample and the second image sample, respectively, to obtain a first key point of the target object in the first image sample and a second key point of the target object in the second image sample; a first network acquisition module, configured to acquire an image generation network for generating a reconstructed image based on an original image and an image discrimination network for discriminating whether an image is an original image or a reconstructed image; The first image reconstruction module is configured to input the first image sample, the first key point and the second key point into the image generation network, so as to perform image reconstruction processing on the first image sample through the image generation network to obtain a reconstructed image of the second image sample; the image discrimination module is configured to input the reconstructed image of the second image sample and the second image sample into the image discrimination network respectively, so as to perform discrimination processing on the input image through the image discrimination network to obtain discrimination information of whether the input image is the original image or the reconstructed image; the first parameter updating module is configured to alternately update the network parameters of the image generation network and the image discrimination network according to the discrimination information.

[0012] In some embodiments of the present application, based on the above technical solution, the device further includes: a second sample acquisition module, configured to acquire a third image sample including a target object and a fourth image sample having a time correlation with the third image sample; a second sample detection module, configured to perform key point detection on the third image sample and the fourth image sample, respectively, to obtain a third key point of the target object in the third image sample and a fourth key point of the target object in the fourth image sample; a second model acquisition module, configured to acquire an image generation network for generating a reconstructed image based on the original image and a similarity discrimination network for discriminating similarity; a second image reconstruction module, configured to convert the third image sample and the fourth image sample into a plurality of images; The third key point and the fourth key point are input into the image generation network, so that the third image sample and the fourth image sample are respectively reconstructed by the image generation network to obtain the reconstructed image of the third image sample and the reconstructed image of the fourth image sample; the similarity discrimination module is configured to input the reconstructed image of the third image sample and the reconstructed image of the fourth image sample into the similarity discrimination network, so that the input images are discriminated by the similarity discrimination network to obtain the image similarity between the reconstructed image of the third image sample and the reconstructed image of the fourth image sample; the second parameter updating module is configured to update the network parameters of the image generation network and the similarity discrimination network according to the image similarity.

[0013] According to one aspect of an embodiment of the present application, an image processing method is provided, including: obtaining a video clip to be transmitted, the video clip including multiple video frames; extracting a first video frame including the target object and a second video frame having content relevance to the first video frame from the video clip; performing key point detection on the video frame to obtain key points of the target object in the video frame, the key points being used to represent the motion posture of the target object in the video frame; encoding the first video frame, the key points in the first video frame, and the key points in the second video frame to obtain image encoding data; transmitting the image encoding data to a decoding end, so that the decoding end generates a reconstructed image of the second video frame based on the image encoding data.

[0014] According to one aspect of an embodiment of the present application, an image processing device is provided, including: an acquisition module, configured to acquire a video clip to be transmitted, the video clip including multiple video frames; an extraction module, configured to extract a first video frame including the target object and a second video frame having content relevance to the first video frame from the video clip; a detection module, configured to perform key point detection on the video frame to obtain key points of the target object in the video frame, and the key points are used to represent the motion posture of the target object in the video frame; an encoding module, configured to perform encoding processing on the first video frame, the key points in the first video frame and the key points in the second video frame to obtain image encoding data; a transmission module, configured to transmit the image encoding data to a decoding end, so that the decoding end generates a reconstructed image of the second video frame based on the image encoding data.

[0015] In some embodiments of the present application, based on the above technical solution, the detection module is configured to: input the video frame into a pre-trained target object recognition model; perform key point detection on the target object in the video frame through the target object recognition model to obtain a first key point for representing the overall motion posture of the target object and a second key point for representing the motion posture of the target area inside the target object.

[0016] In some embodiments of the present application, based on the above technical solution, the device also includes: a sample acquisition module, configured to acquire an image sample including a target object and a sample label associated with the image sample, wherein the sample label is used to mark the position coordinates of the second key point in the image sample; a sample input module, configured to input the image sample and the sample label into the target object recognition model; a sample detection module, configured to perform key point detection on the target object in the image sample through the target object recognition model to obtain the first key point and the second key point of the target object in the image sample; a parameter update module, configured to obtain the loss error between the sample label and the detected second key point, and update the model parameters of the target object recognition model according to the loss error.

[0017] In some embodiments of the present application, based on the above technical solution, the extraction module is configured to: perform feature extraction on each video frame in the video clip to obtain image features for representing the image content of the video frame; perform clustering processing on the video frames according to the image features to obtain clusters composed of video frames with content relevance and the cluster centers of the clusters; determine the video frame including the target object in the cluster that is closest to the cluster center as the first video frame, and determine the other video frames in the cluster as second video frames with content relevance to the first video frame.

[0018] In some embodiments of the present application, based on the above technical solution, the encoding module is configured to: perform network detection on the data transmission network used to transmit the video clip to determine the network type and network signal strength of the data transmission network; determine whether the data transmission network belongs to a weak network based on the network type and network signal strength; if the data transmission network belongs to a weak network, encode the first video frame, the key points in the first video frame, and the key points in the second video frame; if the data transmission network does not belong to a weak network, encode the first video frame, the second video frame, the key points in the first video frame, and the key points in the second video frame.

[0019] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method in the above technical solution is implemented.

[0020] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image processing method in the above technical solution by executing the executable instructions.

[0021] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described in the above technical solution.

[0022] In the technical solution provided in the embodiments of the present application, by encoding key points with extremely little information in the video frame image, the video frame image can be reconstructed based on the key points, thereby reducing the amount of transmitted data and the transmission bit rate while improving the encoding, decoding and transmission efficiency of the video image.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0025] Figure 1 A schematic diagram shows an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

[0026] Figure 2 The diagram shows how a video encoding device and a video decoding device are placed in a streaming environment.

[0027] Figure 3 A basic flow chart of a video encoder is shown, in which intra-frame prediction is taken as an example for explanation.

[0028] Figure 4 The principle block diagram of an embodiment of the present application in video consultation or other application scenarios involving video transmission is schematically shown.

[0029] Figure 5 The following schematically shows a flowchart of the steps of an image processing method in one embodiment of the present application.

[0030] Figure 6 The flowchart schematically shows the steps of extracting video frames in one embodiment of the present application.

[0031] Figure 7 The following schematically illustrates a flowchart of the steps for training a target object recognition model based on semi-supervised learning in one embodiment of the present application.

[0032] Figure 8 The flowchart schematically shows the steps of the image processing method at the decoding end in an embodiment of the present application.

[0033] Figure 9 The flowchart schematically shows the steps of image reconstruction in one embodiment of the present application.

[0034] Figure 10 The network architecture diagram of the image generation network in one embodiment of the present application is schematically shown.

[0035] Figure 11The diagram schematically shows the principle of repairing a reconstructed image based on audio data in one embodiment of the present application.

[0036] Figure 12 The network architecture for joint training of models in one embodiment of the present application is schematically shown.

[0037] Figure 13 The structural block diagram of the image processing device at the decoding end provided in an embodiment of the present application is schematically shown.

[0038] Figure 14 The structural block diagram of the image processing device at the encoding end provided in an embodiment of the present application is schematically shown.

[0039] Figure 15 The following schematically shows a block diagram of a computer system structure of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0041] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0044] Figure 1A schematic diagram shows an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

[0045] like Figure 1 As shown, the system architecture 100 includes a plurality of terminal devices, which can communicate with each other via, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected via the network 150. Figure 1 In the embodiment of the present invention, the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0046] For example, the first terminal device 110 can encode video data (such as a video picture stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device 120 can receive the encoded video data from the network 150, decode the encoded video data to restore the video data, and display the video picture based on the restored video data.

[0047] In one embodiment of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 for performing bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video image stream captured by the terminal device) for transmission to the other of the third terminal device 130 and the fourth terminal device 140 via a network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded video data transmitted by the other of the third terminal device 130 and the fourth terminal device 140, decode the encoded video data to recover the video data, and display the video image on an accessible display device based on the recovered video data.

[0048] exist Figure 1In the embodiments of the present invention, the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140 may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Each terminal device in the embodiments of the present invention may serve as a blockchain node to form a blockchain network, and the encoded data transmitted between the terminal devices may be stored on a blockchain maintained by the blockchain network. Network 150 represents any number of networks, including, for example, wired and / or wireless communication networks, for transmitting encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140. The communication network 150 may exchange data using circuit-switched and / or packet-switched channels. The network may include a telecommunications network, a local area network, a wide area network, and / or the Internet. For the purposes of this application, unless otherwise explained below, the architecture and topology of network 150 may be irrelevant to the operations disclosed herein.

[0049] In one embodiment of the present application, Figure 2 The present invention illustrates the placement of a video encoding device and a video decoding device in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV (television), and storing compressed video on digital media such as CDs, DVDs, and memory sticks.

[0050] The streaming system may include an acquisition subsystem 213, which may include a video source 201, such as a digital camera, that creates an uncompressed video picture stream 202. In one embodiment, the video picture stream 202 includes samples captured by the digital camera. The video picture stream 202 is depicted as a thicker line to emphasize the higher data volume of the video picture stream compared to the encoded video data 204 (or the encoded video stream 204). The video picture stream 202 may be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or embody various aspects of the disclosed subject matter, as described in greater detail below. The encoded video data 204 (or the encoded video stream 204) is depicted as a thinner line to emphasize the lower data volume of the encoded video data 204 (or the encoded video stream 204), which may be stored on a streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2The client subsystem 206 and the client subsystem 208 in the streaming server 205 can access the streaming server 205 to retrieve the copies 207 and 209 of the encoded video data 204. The client subsystem 206 may include, for example, a video decoding device 210 in the electronic device 230. The video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a display screen) or another presentation device. In some streaming systems, the encoded video data 204, the video data 207, and the video data 209 (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0051] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video decoding device, and the electronic device 230 may also include a video encoding device.

[0052] In one embodiment of the present application, taking the international video coding standards HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and China's national video coding standard AVS as examples, after a video frame image is input, the video frame image will be divided into several non-overlapping processing units according to a block size, and each processing unit will perform similar compression operations. This processing unit is called CTU (Coding Tree Unit), or LCU (Largest Coding Unit). The CTU can continue to be divided more finely to obtain one or more basic coding units CU. CU is the most basic element in a coding link. The following introduces some concepts when encoding CU:

[0053] Predictive Coding: Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by a selected reconstructed video signal to produce a residual video signal. The encoder needs to decide which predictive coding mode to use for the current CU and inform the decoder. Intra-frame prediction means the predicted signal comes from an already coded and reconstructed area within the same image; inter-frame prediction means the predicted signal comes from a previously coded image (called a reference image) that is different from the current image.

[0054] Transform & Quantization: After the residual video signal undergoes transformations such as the Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), it is converted to the transform domain, where coefficients are stored. The transform coefficients are then subjected to a lossy quantization operation, which removes some information and makes the quantized signal more suitable for compression. Some video coding standards may offer more than one transform scheme, so the encoder must select one for the current CU and inform the decoder. The level of quantization is typically determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which typically results in greater distortion and a lower bitrate. Conversely, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which typically results in less distortion and a higher bitrate.

[0055] Entropy Coding or Statistical Coding: The quantized transform domain signal is statistically compressed and encoded based on the frequency of occurrence of each value, and finally a binary (0 or 1) compressed code stream is output. At the same time, the encoding generates other information, such as the selected coding mode, motion vector data, etc., which also need to be entropy coded to reduce the bit rate. Statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0056] The context-based adaptive binary arithmetic coding (CABAC) process consists of three main steps: binarization, context modeling, and binary arithmetic coding. After binarization, the input syntax elements can be encoded using both the normal coding mode and the bypass coding mode. In the bypass coding mode, instead of assigning a specific probability model to each binary bit, the input binary bit values are directly encoded using a simple bypass encoder, speeding up both encoding and decoding. Generally, different syntax elements are not completely independent, and even the same syntax elements have some memory. Therefore, based on conditional entropy theory, conditional coding using other coded syntax elements can further improve coding performance compared to independent or memoryless coding. This coded symbol information used as a condition is called context. In the normal coding mode, the binary bits of the syntax elements are sequentially fed into the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values of previously coded syntax elements or binary bits. This process is known as context modeling. The context model corresponding to the syntax element can be located using ctxIdxInc (contextindex increment) and ctxIdxStart (context index Start). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0057] Loop Filtering: The changed and quantized signal will be reconstructed through inverse quantization, inverse transformation and prediction compensation operations to obtain a reconstructed image. Compared with the original image, due to the influence of quantization, some information of the reconstructed image is different from the original image, that is, the reconstructed image will produce distortion. Therefore, the reconstructed image can be filtered, such as deblocking filter (DB), SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) and other filters, which can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images to predict future image signals, the above filtering operation is also called loop filtering, that is, filtering operation within the encoding loop.

[0058] In one embodiment of the present application, Figure 3The basic flow chart of a video encoder is shown in FIG. In this flow chart, intra-frame prediction is used as an example. k [x,y] and predicted image signal Perform difference operation to obtain the residual signal u k [x,y], residual signal u k [x,y] is transformed and quantized to obtain the quantized coefficients. The quantized coefficients are entropy coded to obtain the encoded bit stream, and the reconstructed residual signal u' is obtained by inverse quantization and inverse transformation. k [x,y], predicted image signal and the reconstructed residual signal u' k [x,y] superposition generates image signal Image signal On the one hand, it is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing, and on the other hand, the reconstructed image signal s' is output through loop filtering. k [x,y], reconstructed image signal s' k [x,y] can be used as the reference image for the next frame for motion estimation and motion compensation prediction. Then based on the result of motion compensation prediction s' r [x+m x ,y+m y ] and intra prediction results Get the predicted image signal of the next frame And continue to repeat the above process until the encoding is completed.

[0059] Based on the above encoding process, at the decoding end, after obtaining the compressed code stream (i.e., bitstream), entropy decoding is performed on each CU to obtain various mode information and quantization coefficients. The quantization coefficients are then dequantized and inversely transformed to obtain a residual signal. Furthermore, based on the known coding mode information, the prediction signal corresponding to the CU can be obtained. The residual signal is then added to the prediction signal to obtain a reconstructed signal. The reconstructed signal then undergoes loop filtering and other operations to produce the final output signal.

[0060] One application scenario of the embodiments of the present application may include a video consultation service, which is an online communication tool service for enterprise end users. Users can use terminal devices installed with instant messaging software to communicate with two or more people in real time. In addition, the embodiments of the present application can also be applied to various scenarios involving video transmission, such as video calls and live webcasts, which are not specifically limited in the embodiments of the present application.

[0061] The embodiments of the present application can improve the image clarity problem in video transmission based on artificial intelligence technology, solve problems such as video blur and experience overlap in poor network conditions, and have the advantages of cross-terminal, high security, high simulation, fast generation speed, and strong real-time performance.

[0062] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0063] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0064] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0065] Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0066] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0067] Figure 4 The principle block diagram of the embodiment of the present application in video consultation or other application scenarios involving video transmission is schematically shown. Figure 4 As shown, the sending user at the encoding end 410 can collect video images containing faces through image acquisition devices such as cameras, and further perform face recognition on the video images through the AI coding model to obtain the facial key points included therein. On this basis, the AI coding model can compress and encode the key frames and the facial key points in each frame of the video image to obtain the corresponding image coding data. After the image coding data is transmitted to the decoding end 420 in real time, the AI decoding module at the decoding end 420 can reconstruct the face based on the reference frame and the facial key points of each frame of the video image to obtain a high-definition face reconstructed image, thereby showing a clear and continuous video image to the receiving user. The technical solution provided in the embodiment of the present application can encode only semantic key points with very little information through the encoding end, and reconstruct and restore the high-definition face image after receiving the key points through the decoding end. Compared with the traditional video transmission solution, the present application can well overcome the problems of low timeliness of video transmission, unnatural images, and blurred images.

[0068] The following is a detailed description of the image processing method and other technical solutions provided in the embodiments of the present application from the two aspects of the encoding end and the decoding end.

[0069] Figure 5 The following is a schematic flow chart showing the steps of an image processing method in one embodiment of the present application. The image processing method can be performed by Figure 3 The video encoder shown in FIG. Figure 5 As shown, the image processing method may mainly include the following steps S510 to S550.

[0070] In step S510, a video segment to be transmitted is obtained, where the video segment includes a plurality of video frames.

[0071] In one embodiment of the present application, the video clip may be a video of a specified time length captured in real time by an image capture device, or the video clip may be a clip cropped from a captured complete video.

[0072] A video clip includes multiple video frames, and a video frame is a video image. For example, if a video clip has a duration of 2 seconds and a frame rate of 60 frames per second, then the video clip includes 120 consecutive video frames.

[0073] In step S520 , a first video frame including the target object and a second video frame having content relevance to the first video frame are extracted from the video clip.

[0074] Content relevance means that the first video frame and the second video frame have similar video content, for example, the first video frame and the second video frame include the same target object, and the target object has similar action postures.

[0075] In one embodiment of the present application, the target object may be a human face, the first video frame may be a key frame containing a human face, and the second video frame may be one or more video frames that are temporally continuous with the first video frame and have similar content.

[0076] The video clip may include one or more first video frames, where the multiple first video frames refer to two or more first video frames.

[0077] Figure 6 The flowchart of the steps of extracting video frames in one embodiment of the present application is schematically shown. Figure 6 As shown, based on the above embodiment, extracting the first video frame including the target object and the second video frame having content relevance to the first video frame from the video clip in step S520 may include the following steps S610 to S630.

[0078] Step S610: extracting features from each video frame in the video clip to obtain image features for representing image content of the video frame.

[0079] The image feature may be a feature vector with a specified length used to represent the image content of the video frame. For example, the corresponding feature value may be extracted based on a color histogram of the video frame.

[0080] Step S620: performing clustering processing on the video frames according to the image features to obtain clusters consisting of video frames with content relevance and cluster centers of the clusters.

[0081] When performing clustering, you can first randomly select a number of cluster centers and calculate the distance (such as vector distance) between the image feature and each cluster center. Then, divide the image feature into the cluster center closest to it. After completing the division of the image feature, you will get multiple clusters with the same number of cluster centers. According to the average value of each image feature in the cluster, you can re-determine the new cluster center of each cluster. On this basis, the distance calculation and division steps between the image feature and the cluster center are repeated continuously. Finally, when the clustering stop condition is met, multiple clusters are obtained, and the cluster center of each cluster can be determined.

[0082] Step S630: determining the video frame including the target object that is closest to the cluster center in the cluster as the first video frame, and determining the other video frames in the cluster as second video frames having content relevance to the first video frame.

[0083] Each cluster includes multiple video frames, among which the video frame containing the target object that is closest to the cluster center can be selected as the first video frame, and the other video frames in the cluster except the first video frame are determined to be the second video frames with content relevance thereto.

[0084] In one embodiment of the present application, the video frame at a specified position in the video clip can also be directly used as the first video frame. For example, the first frame, the last frame, or a frame in the middle of the video clip can be used as the first video frame, and the other video frames can be used as the second video frames.

[0085] In one embodiment of the present application, the first video frame can also be determined based on the motion analysis of the target object. For example, the optical flow of the target object can be analyzed in the video clip, and each time the video frame with the least number of optical flow movements in the video clip is selected as the first video frame, and the other video frames are selected as the second video frames.

[0086] In step S530, key point detection is performed on the video frame to obtain key points of the target object in the video frame. The key points are used to represent the motion posture of the target object in the video frame.

[0087] In one embodiment of the present application, the target object is a human face, and the corresponding key points may be facial key points.

[0088] In one embodiment of the present application, a method for performing key point detection on a video frame may include: inputting the video frame into a pre-trained target object recognition model; performing key point detection on the target object in the video frame through the target object recognition model to obtain a first key point for representing the overall motion posture of the target object and a second key point for representing the motion posture of the target area inside the target object.

[0089] In an embodiment of the present application, the target object recognition model may be a face recognition model, based on which the key points of the face in each video frame may be detected. Among them, the first key point used to represent the overall motion posture of the face may be, for example, one or more key points distributed near the edge of the face. This part of the first key points may be used to capture the overall motion posture of the head of the person, such as turning the head, lowering the head, raising the head, etc. The target area located inside the face may be, for example, the area where the eyes, mouth, nose, etc. are located. The second key point used to represent the motion posture of the target area inside the face may be the key point concentrated at the eyes, mouth, nose, etc. to accurately describe the motion posture of the facial features. For example, it can be used to represent the opening and closing of the eyes, the opening and closing of the mouth, and other motion postures.

[0090] Figure 7 The following schematically shows a flowchart of the steps for training the target object recognition model based on semi-supervised learning in one embodiment of the present application. Figure 7 As shown, based on the above embodiments, the method for performing model training on the target object recognition model may include the following steps S710 to S740.

[0091] Step S710: Acquire an image sample including a target object and a sample label associated with the image sample, where the sample label is used to mark the position coordinates of the second key point in the image sample.

[0092] The image sample may be a face image including a human face, and the sample label is the position coordinates of several key points used to represent the facial features of the human face.

[0093] Step S720: Input the image samples and sample labels into the target object recognition model.

[0094] The target object recognition model may be a face recognition model for detecting human faces, and the face image and the corresponding annotated partial face key point labels are input into the face recognition model.

[0095] Step S730: performing key point detection on the target object in the image sample using the target object recognition model to obtain a first key point and a second key point of the target object in the image sample.

[0096] The face recognition model can detect key points of the face in the image. The detected face key points can be divided into two categories. One category belongs to the second key points, that is, the key points in the facial area with position markings, and the other category belongs to the first key points without position markings.

[0097] Step S740: Obtain the loss error between the sample label and the second key point detected, and update the model parameters of the target object recognition model according to the loss error.

[0098] According to the position of the pre-marked second key point in the image and the distribution position of the second key point actually output by the template object, the loss error can be determined. Back propagation based on the loss error can determine the error gradient of each weight in each network layer in the target object recognition model, and then update the weights in part or all of the network layers based on the error gradient.

[0099] The inventors of the present application have discovered that when the mature facial key point detection algorithm dlib is used in the video, the key points obtained will always be unstable in the time domain, such as jitter, and whether the key point information of the driving frame is accurate will greatly affect the final face generation effect. In order to ensure the stability of key point detection in the time domain, a combination of unsupervised and supervised methods is adopted here. The embodiment of the present application detects a total of 2N key points, including N unsupervised key points and N supervised key points. The unsupervised part is to capture the overall movement posture of the head, such as turning the head, lowering the head, raising the head, etc. (as shown in the 5 red key points in the figure below); the supervised key points are mainly concentrated near the eyes and mouth (as shown in the figure below); Figure 5 The purpose of the image is to accurately describe the movement of the facial features, such as the opening and closing of the eyes, the movement of the mouth, etc., so that a natural-looking face can be generated at the decoding end.

[0100] The embodiment of the present application learns N unsupervised key points and N supervised key points together, and uses a face key point detection algorithm to obtain the key point coordinates as labels to supervise the N key points in 2N. This can effectively reduce the instability of these N supervised key points in the time domain.

[0101] In step S540 , the first video frame, the key points in the first video frame, and the key points in the second video frame are coded to obtain image coded data.

[0102] The method for encoding the first video frame may adopt the video encoding method provided in the above embodiment, which will not be described in detail here.

[0103] In one embodiment of the present application, a network detection can be first performed on the data transmission network used to transmit the video clip to determine the network type and network signal strength of the data transmission network, and then it can be determined whether the data transmission network is a weak network based on the network type and network signal strength; if the data transmission network is a weak network, the first video frame, the key points in the first video frame, and the key points in the second video frame are encoded; if the data transmission network is not a weak network, the first video frame, the second video frame, the key points in the first video frame, and the key points in the second video frame are encoded.

[0104] In addition to wired communication links, the types of data transmission networks also include connection methods for various mobile communication networks such as 2G / 3G / Edge / 4G / 5G / Wifi. Different protocols, different standards, and different rates make the scenarios for mobile application operation richer. From the perspective of network transmission, the scenarios that require extra attention are far more than just network disconnection, network failure, etc. In one embodiment of the present application, a weak network refers to a network whose network signal strength is lower than a strength threshold. The strength threshold is a preset threshold associated with the network type, and different network types can be associated with the same or different strength thresholds. For example, when it is detected that the network type of the data transmission network is a 4G network, the corresponding strength threshold is obtained as -50dbm; if the network signal strength of the data transmission network is less than -50dbm, it can be determined that the network belongs to a weak network; if the network signal strength of the data transmission network is greater than or equal to -50dbm, it can be determined that the network does not belong to a weak network.

[0105] In one embodiment of the present application, one or more parameters among network bandwidth, packet loss rate or network delay can also be used as network quality parameters, and a network whose network quality parameters are lower than a parameter threshold is determined to be a weak network. For example, using network delay as a network quality parameter, when the network delay of the data transmission network is greater than 300ms, the network can be determined to be a weak network. For another example, using packet loss rate as a network quality parameter, when the packet loss rate of the data transmission network is greater than 10%, the network can be determined to be a weak network.

[0106] In an embodiment of the present application, when it is determined that the data transmission network is a weak network, that is, the network quality is poor, it is possible to choose to encode only the first video frame, the key points in the first video frame, and the key points in the second video frame, thereby significantly reducing the amount of encoded data, reducing the complexity of the encoded information, and improving the encoding efficiency and data transmission efficiency. When it is determined that the data transmission network is not a weak network, that is, the network quality is good, it is possible to choose to encode the first video frame, the second video frame, the key points in the first video frame, and the key points in the second video frame, thereby further improving the clarity of the video image when the network bandwidth pressure is not large.

[0107] In step S550 , the image encoding data is transmitted to a decoding end, so that the decoding end generates a reconstructed image of the second video frame according to the image encoding data.

[0108] In one embodiment of the present application, before transmitting the video from the encoding end to the decoding end, the image encoding data can be encrypted based on an encryption algorithm to improve the security and reliability of the data transmission. After transmission to the decoding end, the decoding end first decrypts the encrypted encoding data and then performs further decoding processing after the decryption process, including data such as the first video frame, key points in the first video frame, and key points in the second video frame. Based on the corresponding data obtained through decoding, the decoding end can perform image reconstruction processing to generate a reconstructed image of the second video frame.

[0109] Figure 8 The following is a schematic flow chart showing the steps of the image processing method at the decoding end according to an embodiment of the present application. The image processing method can be performed by Figure 3 The decoder shown is executed. Figure 8 As shown, the image processing method may mainly include the following steps S810 to S840.

[0110] In step S810 , image encoding data of a video segment transmitted by an encoding end is received.

[0111] In one embodiment of the present application, a decoder can receive, in real time, encoded image data corresponding to a video clip transmitted from an encoder. If the encoded image data is encrypted by the encoder using an encryption key, the decoder can decrypt the encrypted data using a corresponding decryption key to obtain the original plaintext data.

[0112] In step S820, the image coding data is decoded to obtain the first video frame, key points in the first video frame, and key points in the second video frame. The key points are used to represent the motion posture of the target object in the video frame.

[0113] The method for decoding the image coded data may refer to the decoding method performed by the decoder in the above embodiment, which will not be described in detail here.

[0114] In step S830 , transformation information of the motion posture of the target object is generated according to the key points in the first video frame and the key points in the second video frame.

[0115] In one embodiment of the present application, the transformation information of the motion posture of the target object includes a motion field (Motion Field) between the first video frame and the second video frame. When the motion posture of the target object changes between the two video frames, the corresponding motion field can be obtained by marking the motion speed of each pixel point on the target object.

[0116] In one embodiment of the present application, a method for generating transformation information of the motion posture of a target object based on key points in a first video frame and key points in a second video frame may include: establishing a one-to-one key point mapping relationship between the key points in the first video frame and the key points in the second video frame; generating affine transformation parameters of the neighborhood of each key point based on the key point mapping relationship; performing optical flow estimation on the first video frame based on the affine transformation parameters, the key points in the first video frame and the key points in the second video frame to obtain a motion field between the first video frame and the second video frame.

[0117] The changes in light and dark of an object in an image sequence are called optical flow. Optical flow estimation is a method of inferring the speed and direction of an object's movement by detecting changes in the intensity of image pixels over time. The embodiment of the present application is based on optical flow estimation performed on the first video frame, and a motion field used to describe the motion state of the target object can be obtained.

[0118] In step S840 , a reconstructed image of the second video frame is generated according to the first video frame and the transformation information.

[0119] Figure 9 The flowchart of the steps of image reconstruction in one embodiment of the present application is schematically shown. Figure 9 As shown, based on the above embodiment, generating a reconstructed image of the second video frame according to the first video frame and the transformation information in step S840 may include the following steps S910 to S940.

[0120] Step S910: performing feature extraction on the first video frame to obtain a plurality of first image features corresponding to different feature depths.

[0121] In one embodiment of the present application, a pre-trained feature extraction network can be used to perform feature extraction on the first video frame. The feature extraction network may include multiple continuously distributed feature extraction layers with gradually deepening features. After the first video frame is input into the feature extraction network, multiple first image features corresponding to different feature depths can be extracted in sequence by each feature extraction layer.

[0122] Step S920: performing transformation processing on the first image features of each feature depth respectively according to the transformation information to obtain deformation features corresponding to different feature depths.

[0123] The first image features extracted at each feature depth are transformed according to the transformation information to obtain the corresponding deformation features at each feature depth. In one embodiment of the present application, the transformation information may be a transformation matrix generated based on the motion field, and the first image features may be the extracted feature map. Matrix operations are performed on the feature map and the transformation matrix to obtain the corresponding deformation features.

[0124] Step S930: performing feature restoration on the first image feature to obtain a second image feature corresponding to a different feature depth.

[0125] Corresponding to the feature extraction network, the embodiment of the present application can restore the first image features at each feature depth through a reversely configured feature restoration network, wherein a corresponding second image feature can be restored at each feature depth.

[0126] Step S940: performing feature fusion on the deformation features and the second image features corresponding to the same feature depth, and performing mapping processing on the fused features to obtain a reconstructed image of the second video frame.

[0127] The method for fusing the deformation feature and the second image feature may be to directly sum the feature values of the corresponding dimensions, or to calculate the average value of the feature values of the corresponding dimensions.

[0128] In one embodiment of the present application, after feature fusion, the fused features can be mapped and processed separately through multiple output networks with different network scales to obtain multiple reconstructed images of the second video frame, and the multiple reconstructed images have different image resolutions. For example, the resolution of the first video frame is 256X256. After the image reconstruction is performed in the embodiment of the present application, a reconstructed image of the second video frame with a resolution of 256X256 can be obtained. In addition, by configuring multiple output networks, the embodiment of the present application can also simultaneously obtain a reconstructed image of the second video frame with a resolution of 512X512, thereby meeting the display requirements of video images of different sizes.

[0129] In one embodiment of the present application, a pre-trained image generation network can be used to perform image reconstruction. Figure 10 The network architecture diagram of the image generation network in one embodiment of the present application is schematically shown. The image generation network may include multiple parts such as a feature extraction network, an optical flow estimation network, a feature restoration network, and a feature output network.

[0130] Take convolutional neural networks as an example, such as Figure 10As shown, the feature extraction network includes multiple successively deeper convolutional layers 1001. After the first video frame 1002 is input, each convolutional layer 1001 performs convolution processing on the input features in turn, thereby extracting first image features corresponding to different feature depths.

[0131] The first video frame 1002, the key points 1003 of the first video frame, and the key points 1004 of the second video frame are input to the optical flow estimation network 1005. Based on the key points 1003 of the first video frame and the key points 1004 of the second video frame, affine transformation parameters of the key point neighborhood can be generated. Then, based on the key point information and the affine transformation parameters, optical flow estimation can be performed by the optical flow estimation network 1005 to generate a dense motion field 1006 from the first video frame to the second video frame.

[0132] The dense motion field 1006 can be used to perform feature warping on first image features of different feature depths to obtain corresponding warped features.

[0133] The feature restoration network includes multiple deconvolution layers 1007 of the second image feature depth in sequence. After each deconvolution layer 1007 performs deconvolution processing on the input features in sequence, the second image features at different feature depths can be restored.

[0134] At the same feature depth, the deformation feature and the second image feature may be fused to achieve image reconstruction from the first video frame to the second video frame.

[0135] The network output layer may be a fully connected layer 1008 with different output dimensions, and based on the fully connected layer 1008 , reconstructed images corresponding to different image resolutions may be output.

[0136] In the embodiment of the present application, without changing the scale of the reference frame face, the image generation network can output reconstructed faces of different scales to meet the display requirements of different sizes. Since the AI face coding scheme based on semantic information is very sensitive to face resolution, when the face resolution in the video conference is improved, the overall complexity of the scheme will also increase, making it difficult to meet the low latency requirements of real-time video communication. For this reason, the resolution of the reference frame at the encoding end is 256x256 by default. At the decoding end, in addition to generating a 256x256 driving frame face, a higher resolution (such as 512x512) driving frame face can also be output. For this reason, a multi-scale output layer is used at the output end of the generation network, and a multi-scale supervisory signal (dual supervisory signal of 256 and 512 resolutions) is used during training. In this way, without significantly increasing the time complexity, the face generation network in the embodiment of the present application can output a multi-scale reconstructed face at the decoding end. It is worth noting that with the optimization of multi-scale supervision, not only can the 256x256 reference frame be used to generate a 512x512 driving frame to reconstruct the face, but the reconstructed face at the 256x256 scale is also clearer than the face obtained by single-scale supervision.

[0137] In one embodiment of the present application, although supervised key points are used to describe the movement of the mouth as much as possible, due to the limited number of key points, it is still difficult to accurately describe the movement of the mouth. Considering that video conferencing is generally accompanied by speech signals, and speech signals are highly correlated with the movement of the mouth of the face, this embodiment of the present application can combine speech signals to repair the mouth area in the generated facial image, making the mouth posture more realistic and natural.

[0138] Figure 11 The schematic diagram shows the principle of repairing the reconstructed image based on audio data in one embodiment of the present application. Figure 11 As shown, in the embodiment of the present application, audio data 1101 associated with the image coding data can be obtained. Based on the long short-term memory network LSTM or other network models, feature extraction can be performed on the audio data 1101 to obtain corresponding audio features.

[0139] Key point detection is performed on the reconstructed image 1102 of the second video frame to obtain key points 1103 of the target object in the reconstructed image.

[0140] According to the distribution positions of the key points 1103 in the reconstructed image, the key points of the target area located inside the target object can be determined; for example, the key points of the mouth area of a human face can be determined.

[0141] The key points 1103 of the target area are position-corrected according to the audio data 1101 to obtain corrected key points 1104. In the embodiment of the present application, a pre-trained multi-layer perceptron model MLP can be used to map the reconstructed key points and audio features to obtain the corrected key points.

[0142] The reconstructed image 1102 is locally modified according to the corrected key points 1104 to obtain a repaired reconstructed image 1105 .

[0143] In one embodiment of the present application, the image generation network can be jointly trained by configuring an image discrimination network or a similarity discrimination network, thereby obtaining an image generation network that can clearly and accurately reconstruct images.

[0144] Figure 12 The network architecture for joint training of models in one embodiment of the present application is schematically shown. Figure 12 As shown, the overall network architecture includes an image generation network 1201, an image discrimination network 1202 and a similarity discrimination network 1203.

[0145] In one embodiment of the present application, a method for jointly training the image generation network 1201 and the image discrimination network 1202 may include the following content.

[0146] First, a first image sample including a target object and a second image sample having content relevance to the first image sample may be obtained.

[0147] Key point detection is performed on the first image sample (such as the i-1th frame image Frame i-1) and the second image sample (such as the i-th frame image Frame i) respectively to obtain the first key point of the target object in the first image sample and the second key point of the target object in the second image sample.

[0148] An image generation network 1201 for generating a reconstructed image from an original image and an image discrimination network 1202 for discriminating whether an image is an original image or a reconstructed image are obtained.

[0149] The first image sample, the first key point and the second key point are input into the image generation network 1201 , so as to perform image reconstruction processing on the first image sample through the image generation network 1201 to obtain a reconstructed image Gen i of the second image sample.

[0150] The reconstructed image Gen i of the second image sample and the second image sample Frame i are respectively input into the image discrimination network 1202, so as to perform discrimination processing on the input image through the image discrimination network 1202 to obtain discrimination information of whether the input image is the original image or the reconstructed image.

[0151] The network parameters of the image generation network and the image discrimination network are updated alternately according to the discriminant information.

[0152] In one embodiment of the present application, a method for jointly training the image generation network 1201 and the similarity discrimination network 1203 may include the following content.

[0153] Acquire a third image sample (such as the i-1th image frame Frame i-1) including the target object and a fourth image sample (such as the i-th image frame Frame i) having a temporal correlation with the third image sample;

[0154] Performing key point detection on the third image sample Frame i-1 and the fourth image sample Frame i respectively to obtain a third key point of the target object in the third image sample Frame i-1 and a fourth key point of the target object in the fourth image sample Frame i;

[0155] An image generation network 1201 for generating a reconstructed image from an original image and a similarity determination network 1203 for determining similarity are obtained.

[0156] Inputting the third image sample Frame i-1, the fourth image sample Frame i, the third key point, and the fourth key point into the image generation network, so as to perform image reconstruction processing on the third image sample and the fourth image sample respectively through the image generation network to obtain a reconstructed image of the third image sample and a reconstructed image of the fourth image sample;

[0157] Inputting the reconstructed image of the third image sample and the reconstructed image of the fourth image sample into the similarity discrimination network 1203, so as to perform discrimination processing on the input images through the similarity discrimination network 1203 to obtain image similarity between the reconstructed image of the third image sample and the reconstructed image of the fourth image sample;

[0158] The network parameters of the image generation network 1201 and the similarity discrimination network 1203 are updated according to the image similarity.

[0159] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0160] The following introduces an embodiment of the device of the present application, which can be used to execute the image processing method in the above-mentioned embodiment of the present application.

[0161] Figure 13 The structural block diagram of the image processing device at the decoding end provided by the embodiment of the present application is schematically shown. Figure 13 As shown, the image processing apparatus 1300 includes:

[0162] The acquisition module 1310 is configured to acquire a video segment to be transmitted, wherein the video segment includes multiple video frames; the extraction module 1320 is configured to extract a first video frame including the target object and a second video frame having content relevance to the first video frame from the video segment; the detection module 1330 is configured to perform key point detection on the video frame to obtain the key points of the target object in the video frame, and the key points are used to represent the motion posture of the target object in the video frame; the encoding module 1340 is configured to encode the first video frame, the key points in the first video frame, and the key points in the second video frame to obtain image encoding data; the transmission module 1350 is configured to transmit the image encoding data to the decoding end, so that the decoding end generates a reconstructed image of the second video frame based on the image encoding data.

[0163] In some embodiments of the present application, based on the above embodiments, the detection module 1330 is configured to: input the video frame into a pre-trained target object recognition model; perform key point detection on the target object in the video frame through the target object recognition model to obtain a first key point for representing the overall motion posture of the target object and a second key point for representing the motion posture of the target area inside the target object.

[0164] In some embodiments of the present application, based on the above embodiments, the image processing device 1300 further includes: a sample acquisition module, configured to acquire an image sample including a target object and a sample label associated with the image sample, wherein the sample label is used to mark the position coordinates of the second key point in the image sample; a sample input module, configured to input the image sample and the sample label into the target object recognition model; a sample detection module, configured to perform key point detection on the target object in the image sample through the target object recognition model to obtain the first key point and the second key point of the target object in the image sample; a parameter updating module, configured to obtain the loss error between the sample label and the detected second key point, and update the model parameters of the target object recognition model according to the loss error.

[0165] In some embodiments of the present application, based on the above embodiments, the extraction module 1320 is configured to: perform feature extraction on each video frame in the video clip to obtain image features for representing the image content of the video frame; perform clustering processing on the video frames according to the image features to obtain clusters composed of video frames with content relevance and the cluster centers of the clusters; determine the video frame including the target object in the cluster that is closest to the cluster center as the first video frame, and determine the other video frames in the cluster as second video frames with content relevance to the first video frame.

[0166] In some embodiments of the present application, based on the above embodiments, the encoding module 1340 is configured to: perform network detection on the data transmission network used to transmit the video clip to determine the network type and network signal strength of the data transmission network; determine whether the data transmission network belongs to a weak network based on the network type and network signal strength; if the data transmission network belongs to a weak network, encode the first video frame, the key points in the first video frame, and the key points in the second video frame; if the data transmission network does not belong to a weak network, encode the first video frame, the second video frame, the key points in the first video frame, and the key points in the second video frame.

[0167] Figure 14 The structural block diagram of the image processing device at the encoding end provided by the embodiment of the present application is schematically shown. Figure 14 As shown, the image processing device 1400 includes: a receiving module 1410, configured to receive image coding data of a video clip transmitted by an encoding end; a decoding module 1420, configured to decode the image coding data to obtain a first video frame, key points in the first video frame, and key points in the second video frame, wherein the key points are used to represent the motion posture of the target object in the video frame; a generating module 1430, configured to generate transformation information of the motion posture of the target object based on the key points in the first video frame and the key points in the second video frame; and a reconstruction module 1440, configured to generate a reconstructed image of the second video frame based on the first video frame and the transformation information.

[0168] In some embodiments of the present application, based on the above embodiments, the transformation information of the motion posture of the target object includes the motion field between the first video frame and the second video frame; the generation module 1430 is configured to: establish a one-to-one corresponding key point mapping relationship between the key points in the first video frame and the key points in the second video frame; generate affine transformation parameters of each key point neighborhood according to the key point mapping relationship; perform optical flow estimation on the first video frame according to the affine transformation parameters, the key points in the first video frame and the key points in the second video frame to obtain the motion field between the first video frame and the second video frame.

[0169] In some embodiments of the present application, based on the above embodiments, the reconstruction module 1440 is configured to: perform feature extraction on the first video frame to obtain multiple first image features corresponding to different feature depths; perform transformation processing on the first image features of each feature depth according to the transformation information to obtain deformation features corresponding to different feature depths; perform feature restoration on the first image features to obtain second image features corresponding to different feature depths; perform feature fusion on the deformation features and the second image features corresponding to the same feature depth, and perform mapping processing on the fused features to obtain a reconstructed image of the second video frame.

[0170] In some embodiments of the present application, based on the above embodiments, the reconstruction module 1440 is further configured to: map the fused features separately through multiple output networks with different network scales to obtain multiple reconstructed images of the second video frame, and the multiple reconstructed images have different image resolutions.

[0171] In some embodiments of the present application, based on the above embodiments, the image processing device 1400 also includes: an audio acquisition module, configured to acquire audio data associated with the image encoding data; an image detection module, configured to perform key point detection on the reconstructed image of the second video frame to obtain the key points of the target object in the reconstructed image; a key point determination module, configured to determine the key points of the target area located inside the target object based on the distribution positions of each key point in the reconstructed image; a position correction module, configured to perform position correction on the key points of the target area according to the audio data to obtain corrected key points; an image repair module, configured to perform local image modification on the reconstructed image according to the corrected key points to obtain a repaired reconstructed image.

[0172] In some embodiments of the present application, based on the above embodiments, the image processing device 1400 further includes: a first sample acquisition module, configured to acquire a first image sample including a target object and a second image sample having content relevance to the first image sample; a first sample detection module, configured to perform key point detection on the first image sample and the second image sample respectively, to obtain a first key point of the target object in the first image sample and a second key point of the target object in the second image sample; a first network acquisition module, configured to acquire an image generation network for generating a reconstructed image based on an original image and an image judgment network for determining whether an image is an original image or a reconstructed image. a first image reconstruction module, configured to input the first image sample, the first key point and the second key point into the image generation network, so as to perform image reconstruction processing on the first image sample through the image generation network to obtain a reconstructed image of the second image sample; an image discrimination module, configured to input the reconstructed image of the second image sample and the second image sample into the image discrimination network respectively, so as to perform discrimination processing on the input image through the image discrimination network to obtain discrimination information of whether the input image is the original image or the reconstructed image; a first parameter updating module, configured to alternately update the network parameters of the image generation network and the image discrimination network according to the discrimination information.

[0173] In some embodiments of the present application, based on the above embodiments, the image processing device 1400 further includes: a second sample acquisition module, configured to acquire a third image sample including a target object and a fourth image sample having a time correlation with the third image sample; a second sample detection module, configured to perform key point detection on the third image sample and the fourth image sample, respectively, to obtain a third key point of the target object in the third image sample and a fourth key point of the target object in the fourth image sample; a second model acquisition module, configured to acquire an image generation network for generating a reconstructed image based on the original image and a similarity discrimination network for discriminating similarity; a second image reconstruction module, configured to convert the third image sample and the fourth image sample into a reconstructed image; The image sample, the third key point and the fourth key point are input into the image generation network, so that the image generation network performs image reconstruction processing on the third image sample and the fourth image sample respectively to obtain the reconstructed image of the third image sample and the reconstructed image of the fourth image sample; the similarity discrimination module is configured to input the reconstructed image of the third image sample and the reconstructed image of the fourth image sample into the similarity discrimination network, so that the input image is discriminated by the similarity discrimination network to obtain the image similarity between the reconstructed image of the third image sample and the reconstructed image of the fourth image sample; the second parameter updating module is configured to update the network parameters of the image generation network and the similarity discrimination network according to the image similarity.

[0174] The specific details of the image processing device provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0175] Figure 15 The block diagram schematically shows a computer system structure of an electronic device used to implement an embodiment of the present application.

[0176] It should be noted that Figure 15 The computer system 1500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0177] like Figure 15As shown, the computer system 1500 includes a central processing unit (CPU) 1501, which can perform various appropriate actions and processes according to the program stored in the read-only memory 1502 (ROM) or the program loaded from the storage part 1508 into the random access memory 1503 (RAM). Various programs and data required for system operation are also stored in the random access memory 1503. The CPU 1501, the read-only memory 1502, and the random access memory 1503 are connected to each other via a bus 1504. An input / output interface 1505 (i.e., an I / O interface) is also connected to the bus 1504.

[0178] The following components are connected to the input / output interface 1505: an input section 1506 including a keyboard, a mouse, and the like; an output section 1507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1508 including a hard disk; and a communication section 1509 including a network interface card such as a local area network card or a modem. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the input / output interface 1505 as needed. Removable media 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1510 as needed, so that computer programs read therefrom can be installed into the storage section 1508 as needed.

[0179] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1509 and / or installed from a removable medium 1511. When the computer program is executed by the central processing unit 1501, the various functions defined in the system of the present application are performed.

[0180] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0183] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0184] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0185] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: receiving image encoding data of a video segment transmitted by an encoding end; Decoding the coded image data to obtain a first video frame, key points in the first video frame, and key points in a second video frame; the first video frame is a key frame containing a target object, the second video frame and the first video frame have the same target object, the key points are used to represent a motion posture of the target object in the video frame, and the target object has a similar motion posture in the first video frame and the second video frame; generating transformation information of the motion posture of the target object according to the key points in the first video frame and the key points in the second video frame; Performing feature extraction on the first video frame to obtain a plurality of first image features corresponding to different feature depths; Performing transformation processing on the first image features of each feature depth respectively according to the transformation information to obtain deformation features corresponding to different feature depths; Performing feature restoration on the first image feature to obtain a second image feature corresponding to a different feature depth; The deformation features corresponding to the same feature depth are fused with the second image features, and the fused features are mapped to obtain a reconstructed image of the second video frame.

2. The image processing method according to claim 1, wherein: The transformation information of the motion posture of the target object includes a motion field between the first video frame and the second video frame; and generating the transformation information of the motion posture of the target object according to key points in the first video frame and key points in the second video frame includes: Establishing a one-to-one key point mapping relationship between key points in the first video frame and key points in the second video frame; Generate affine transformation parameters of the neighborhood of each key point according to the key point mapping relationship; Optical flow estimation is performed on the first video frame according to the affine transformation parameters, key points in the first video frame, and key points in the second video frame to obtain a motion field between the first video frame and the second video frame.

3. The image processing method according to claim 1, wherein: Mapping the fused features to obtain a reconstructed image of the second video frame includes: The fused features are mapped and processed respectively through multiple output networks with different network scales to obtain multiple reconstructed images of the second video frame, where the multiple reconstructed images have different image resolutions.

4. The image processing method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquiring audio data associated with the image encoding data; Performing key point detection on the reconstructed image of the second video frame to obtain key points of the target object in the reconstructed image; Determining key points of a target area located inside the target object according to distribution positions of the key points in the reconstructed image; Correcting the positions of the key points of the target area according to the audio data to obtain corrected key points; The reconstructed image is locally modified according to the corrected key points to obtain a repaired reconstructed image.

5. The image processing method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a first image sample including a target object and a second image sample having content relevance to the first image sample; Performing key point detection on the first image sample and the second image sample respectively to obtain a first key point of the target object in the first image sample and a second key point of the target object in the second image sample; Obtaining an image generation network for generating a reconstructed image based on an original image and an image discrimination network for discriminating whether an image is an original image or a reconstructed image; Inputting the first image sample, the first key point, and the second key point into the image generation network, so as to perform image reconstruction processing on the first image sample through the image generation network to obtain a reconstructed image of the second image sample; inputting the reconstructed image of the second image sample and the second image sample into the image discrimination network respectively, so as to perform discrimination processing on the input image through the image discrimination network to obtain discrimination information of whether the input image is an original image or a reconstructed image; The network parameters of the image generation network and the image discrimination network are alternately updated according to the discrimination information.

6. The image processing method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a third image sample including the target object and a fourth image sample having a temporal correlation with the third image sample; performing key point detection on the third image sample and the fourth image sample respectively to obtain a third key point of the target object in the third image sample and a fourth key point of the target object in the fourth image sample; Obtaining an image generation network for generating a reconstructed image based on an original image and a similarity discrimination network for discriminating similarity; inputting the third image sample, the fourth image sample, the third key point, and the fourth key point into the image generation network, so as to perform image reconstruction processing on the third image sample and the fourth image sample respectively through the image generation network to obtain a reconstructed image of the third image sample and a reconstructed image of the fourth image sample; inputting the reconstructed image of the third image sample and the reconstructed image of the fourth image sample into the similarity discrimination network, so as to perform discrimination processing on the input images through the similarity discrimination network to obtain image similarity between the reconstructed image of the third image sample and the reconstructed image of the fourth image sample; The network parameters of the image generation network and the similarity discrimination network are updated according to the image similarity.

7. The image processing method according to claim 1, wherein: The method for transmitting the image coded data from the encoding end to the decoding end includes: Acquire the video clip to be transmitted, where the video clip includes multiple video frames; extracting, from the video clip, the first video frame including the target object and the second video frame having content relevance to the first video frame; Performing key point detection on the video frame to obtain key points of the target object in the video frame; performing encoding processing on the first video frame, key points in the first video frame, and key points in the second video frame to obtain the image encoding data; The image coding data is transmitted to a decoding end, so that the decoding end generates a reconstructed image of the second video frame according to the image coding data.

8. The image processing method according to claim 7, wherein: Performing key point detection on the video frame, including: Inputting the video frame into a pre-trained target object recognition model; Key point detection is performed on the target object in the video frame using the target object recognition model to obtain a first key point for representing the overall motion posture of the target object and a second key point for representing the motion posture of the target area inside the target object.

9. The image processing method according to claim 8, characterized in that: The method further comprises: Acquire an image sample including a target object and a sample label associated with the image sample, where the sample label is used to mark the position coordinates of the second key point in the image sample; Inputting the image sample and the sample label into the target object recognition model; Performing key point detection on the target object in the image sample using the target object recognition model to obtain a first key point and a second key point of the target object in the image sample; Obtain a loss error between the sample label and the second key point obtained by detection, and update model parameters of the target object recognition model according to the loss error.

10. The image processing method according to any one of claims 7 to 9, characterized in that: Extracting a first video frame including the target object and a second video frame having content relevance to the first video frame from the video clip includes: Performing feature extraction on each video frame in the video clip to obtain image features for representing image content of the video frame; performing clustering processing on the video frames according to the image features to obtain clusters consisting of video frames with content relevance and cluster centers of the clusters; A video frame including a target object that is closest to the cluster center in the cluster is determined as a first video frame, and other video frames in the cluster are determined as second video frames having content relevance with the first video frame.

11. An image processing device, characterized in that: include: a receiving module configured to receive image coded data of a video segment transmitted by an encoding end; a decoding module configured to decode the image coded data to obtain a first video frame, key points in the first video frame, and key points in a second video frame; the first video frame is a key frame containing a target object, the second video frame and the first video frame have the same target object, the key points are used to represent a motion posture of the target object in the video frame, and the target object has a similar motion posture in the first video frame and the second video frame; a generating module configured to generate transformation information of the motion posture of the target object according to the key points in the first video frame and the key points in the second video frame; The reconstruction module is configured to perform feature extraction on the first video frame to obtain multiple first image features corresponding to different feature depths; transform the first image features of each feature depth according to the transformation information to obtain deformation features corresponding to different feature depths; restore the first image features to obtain second image features corresponding to different feature depths; fuse the deformation features and the second image features corresponding to the same feature depth, and map the fused features to obtain a reconstructed image of the second video frame.

12. A computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

13. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the image processing method according to any one of claims 1 to 10 by executing the executable instructions.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

Images