Image processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide an image processing method, device, electronic device and storage medium, which relate to the field of image processing technology. The method is applied to a server, and includes: obtaining the performance status of a terminal device; generating information to be processed through a digital human model according to the performance status, wherein the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; sending the at least one frame of video image and the optical flow information to the terminal device, so that the terminal device performs interpolation processing on the at least one frame of video image according to the optical flow information. Since the data volume of the optical flow information is small, the amount of calculation required to generate the optical flow information is small, and the amount of calculation required to transmit the optical flow information is also small, so that the method can save the computing resources of the server, improve the computing speed of the server, and reduce the burden on the system.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technology, and more specifically, to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of image processing technology and computer technology, artificial intelligence has become a widely studied topic. Among them, digital humans included in artificial intelligence can be used in banking, news broadcasting and other fields. At present, in the process of real-time interaction between digital humans and users, it is necessary to synthesize digital human images in real time. As we all know, synthesizing digital human images requires synthesizing digital human videos through video images frame by frame. Therefore, the generation efficiency of synthesized digital human images is low, the cost is high, and it also takes up a lot of system memory, which increases the system burden. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a server, and the method includes: obtaining a performance status of a terminal device; generating information to be processed through a digital human model based on the performance status, the information to be processed including at least one frame of video image to be processed and optical flow information to be processed; sending at least one frame of video image and the optical flow information to the terminal device, so that the terminal device performs frame interpolation processing on at least one frame of video image according to the optical flow information.

[0005] Optionally, information to be processed is generated through a digital human model according to the performance state, including: when the performance state satisfies a first preset state, determining a first image processing strategy and a first digital human model corresponding to the first preset state; according to the first image processing strategy, controlling the first digital human model to generate the first frame of video image to be processed and optical flow information corresponding to each frame of video image in a target video image sequence to be processed, wherein the target video image sequence includes the first frame of video image.

[0006] Optionally, when the performance state satisfies a first preset state, before determining a first image processing strategy and a first digital human model corresponding to the first preset state, the image processing method further includes: obtaining a sample training set, the sample training set including sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images respectively; taking the sample interaction information as input, taking the first frame of the sample video image and the optical flow information corresponding to the multiple frames of the sample video images respectively as output, and obtaining the first digital human model based on training of a machine learning algorithm.

[0007] Optionally, based on the performance status, information to be processed is generated through a digital human model, including: when the performance status satisfies a second preset state, determining a second image processing strategy and a second digital human model corresponding to the second preset state; based on the second image processing strategy, controlling the second digital human model to generate multiple frames of video images to be processed and optical flow information corresponding to the multiple frames of video images to be processed.

[0008] Optionally, when the performance state satisfies the second preset state, before determining the second image processing strategy and the second digital human model corresponding to the second preset state, the image processing method also includes: obtaining a sample training set, the sample training set including sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images; taking the sample interaction information as input, and the multiple frames of sample video images and the optical flow information as output, and obtaining the second digital human model based on training of a machine learning algorithm.

[0009] Optionally, after obtaining the performance status of the terminal device, the image processing method also includes: when the performance status does not meet the target status, generating a default video image through a default digital human model, the default video image is a default reply video image determined by the server based on the interactive information input by the user, and the target status is used to characterize the minimum performance status required for the terminal device to perform the interpolation processing operation; based on the default video image, generating a default reply video, and sending the default reply video to the terminal device.

[0010] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to a terminal device, and the method includes: receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server; performing interpolation processing on at least one frame of video image based on the at least one frame of video image and the optical flow information; and obtaining a target video image sequence after the interpolation processing.

[0011] Optionally, at least one frame of video image to be processed is a first frame of video image, and the optical flow information to be processed is optical flow information corresponding to each frame of video image in a target video image sequence to be processed. Interpolation processing is performed on at least one frame of video image based on the at least one frame of video image and the optical flow information, including: generating an intermediate frame of video image based on the first frame of video image and the optical flow information; generating a final frame of video image based on the intermediate frame of video image and the optical flow information corresponding to the intermediate frame of video image; and synthesizing a target video image sequence based on the first frame of video image, the intermediate frame of video image and the final frame of video image.

[0012] Optionally, at least one frame of video image to be processed is a multi-frame video image, the multi-frame video image includes multiple groups of adjacent video frame images, each group of adjacent video frame images in the multiple groups of adjacent video frame images includes a forward frame video image and a backward frame video image, the optical flow information to be processed is the optical flow information corresponding to the multiple frames of video images, and at least one frame of video image is interpolated according to the at least one frame of video image and the optical flow information, including: determining a first transition frame video image according to the forward frame video image and the optical flow information corresponding to the forward frame video image; determining a second transition frame video image according to the backward frame video image and the optical flow information corresponding to the backward frame video image; generating a target transition frame video image according to the first transition frame video image and the second transition frame video image; and inserting the target transition frame video image between the forward frame video image and the backward frame video image to generate a target video image sequence.

[0013] Optionally, at least one frame of video image to be processed is a default reply video generated by the server based on interactive information input by the user. After receiving at least one frame of video image to be processed sent by the server, the image processing method also includes: receiving the default reply video sent by the server, and using the default reply video as the target video image sequence; outputting the target video image sequence.

[0014] In a third aspect, an embodiment of the present application provides an image processing method, which is applied to an image processing system, wherein the image processing system includes a server and a terminal device, and the server and the terminal device are electrically connected. The method includes: the server obtains a performance status of the terminal device; the server generates information to be processed through a digital human model based on the performance status, and the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; the server sends at least one frame of video image and the optical flow information to the terminal device; the terminal device performs frame interpolation processing on at least one frame of video image based on the at least one frame of video image and the optical flow information; the terminal device obtains a target video image sequence after the frame interpolation processing.

[0015] In a fourth aspect, an embodiment of the present application provides an image processing device, which is applied to a server, and the device includes: an acquisition module, which is used to obtain the performance status of a terminal device; a generation module, which is used to generate information to be processed through a digital human model according to the performance status, wherein the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; a sending module, which is used to send at least one frame of video image and optical flow information to the terminal device, so that the terminal device performs interpolation processing on at least one frame of video image according to the optical flow information.

[0016] Optionally, the generation module includes: a first determination submodule, used to determine a first image processing strategy and a first digital human model corresponding to the first preset state when the performance state satisfies a first preset state; a first generation submodule, used to control the first digital human model to generate the first frame of video image to be processed and optical flow information corresponding to each frame of video image in the target video image sequence to be processed according to the first image processing strategy, wherein the target video image sequence includes the first frame of video image.

[0017] Optionally, the image processing device also includes: a first sample acquisition module, used to acquire a sample training set, the sample training set including sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images; a first model training module, used to take the sample interaction information as input, and take the first frame of sample video image and the optical flow information corresponding to the multiple frames of sample video images as output, and obtain the first digital human model based on machine learning algorithm training.

[0018] Optionally, the generation module also includes: a second determination submodule, used to determine a second image processing strategy and a second digital human model corresponding to the second preset state when the performance state satisfies the second preset state; a second generation submodule, used to control the second digital human model to generate multiple frames of video images to be processed and optical flow information corresponding to the multiple frames of video images to be processed according to the second image processing strategy.

[0019] Optionally, the image processing device also includes: a second sample acquisition module, used to acquire a sample training set, the sample training set including sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images; a second model training module, used to take the sample interaction information as input and the multiple frames of sample video images and the optical flow information as output, and obtain a second digital human model based on machine learning algorithm training.

[0020] Optionally, the generation module also includes: a third determination submodule, which is used to generate a default video image through a default digital human model when the performance status does not meet the target status, the default video image is a default reply video image determined by the server based on the interactive information input by the user, and the target status is used to characterize the minimum performance status required for the terminal device to perform the interpolation processing operation; the third generation submodule is used to generate a default reply video based on the default video image, and send the default reply video to the client.

[0021] In a fifth aspect, an embodiment of the present application provides an image processing device, which is applied to a terminal device, and the device includes: a receiving module, which is used to receive at least one frame of video image to be processed and optical flow information to be processed sent by a server; a processing module, which is used to perform interpolation processing on at least one frame of video image according to at least one frame of video image and the optical flow information; and a synthesis module, which is used to obtain a target video image frame sequence after interpolation processing.

[0022] Optionally, at least one frame of video image to be processed is a first frame of video image, and the optical flow information to be processed is the optical flow information corresponding to each frame of video image in the target video image to be processed. The processing module includes: a first generation submodule, used to generate an intermediate frame of video image according to the first frame of video image and the optical flow information corresponding to the first frame of video image; a second generation submodule, used to generate a final frame of video image according to the intermediate frame of video image and the optical flow information corresponding to the intermediate frame of video image; a synthesis submodule, used to synthesize a target video image sequence according to the first frame of video image, the intermediate frame of video image and the final frame of video image.

[0023] Optionally, at least one frame of video image to be processed is a multi-frame video image, the multi-frame video image includes multiple groups of adjacent video frame images, each group of adjacent video frame images in the multiple groups of adjacent video frame images includes a forward frame video image and a backward frame video image, and the optical flow information to be processed is the optical flow information corresponding to the multi-frame video image. The processing module also includes: a first transition frame determination submodule, used to determine the first transition frame video image according to the forward frame video image and the optical flow information corresponding to the forward frame video image; a second transition frame determination submodule, used to determine the second transition frame video image according to the backward frame video image and the optical flow information corresponding to the backward frame video image; a target transition frame generation submodule, used to generate a target transition frame video image according to the first transition frame video image and the second transition frame video image; a sequence generation submodule, used to insert the target transition frame video image between the forward frame video image and the backward frame video image to generate a target video image sequence.

[0024] Optionally, at least one frame of video image to be processed is a default reply video generated by the server based on interactive information input by the user, and the image processing device also includes: a determination module for using the default reply video as a target video image sequence; and an output module for outputting the target video image sequence.

[0025] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory; one or more processors, the one or more processors being coupled to the memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the image processing method provided in an embodiment of the present application.

[0026] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the image processing method provided by the embodiment of the present application.

[0027] The embodiment of the present application provides an image processing method, device, electronic device and storage medium. The method is applied to a server, and the method obtains the performance status of a terminal device; generates information to be processed through a digital human model according to the performance status, and the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; sends at least one frame of video image and optical flow information to the terminal device, so that the terminal device performs interpolation processing on at least one frame of video image according to the optical flow information. Since the data volume of the optical flow information is small, the amount of calculation required to generate the optical flow information is small, and the amount of calculation required to transmit the optical flow information is also small, so the method can save the computing resources of the server, improve the computing speed of the server, and reduce the system burden. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. It should be noted that the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0029] Figure 1 A schematic diagram showing an application environment of an image processing method provided by an embodiment of the present application is shown;

[0030] Figure 2 A schematic diagram showing a flow chart of an image processing method provided by another embodiment of the present application is shown;

[0031] Figure 3 A schematic diagram showing a flow chart of an image processing method provided by another embodiment of the present application is shown;

[0032] Figure 4 A schematic diagram showing a flow chart of an image processing method provided in yet another embodiment of the present application is shown;

[0033] Figure 5 A schematic diagram showing a flow chart of an image processing method provided by another embodiment of the present application is shown;

[0034] Figure 6 A schematic diagram showing a flow chart of an image processing method provided by yet another embodiment of the present application;

[0035] Figure 7 A schematic diagram showing a flow chart of an image processing method provided by yet another embodiment of the present application is shown;

[0036] Figure 8 A schematic diagram showing a flow chart of an image processing method provided by yet another embodiment of the present application is shown;

[0037] Fig. 9 A schematic diagram showing a flow chart of an image processing method provided in yet another embodiment of the present application is shown;

[0038] Fig.10 A schematic diagram showing a flow chart of an image processing method provided by yet another embodiment of the present application is shown;

[0039] Fig.11 A schematic diagram of a process of executing an image processing method provided by an exemplary embodiment of the present application is shown;

[0040] Fig.12 A structural block diagram of an image processing device provided by an embodiment of the present application is shown;

[0041] Fig.13 A structural block diagram of an image processing device provided by another embodiment of the present application is shown;

[0042] Fig.14 A structural block diagram of an electronic device provided by an embodiment of the present application is shown;

[0043] Fig.15 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0045] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0046] See also Figure 1 , Figure 1A schematic diagram of an application environment for image processing provided by an embodiment of the present application is shown. Among them, the image processing system 100 may include a server 110 and a terminal device 120. The server 110 is electrically connected to the terminal device 120. The server 110 may be a cloud server or a traditional server. The terminal device 120 may include but is not limited to an intelligent robot, a tablet computer, a smart phone, etc., and the embodiment of the present application does not limit the specific type of the terminal device 120. The server 110 and the terminal device 120 can communicate and exchange data through technologies such as Wireless-Fidelity (WI-FI) and cellular.

[0047] See also Figure 2 , Figure 2 A schematic flow chart of an image processing method provided by another embodiment of the present application is shown. The image processing method is applied to the server 110, and the method may specifically include the following steps:

[0048] Step S110: Acquire the performance status of the terminal device.

[0049] Among them, the performance state of the terminal device 120 can be determined according to the computing resources of the terminal device 120, and more specifically, can be determined according to the computing power (i.e., computing power, also known as hash rate) of the terminal device 120. Generally, the computing power of the terminal device 120 with better configuration is higher, and the performance state of the terminal device with higher computing power is better, wherein the configuration of the terminal device 120 may include at least one configuration such as a central processing unit (CPU), a graphics processing unit (GPU), and a memory; generally, the computing power of the terminal device 120 with poorer configuration is lower, and the performance state of the terminal device with lower computing power is poor. In addition, the less the content currently processed by the CPU or / and GPU of the terminal device 120, the larger the memory, it can be determined that the system of the terminal device 120 is idle at this time, the computing power of the terminal device 120 is high, and the performance state of the terminal device is good at this time; and the more the content currently processed by the CPU / and GPU of the terminal device, the smaller the memory, it can be determined that the system of the terminal device is busy at this time, the computing power of the terminal device 120 is low, and the performance state of the terminal device is poor at this time. The system busyness of the terminal device 120 can be determined by the terminal device 120 (or the server 110) obtaining the number of applications (or loads) running on the terminal device 120. When the number of applications (or loads) exceeds the system threshold, the system can be determined to be busy. When the number of applications (or loads) does not exceed the system threshold, the system can be determined to be idle. The system threshold can be set according to actual needs. For example, the system threshold can be 5. The above method of determining the performance status of the terminal device is only an example and is not limited here.

[0050] In some embodiments, the server 110 may receive status information sent by the terminal device 120. The status information may include configuration information of the terminal device 120 and the current system busy status of the terminal device 120. The configuration information may include at least one of the percentage of the CPU of the terminal device 120 that can be allocated to the total amount, the percentage of the GPU that can be allocated to the total amount, and the percentage of the remaining memory of the terminal device 120 to the total memory. The system busy status may include system idle, system busy, and other states. The server may determine the performance status of the terminal device 120 based on the status information sent by the terminal device.

[0051] Next, taking the configuration information of the percentage of the CPU of the terminal device 120 that can be allocated, the percentage of the GPU that can be allocated, and the percentage of the remaining memory of the terminal device 120 as an example, it is explained how the server 110 obtains the performance status of the terminal device 120.

[0052] It should be noted that in the following implementations, the first CPU threshold, the first GPU threshold, the first memory threshold, the second CPU threshold, the second GPU threshold, and the second memory threshold can all be set according to actual needs, and the embodiments of the present application do not impose specific restrictions on the values ​​of these thresholds. For example, the first CPU threshold can be 75%; the first GPU threshold can be 80%; the first memory threshold can be 60%; the second CPU threshold can be 50%; the second GPU threshold can be 50% and the second memory threshold can be 40%.

[0053] In some embodiments, the server 110 may obtain the percentage of the CPU of the terminal device 120 that can be allocated, the percentage of the GPU that can be allocated, and the percentage of the remaining memory of the terminal device 120 that accounts for the total memory. When the percentage of the part of the CPU that can be allocated in the total amount is greater than the first CPU threshold, the percentage of the part of the GPU that can be allocated in the total amount is greater than the first GPU threshold, and the percentage of the remaining memory in the total memory is greater than the first memory threshold, it can be explained that the computing power of the terminal device 120 is very high at this time, and then the performance state of the terminal device 120 can be determined to be very good; when the percentage of the part of the CPU that can be allocated in the total amount is less than the first CPU threshold and greater than the second CPU threshold, the percentage of the part of the GPU that can be allocated in the total amount is less than the first GPU threshold and greater than the second GPU threshold, and the percentage of the remaining memory in the total memory is less than the first memory threshold and greater than the second memory threshold, it can be explained that the computing power of the terminal device 120 is high at this time, and then the performance state of the terminal device 120 can be determined to be good; when the percentage of the part of the CPU that can be allocated in the total amount is less than the second CPU threshold, the percentage of the part of the GPU that can be allocated in the total amount is less than the second GPU threshold, and the percentage of the remaining memory in the total memory is less than the second memory threshold, it can be explained that the computing power of the terminal device 120 is low at this time, and then the performance state of the terminal device 120 can be determined to be poor.

[0054] Step S120: Generate information to be processed through the digital human model according to the performance status, wherein the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed.

[0055] In this embodiment, the server 110 may pre-train a digital human model based on the interactive information input by the user, and the digital human model is used to generate at least one of a video image, optical flow information, and a video sequence. The interactive information input by the user may include a voice, a text, and other content, and may also include content generated by the terminal device 120 when the user clicks a button of the terminal device 120. This application does not specifically limit the form of the interactive information input by the user and the content of the interactive information.

[0056] The information to be processed may be information generated by the digital human model according to the interactive information input by the user.

[0057] Among them, the at least one frame of video image can be one frame of video image, or multiple frames of video image, or a video image sequence, etc. The present application does not impose any specific restrictions on the content of the at least one frame of video image.

[0058] The optical flow information to be processed may be the image change information between each two adjacent frames of the video image sequence, and the image change information may include at least one of the image brightness change information and the image chromaticity change information. For example, the first optical flow information corresponding to the first frame of the video image may be the image change information between the first frame of the video image and the second frame of the video image; the second optical flow information corresponding to the second frame of the video image may be the image change information between the second frame of the video image and the third frame of the video image; and so on, the optical flow information corresponding to the second-to-last frame of the video image (i.e., the last optical flow information) may be the image change information between the second-to-last frame of the video image and the first-to-last frame of the video image.

[0059] In some implementations, following the above description of S110, when the performance status of the terminal device 120 is very good, the server 110 can control the digital human model to generate the first frame of video image and optical flow information.

[0060] In other embodiments, following the above description of S110, when the performance status of the terminal device 120 is good, the server 110 can control the digital human model to generate multiple frames of video images and optical flow information, wherein the multiple frames of video images may include key frame images in the default response video image sequence.

[0061] In some further implementations, following the above description of S110, when the performance status of the terminal device 120 is poor, the server 110 may control the digital human model to generate a default reply video image sequence.

[0062] Step S130: sending at least one frame of video image and optical flow information to a terminal device, so that the terminal device interpolates at least one frame of video image according to the optical flow information.

[0063] In some embodiments, when at least one frame of video image is the first frame of video image, the optical flow information to be processed is the optical flow information corresponding to each frame of video image in the target video image sequence to be processed, and the server 110 can send the first frame of video image and the optical flow information corresponding to each frame of video image in the target video image sequence to be processed to the terminal device 120.

[0064] In other embodiments, when at least one frame of video image is a multi-frame video image, the optical flow information to be processed is the optical flow information corresponding to the multiple frames of video images respectively, and the server 110 can send the multiple frames of video images and the optical flow information corresponding to the multiple frames of video images respectively to the terminal device 120.

[0065] In some further embodiments, when at least one frame of video image is a default reply video image sequence, the server 110 may send the default reply video image sequence to the terminal device 120 .

[0066] The image processing method provided by the embodiment of the present application obtains the performance status of the terminal device; generates information to be processed through a digital human model according to the performance status, wherein the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; sends at least one frame of video image and optical flow information to the terminal device, so that the terminal device interpolates at least one frame of video image according to the optical flow information. Thus, optical flow information is added during the image processing process. Since the data volume of optical flow information is small, the amount of calculation required to generate optical flow information is small, and the amount of calculation required to transmit optical flow information is also small, so this method can save the computing resources of the server, improve the computing speed of the server, and reduce the burden on the system.

[0067] See also Figure 3 , Figure 3 A schematic flow chart of an image processing method provided by another embodiment of the present application is shown. The image processing method is applied to a server. The image processing method may specifically include the following steps:

[0068] Step S210: Obtain the performance status of the terminal device.

[0069] The specific description of step S210 can be found in step S110 and will not be repeated here.

[0070] Step S220: obtaining a sample training set, where the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images.

[0071] In this embodiment, the sample training set can be a collection of multiple samples, which can be used to train a digital human model. The sample training set can include at least one content such as multiple videos, multiple images, and multiple voices. The embodiment of the present application does not limit the specific content of the sample training set. The sample training set can be collected in multiple scenarios. For example, in the scenario where bank staff handles business for customers, the conversation information between customers and bank staff can be collected; in the scenario where the news broadcast is recorded, the demeanor, movements, and other information of the news broadcaster during the broadcast can also be collected. This application does not limit the collection method and specific content of the sample training set.

[0072] The sample interaction information may include at least one type of information such as the user's conversation information, the user's voice, and the user's demeanor and actions during the conversation. The embodiment of the present application does not limit the specific content of the sample interaction information.

[0073] The multiple frames of sample video images corresponding to the sample interaction information may include a sequence of sample video images that respond to the sample interaction information.

[0074] The optical flow information corresponding to the multiple frames of sample video images may be image change information between each two adjacent frames of the multiple frames of sample video images, and the image change information may include at least one of image brightness change information and image chromaticity change information. For example, the optical flow information corresponding to the first frame of sample video images may be image change information between the first frame of sample video images moving to the second frame of sample video images; the optical flow information corresponding to the second frame of sample video images may be image change information between the second frame of sample video images moving to the third frame of sample video images; and so on, the optical flow information corresponding to the second-to-last frame of video images may be image change information between the second-to-last frame of video images moving to the first-to-last frame of video images.

[0075] In some embodiments, the server 110 can be connected to a camera (or a video recorder, etc.) and a computer. The camera (or video recorder) can record or shoot the sample training set, and store the collected sample training set in the computer through a video capture card. The video capture card is also called a video card, which is used to input the video signal output by the camera into the computer, and convert it into digital data that can be identified by the computer, and store it in the computer to become an editable and processable video data file. The server 110 can receive the sample training set sent by the computer.

[0076] Step S230: taking the sample interaction information as input, taking the optical flow information corresponding to the first frame of the sample video image and the multiple frames of the sample video images as output, and obtaining a first digital human model based on machine learning algorithm training.

[0077] The first frame sample video image refers to the first frame sample video image of the sample video image sequence. There are multiple sample video image sequences, which can be selected from the sample training set according to actual needs. For example, when the clarity of the sample video image sequence is required to be high, the sample training set may include multiple videos collected in the same application scenario (for example: a scene where a journalist reports news), including the first video, the second video, the third video, and the fourth video. The video images of the first video, the second video, and the third video are clear, and the fourth video is blurred. Then the server 110 may use the first video, the second video, and the third video as the sample video image sequence.

[0078] Among them, machine learning involves cross-disciplinary learning in probability theory, statistics, approximation theory, convex analysis, and algorithms, and specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge points or skills and reorganize existing knowledge structures. At present, machine learning algorithms can include linear regression algorithms, support vector machine algorithms, nearest neighbor / k-nearest neighbor algorithms, logistic regression algorithms, decision tree algorithms, k-means algorithms, random forest algorithms, naive Bayes algorithms, dimensionality reduction algorithms, and gradient enhancement algorithms.

[0079] In some embodiments, the server 110 can take the sample interaction information as input, take the first frame sample video image and the optical flow information corresponding to the multiple frames of sample video images as output, select at least one from the above machine learning algorithms, and train the first digital human model. After the first digital human model is trained, the server 110 only needs to input the interaction information input by the user into the first digital human model, and the first digital human model can output the optical flow information corresponding to the first frame sample image and the multiple frames of video images.

[0080] Step S240: When the performance state satisfies the first preset state, a first image processing strategy and a first digital human model corresponding to the first preset state are determined.

[0081] The first preset state may be set according to actual needs. In this embodiment, when the performance state of the terminal device 120 is very good, the server 110 may determine that the performance state of the terminal device 120 satisfies the first preset state.

[0082] The first image processing strategy may be: generating the optical flow information corresponding to the first frame of the video image to be processed and each frame of the video image in the target video image sequence to be processed according to the interactive information input by the user.

[0083] In some embodiments, when the performance state of the terminal device 120 is very good, the server 110 may determine that the performance state of the terminal device 120 satisfies the first preset state. When it is determined that the performance state of the terminal device 120 satisfies the first preset state, the server 110 may determine a first image processing strategy and a first digital human model corresponding to the first preset state.

[0084] Step S250: According to the first image processing strategy, the first digital human model is controlled to generate optical flow information corresponding to the first frame of video image to be processed and each frame of video image in the target video image sequence to be processed, wherein the target video image sequence includes the first frame of video image.

[0085] The target video image sequence may be a default reply video image sequence, or may be a video image sequence having the same content as the default reply video image sequence but with higher image quality than the default reply video image sequence.

[0086] In some embodiments, the server 110 can receive interaction information input by the user and input the interaction information input by the user into the first digital human model. The first digital human model can output the optical flow information corresponding to the first frame of video image to be processed and each frame of video image in the target video image sequence to be processed.

[0087] Step S260: sending at least one frame of video image and the optical flow information to the terminal device, so that the terminal device performs frame insertion processing on the at least one frame of video image according to the optical flow information.

[0088] For the detailed description of step S260, please refer to step S130, which will not be repeated here.

[0089] The image processing method provided by the embodiment of the present application obtains the performance state of the terminal device; obtains a sample training set, wherein the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images; takes the sample interaction information as input, takes the first frame of sample video image and the optical flow information corresponding to the multiple frames of sample video images as output, and obtains a first digital human model based on a machine learning algorithm training; when the performance state meets the first preset state, determines the first image processing strategy and the first digital human model corresponding to the first preset state; according to the first image processing strategy, controls the first digital human model to generate the optical flow information corresponding to the first frame of image to be processed and each frame of video image in the target video image sequence to be processed, wherein the target video image sequence includes the first frame of video image; sends at least one frame of video image and the optical flow information to the terminal device, so that the terminal device performs interpolation processing on at least one frame of video image according to the optical flow information. The method can train the first digital human model according to the sample training set, and generate the optical flow information corresponding to the first frame of video image to be processed and each frame of video image in the target video image sequence to be processed through the first digital human model. After the training of the first digital human model is completed, it is only necessary to input the interaction information input by the user into the first digital human model to obtain the corresponding first frame of video image and the optical flow information corresponding to each frame of video image in the target video image sequence to be processed, which greatly saves computing resources, thereby improving the computing speed and reducing the system burden. In addition, since it takes a long time to output video images, this method only generates and outputs the optical flow information corresponding to the first frame of video image to be processed and each frame of video image in the target video image sequence to be processed, which greatly saves computing time and output time.

[0090] See also Figure 4 , Figure 4 A schematic flow chart of an image processing method provided in another embodiment of the present application is shown. The image processing method is applied to a server. The image processing method may specifically include the following steps:

[0091] Step S310: Obtain the performance status of the terminal device.

[0092] The specific description of step S310 can be found in step S110 , which will not be repeated here.

[0093] Step S320: obtaining a sample training set, where the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images.

[0094] For the detailed description of step S320, please refer to step S220, which will not be repeated here.

[0095] S330: Taking the sample interaction information as input, taking multiple frames of sample video images and optical flow information as output, and obtaining a second digital human model based on machine learning algorithm training.

[0096] The multiple frames of sample video images may be multiple frames of sample video images in a sample video image sequence. The sample video image sequence may be multiple and may be selected from a sample training set.

[0097] Among them, machine learning involves cross-disciplinary learning in probability theory, statistics, approximation theory, convex analysis, and algorithms, and specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge points or skills and reorganize existing knowledge structures. At present, machine learning algorithms can include linear regression algorithms, support vector machine algorithms, nearest neighbor / k-nearest neighbor algorithms, logistic regression algorithms, decision tree algorithms, k-means algorithms, random forest algorithms, naive Bayes algorithms, dimensionality reduction algorithms, and gradient enhancement algorithms.

[0098] In some embodiments, the server 110 can take the sample interaction information as input, take the multiple frames of sample video images and the optical flow information corresponding to the multiple frames of sample video images as output, select at least one from the above machine learning algorithms, and train the second digital human model. After the second digital human model is trained, the server 110 only needs to input the interaction information input by the user into the first digital human model, and the second digital human model can output the optical flow information corresponding to the multiple frames of sample images and the multiple frames of video images.

[0099] Step S340: When the performance state satisfies the second preset state, a second image processing strategy and a second digital human model corresponding to the second preset state are determined.

[0100] In this embodiment, the second preset state can be set according to actual needs. In this embodiment, when the performance state of the terminal device 120 is good, the server 110 can determine that the performance state of the terminal device 120 meets the second preset state.

[0101] The second image processing strategy may be: generating multiple frames of video images to be processed and optical flow information respectively corresponding to the multiple frames of video images to be processed according to the interactive information input by the user.

[0102] In some implementations, when the performance state of the terminal device 120 is good, the server 110 may determine that the performance state of the terminal device 120 satisfies the second preset state. When it is determined that the performance state of the terminal device 120 satisfies the second preset state, the server 110 may determine a second image processing strategy and a second digital human model corresponding to the second preset state.

[0103] Step S350: According to the second image processing strategy, the second digital human model is controlled to generate multiple frames of video images to be processed and optical flow information respectively corresponding to the multiple frames of video images to be processed.

[0104] The target video image sequence may be a default reply video image sequence, or may be a video image sequence having the same content as the default reply video image sequence but with higher image quality than the default reply video image sequence.

[0105] In some embodiments, the server 110 may receive interaction information input by the user and input the interaction information input by the user into the second digital human model, and the second digital human model may output multiple frames of video images to be processed and optical flow information corresponding to the multiple frames of video images to be processed.

[0106] Step S360: sending at least one frame of video image and the optical flow information to the terminal device, so that the terminal device performs frame insertion processing on the at least one frame of video image according to the optical flow information.

[0107] For the detailed description of step S360, please refer to step S130, which will not be repeated here.

[0108] The image processing method provided by the embodiment of the present application is to obtain the performance status of the terminal device; obtain a sample training set, the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images; use the sample interaction information as input, and use the multiple frames of sample video images and optical flow information as output, and obtain a second digital human model based on machine learning algorithm training; when the performance status meets the second preset state, determine the second image processing strategy and the second digital human model corresponding to the second preset state; according to the second image processing strategy, control the second digital human model to generate multiple frames of video images to be processed and optical flow information corresponding to the multiple frames of video images to be processed; send at least one frame of video image and optical flow information to the terminal device, so that the terminal device performs interpolation processing on at least one frame of video image according to the optical flow information. The method can train the second digital human model according to the sample training set, and generate multiple frames of video images to be processed and optical flow information corresponding to the multiple frames of video images to be processed through the second digital human model. After the training of the second digital human model is completed, it is only necessary to input the interaction information input by the user into the second digital human model to obtain the corresponding multiple frames of video images and the optical flow information corresponding to the multiple frames of video images, which saves computing resources to a certain extent, thereby improving the computing speed and reducing the system burden. In addition, since it takes a long time to output video images, this method generates and outputs multiple frames of video images and optical flow information. Compared with generating and outputting all frames of video images, this method can save computing time and output time to a certain extent.

[0109] See also Figure 5 , Figure 5 A schematic flow chart of an image processing method provided by another embodiment of the present application is shown. The image processing method is applied to a server. The image processing method may specifically include the following steps:

[0110] Step S410: Acquire the performance status of the terminal device.

[0111] The specific description of step S410 can be found in step S110 , which will not be repeated here.

[0112] Step S420: Obtain a sample training set, where the sample training set includes sample interaction information and a default reply video image corresponding to the sample interaction information.

[0113] The sample training set may be a collection of multiple samples, which can be used to train a digital human model. The sample training set may include at least one content such as multiple videos, multiple images, and multiple voices. The embodiment of the present application does not limit the specific content of the sample training set. The sample training set can be collected in multiple scenarios. For example, in a scenario where bank staff handles business for customers, the conversation information between customers and bank staff can be collected; in a scenario where news broadcasters are recorded, the demeanor and movements of the news broadcasters can be collected. This application does not limit the collection method and specific content of the sample training set.

[0114] The sample interaction information may include at least one type of information such as the user's conversation information, the user's voice, and the user's demeanor and actions during the conversation. The embodiment of the present application does not limit the specific content of the sample interaction information.

[0115] The default reply video image corresponding to the sample interaction information can be screened from the sample video image sequence according to the interaction information input by the user.

[0116] In some embodiments, the server 110 can be connected to a camera (or a video recorder, etc.) and a computer. The camera (or video recorder) can record or shoot the sample training set, and store the collected sample training set in the computer through a video capture card. The video capture card is also called a video card, which is used to input the video signal output by the camera into the computer, and convert it into digital data that can be identified by the computer, and store it in the computer to become an editable and processable video data file. The server 110 can receive the sample training set sent by the computer.

[0117] Step S430: Taking the sample interaction information as input and the default reply video image as output, a default digital human model is obtained through training based on a machine learning algorithm.

[0118] The default reply video image may include each frame of video image in a sample video image sequence, wherein there are multiple sample video image sequences, which may be selected from a sample training set.

[0119] Among them, machine learning involves cross-disciplinary learning in probability theory, statistics, approximation theory, convex analysis, and algorithms, and specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge points or skills and reorganize existing knowledge structures. At present, machine learning algorithms can include linear regression algorithms, support vector machine algorithms, nearest neighbor / k-nearest neighbor algorithms, logistic regression algorithms, decision tree algorithms, k-means algorithms, random forest algorithms, naive Bayes algorithms, dimensionality reduction algorithms, and gradient enhancement algorithms.

[0120] In some implementations, the server 110 may use sample interaction information as input and a default reply video image as output, and select at least one of the above machine learning algorithms to train the default digital human model. After the default digital human model is trained, the server 110 only needs to input the interaction information input by the user into the default digital human model, and the default digital human model can output a default reply video image.

[0121] Step S440: When the performance state does not meet the target state, a default video image is generated through a default digital human model, wherein the default video image is a default reply video image determined by the server according to the interactive information input by the user, and the target state is used to represent the minimum performance state required for the terminal device to perform the interpolation processing operation.

[0122] The minimum performance state may be set according to actual needs. In this embodiment, when the performance state of the terminal device 120 is poor, it may be determined that the performance state of the terminal device 120 does not meet the minimum performance state.

[0123] In some implementations, when the performance status of the terminal device 120 is poor, the server 110 may determine that the performance status of the terminal device 120 does not meet the target status. When the performance status of the terminal device 120 does not meet the target status, the server 110 may input the interaction information input by the user into the default digital human model, and the default digital human model may output a default reply video image.

[0124] Step S450: Generate a default reply video based on the default video image, and send the default reply video to the terminal device.

[0125] Among them, the default reply video may include a digital human, wherein the digital human can be modeled through 3D modeling software, or synthesized by the server 110 calling the first program code (the first program code can be pre-written into the server 110 for generating the digital human), or generated by a modeling model pre-trained by the server 110, etc. The embodiment of the present application does not limit the specific method of obtaining the digital human.

[0126] In some implementations, the server 110 may combine the default video images to synthesize a default reply video, and send the default reply video to the terminal device.

[0127] In other embodiments, the server 110 may input the default reply video image into a pre-trained interpolation model of the server 110, and the interpolation model may output the default reply video after interpolation, wherein the default reply video after interpolation has better video quality than the default reply video before interpolation, that is, the quality of each frame of video image in the default reply video is better.

[0128] The image processing method provided by the embodiment of the present application obtains the performance state of the terminal device; obtains a sample training set, the sample training set includes sample interaction information and a default reply video image corresponding to the sample interaction information; uses the sample interaction information as input, uses the default reply video image as output, and obtains a default digital human model based on a machine learning algorithm training; when the performance state does not meet the target state, generates a default video image through the default digital human model, wherein the default video image is a default reply video image determined by the server according to the interaction information input by the user, and the target state is used to characterize the minimum performance state required when the terminal device performs the interpolation processing operation; generates a default reply video according to the default video image, and sends the default reply video to the client. The method can train the default digital human model according to the sample training set, and generate a default reply video image through the default digital human model. Since after the training of the default digital human model is completed, it only needs to input the interaction information input by the user into the default digital human model to obtain the corresponding default reply video image, which saves computing resources to a certain extent, thereby improving the computing speed and reducing the system burden.

[0129] It should be noted that in all the embodiments and implementations mentioned below, the interpolation algorithm may include but is not limited to at least one of a depth-aware video frame interpolation (DAIN) algorithm and a real-time video interpolation (RIFE) algorithm. The embodiment of the present application does not specifically limit the interpolation algorithm. The interpolation model can be pre-trained by the terminal device 120 or the server 110 and then stored in the terminal device 120.

[0130] See also Figure 6 , Figure 6 A flow chart of an image processing method provided by another embodiment of the present application is shown. The image processing method is applied to a terminal device. The image processing method may specifically include the following steps:

[0131] Step S510: receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server.

[0132] Among them, the at least one frame of video image can be one frame of video image or multiple frames of video image, etc. The present application does not impose any specific restrictions on the content of the at least one frame of video image.

[0133] The optical flow information to be processed may be the image change information between each two adjacent frames of the video image sequence, and the image change information may include at least one of the image brightness change information and the image chromaticity change information. For example, the first optical flow information corresponding to the first frame of the video image may be the image change information between the first frame of the video image and the second frame of the video image; the second optical flow information corresponding to the second frame of the video image may be the image change information between the second frame of the video image and the third frame of the video image; and so on, the optical flow information corresponding to the second-to-last frame of the video image (i.e., the last optical flow information) may be the image change information between the second-to-last frame of the video image and the first-to-last frame of the video image.

[0134] In some implementations, the terminal device 120 may receive at least one frame of video image and optical flow information sent by the server 110 .

[0135] Step S520: performing frame interpolation processing on at least one frame of video image according to at least one frame of video image and optical flow information.

[0136] In some embodiments, at least one frame of video image is a first frame of video image. The terminal device 120 may call an interpolation algorithm (or an interpolation model) based on the first frame of video image and the optical flow information corresponding to the first frame of video image to generate a second frame of video image; then, the interpolation algorithm (or an interpolation model) may be called based on the generated second frame of video image and the optical flow information corresponding to the second frame of video image to generate a third frame of video image; and so on, the last frame of video image may be generated by calling a frame drop algorithm based on the generated second-to-last frame of video image and the second-to-last frame of video image.

[0137] In other embodiments, when at least one frame of video image is a multi-frame video image. The terminal device 120 can interpolate between every two adjacent frames of video images in the multi-frame video image to obtain a target video image sequence. Specifically, for example, two adjacent frames of video images are a first video image and a second video image. The terminal device 120 can call an interpolation algorithm (or an interpolation model) based on the first video image and the optical flow information corresponding to the first video image to generate a first transition frame video image; the terminal device 120 can call an interpolation algorithm (or an interpolation model) based on the second video image and the optical flow information corresponding to the second video image to generate a second transition frame video image; then the terminal device 120 can compare the image quality of the first transition frame video image and the second transition frame video image, and insert a transition frame video image with higher image quality between the first video image and the second video image. The terminal device 120 may compare the image quality based on at least one parameter such as Peak-Signal to Noise Ratio (PSNR), Mean Square Error (MSE), MAE (Mean Absolute Error, MSE) and Signal to Noise Ratio (SNR). Specifically, these parameters may measure the quality of the image to be evaluated from a statistical perspective by calculating the difference between the grayscale values ​​of corresponding pixels of the image to be evaluated and the reference image.

[0138] In some other embodiments, at least one frame of video image is a multi-frame video image. The terminal device 120 can call an interpolation algorithm (or an interpolation model) according to each frame of video image in the multi-frame video image and the optical flow information corresponding to each frame of video image to generate a transition frame video image corresponding to each frame of video image, and insert the transition frame video image after each frame of video image to obtain a target video image sequence.

[0139] Step S530: Obtain the target video image sequence after the frame insertion process.

[0140] In some implementations, the terminal device 120 may acquire the processed target video image sequence, and transmit the target video image sequence to a display of the terminal device 120 , so that the display displays the target video image sequence.

[0141] In other implementations, the terminal device 120 may also acquire the processed target video image sequence, and transmit the target video image sequence to a display of another device connected to the terminal device 120, so that the display displays the target video image sequence.

[0142] The image processing method provided in the embodiment of the present application is by receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server; performing interpolation processing on at least one frame of video image according to the at least one frame of video image and the optical flow information; and obtaining a target video image sequence after interpolation processing. The method interpolates video images according to the optical flow information to obtain a target video image sequence. Since the optical flow information contains image change information between the current frame of video image and the next frame of video image, more specifically, the optical flow information may include at least one of image brightness change information and image chromaticity change information. Therefore, interpolating video images according to the optical flow information can improve the image quality in the obtained target video image sequence. In addition, by interpolating video images through a terminal device to obtain a target video image sequence, the computing resources of the terminal device can be fully utilized.

[0143] See also Figure 7 , Figure 7 A flow chart of an image processing method provided by yet another embodiment of the present application is shown. The image processing method is applied to a terminal device. The at least one frame of video image to be processed is a first frame of video image, and the optical flow information to be processed is the optical flow information corresponding to each frame of video image in the target video image sequence to be processed. The image processing method may specifically include the following steps:

[0144] Step S610: receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server.

[0145] For the detailed description of step S610, please refer to step S510, which will not be repeated here.

[0146] Step S620: Generate an intermediate frame video image according to the first frame video image and the optical flow information.

[0147] In this embodiment, the at least one frame of video image to be processed is the first frame of video image, and the optical flow information to be processed is the optical flow information corresponding to each frame of video image in the target video image sequence to be processed. The optical flow information corresponding to the first frame of video image can be image change information between the first frame of video image and the second frame of video image, and the image change information can include at least one of image brightness change information and grayscale change information.

[0148] The intermediate frame video image may include at least one frame video image, and the intermediate frame video image may be multiple frame video images.

[0149] In some implementations, the intermediate frame video image is a frame video image. The terminal device 120 may call an interpolation algorithm (or an interpolation model) to generate an intermediate frame video image based on the first frame video image and the optical flow information corresponding to the first frame video image.

[0150] In other embodiments, the intermediate frame video image is a multi-frame video image. The terminal device 120 can call an interpolation algorithm (or an interpolation model) based on the first frame video image and the optical flow information corresponding to the first frame video image to generate a second frame video image. The terminal device 120 can call an interpolation algorithm (or an interpolation model) based on the generated second frame video image and the optical flow information corresponding to the second frame video image to generate a third frame video image; and so on, the terminal device 120 can call an interpolation algorithm (or an interpolation model) based on the generated third-to-last frame video image and the optical flow information corresponding to the third-to-last frame video image to generate the second-to-last frame video image. Among them, the second frame video image, the third frame video image...the third-to-last frame video image and the second-to-last frame video image are all intermediate frame video images.

[0151] Step S630: generating a final frame video image according to the intermediate frame video image and the optical flow information corresponding to the intermediate frame video image.

[0152] Among them, the optical flow information corresponding to the intermediate frame video image can be the image change information between the intermediate frame video image moving to the next frame video image of the intermediate frame, and the image change information may include at least one of the image brightness change information and the image chromaticity change information.

[0153] The final frame video image may be the last frame video image.

[0154] In some embodiments, following the description of step S620, when the intermediate frame video image is a frame video image, the terminal device 120 may call an interpolation algorithm (or an interpolation model) based on the intermediate frame video image and the optical flow information corresponding to the intermediate frame video image to generate a final frame video image. The terminal device 120 may arrange the first frame video image, the intermediate frame video image, and the final frame video image in a video playback order to obtain a target video image sequence.

[0155] In other embodiments, following the above description of step S620, when the intermediate video frame image is the second video frame image, the third video frame image, ... the third-to-last video frame image, and the second-to-last video frame image, the terminal device 120 can call the interpolation algorithm (or interpolation model) based on the optical flow information of the second-to-last video frame image and the second-to-last video frame image to generate the final video frame image. The terminal device 120 can combine and arrange the first video frame image, the intermediate video frame image, and the final video frame image in the video playback order to obtain a target video image sequence.

[0156] Step S640: Obtain the target video image sequence after the frame insertion process.

[0157] For the detailed description of step S640, please refer to step S530, which will not be repeated here.

[0158] The image processing method provided by the embodiment of the present application is by receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server; generating an intermediate frame video image according to the first frame video image and the optical flow information corresponding to the first frame video image; generating a final frame video image according to the intermediate frame video image and the optical flow information corresponding to the intermediate frame video image; and obtaining a target video image sequence after interpolation processing. The method can generate an intermediate frame video image according to the first frame video image and the optical flow information, and can generate a final frame video image according to the intermediate frame image and the optical flow information, thereby obtaining a target video image sequence. Interpolating video images according to the optical flow information can improve the image quality in the obtained target video image sequence. In addition, by interpolating video images through a terminal device to obtain a target video image sequence, the computing resources of the terminal device can be fully utilized and the computing resources of the server can be saved.

[0159] See also Figure 8 , Figure 8 A flow chart of an image processing method provided by yet another embodiment of the present application is shown. The image processing method is applied to a terminal device. Among them, at least one frame of image to be processed is a multi-frame video image, and the multi-frame video image includes a plurality of groups of adjacent video frame images, and each group of adjacent video frame images in the plurality of groups of adjacent video frame images includes a forward frame video image and a backward frame video image. The optical flow information to be processed is the optical flow information corresponding to the multi-frame video image. The image processing method may specifically include the following steps:

[0160] Step S710: receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server.

[0161] For the detailed description of step S710, please refer to step S510, which will not be repeated here.

[0162] Step S720: determining a first transition frame video image according to the forward frame video image and the optical flow information corresponding to the forward frame video image.

[0163] In this embodiment, at least one frame of image to be processed is a multi-frame video image, and the multi-frame video image includes a plurality of groups of adjacent video frame images, and each group of adjacent video frame images in the plurality of groups of adjacent video frame images includes a forward frame video image and a backward frame video image. The optical flow information to be processed is the optical flow information corresponding to the multi-frame video image. Among them, the forward frame video image can be the frame video image played first in the two adjacent frames of video images. The first transition frame video image can be used as an alternative frame for inserting between the two adjacent frames of video images.

[0164] In some embodiments, the terminal device 120 may call an interpolation algorithm (or an interpolation model) to generate a first transition frame video image based on the forward frame video image and the optical flow information corresponding to the forward frame video image.

[0165] Step S730: determining a second transition frame video image according to the backward frame video image and the optical flow information corresponding to the backward frame video image.

[0166] The backward frame video image may be the frame video image played later of two adjacent frames of video images. The second transition frame video image may be used as an alternative frame to be inserted between two adjacent frames of video images.

[0167] In some implementations, the terminal device 120 may call a frame interpolation algorithm (or a frame interpolation model) to generate a second transition frame video image based on the backward frame video image and the optical flow information corresponding to the backward frame video image.

[0168] Step S740: generating a target transition frame video image according to the first transition frame video image and the second transition frame video image.

[0169] The target transition frame video image may be used to insert a video image between two adjacent frames of video images.

[0170] In some implementations, the terminal device 120 may call a frame interpolation algorithm (or a frame interpolation model) based on the first transition frame video image and the second transition frame video image to generate a target frame video image.

[0171] As an example, the terminal device 120 can call the interpolation model to generate a target frame video image based on the first transition frame video image and the second transition frame video image. Specifically, the interpolation model can use the video image with higher image quality between the first transition frame video image and the second transition frame video image as the target frame video image. Among them, the comparison of image quality can be based on at least one parameter such as PSNR, MSE, MAE, SNR, etc. Specifically, the interpolation model can measure the quality of the image to be evaluated from a statistical perspective by calculating the difference between the grayscale values ​​of the corresponding pixels of the image to be evaluated and the reference image. Step S750: Insert the target transition frame video image between the forward frame video image and the backward frame video image to generate a target video image sequence.

[0172] It should be noted that the forward frame video images and the backward frame video images included in each group of the multiple groups of adjacent video frame images can generate corresponding target frame video images according to the interpolation method of S720 to S740.

[0173] In some embodiments, the terminal device 120 may insert the generated multiple target frame video images between the corresponding forward frame video images and the backward frame video images to generate a target video image video sequence. More specifically, the terminal device 120 may insert the first target frame video image between the first forward frame video image and the first backward frame video image; the terminal device 120 may insert the second target frame video image between the second forward frame video image and the second backward frame video image; and so on, the terminal device 120 may insert the last target frame video image between the last forward frame video image and the last backward frame video image, so as to obtain a target video image sequence.

[0174] Step S760: Obtain the target video image sequence after the frame insertion process.

[0175] For the detailed description of step S760, please refer to step S530, which will not be repeated here.

[0176] The image processing method provided by the embodiment of the present application is as follows: receiving at least one frame of video image to be processed and optical flow information to be processed sent by a server; determining a first transition frame video image according to the forward frame video image and the optical flow information corresponding to the forward frame video image; determining a second transition frame video image according to the backward frame video image and the optical flow information corresponding to the backward frame video image; generating a target transition frame video image according to the first transition frame video image and the second transition frame video image; inserting the target transition frame video image between the forward frame video image and the backward frame video image to generate a target video image sequence; and obtaining a target video image sequence after interpolation processing. The method can generate corresponding two transition frame video images according to any two adjacent frames of video images and the optical flow information corresponding to the two frames of video images. Furthermore, the method can select a transition frame video image with better image quality from the generated two transition frame video images, and use the transition frame video image with better image quality as the target transition frame video image, so that the method can improve the image quality of the transition frame video image, and further improve the image quality of the target video image sequence.

[0177] See also Fig. 9 , Fig. 9 A flow chart of an image processing method provided in another embodiment of the present application is shown. The image processing method is applied to a terminal device. The at least one frame of video image to be processed is a default reply video generated by the server according to the interactive information input by the user. The image processing method may specifically include the following steps:

[0178] Step S810: receiving the default reply video sent by the server, and using the default reply video as the target video image sequence.

[0179] In some implementations, the terminal device 120 may receive the default reply video sent by the server 110 and use the default reply video as the target video image sequence.

[0180] Step S820: Output the target video image sequence.

[0181] In some implementations, the terminal device 120 may transmit the target video image sequence to a display of the terminal device 120 , so that the display displays the target video image sequence.

[0182] In other implementations, the terminal device 120 may also transmit the target video image sequence to a display of another device connected to the terminal device 120 , so that the display displays the target video image sequence.

[0183] In the image processing method provided in this embodiment, the terminal device only needs to receive and output the default reply video sent by the server, which is convenient and fast, does not add burden to the system of the terminal device, and can also save the time of the terminal device outputting the video.

[0184] See also Fig.10 , Fig.10 A flow chart of an image processing method provided by yet another embodiment of the present application is shown. The image processing method is applied to an image processing system. The image processing system includes a server and a terminal device, and the server is electrically connected to the terminal device. The image processing method may specifically include the following steps:

[0185] Step S910: The server obtains the performance status of the terminal device.

[0186] Step S920: The server generates information to be processed through the digital human model according to the performance status, and the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed.

[0187] Step S930: the server performs frame insertion processing on at least one frame of video image and the optical flow information.

[0188] For the detailed description of step S910 to step S930 , please refer to step S110 to step S130 , which will not be repeated here.

[0189] Step S940: The terminal device performs frame interpolation processing on at least one frame of video image according to at least one frame of video image and optical flow information.

[0190] Step S950: The terminal device obtains the target video image sequence after the frame insertion processing.

[0191] For the detailed description of step S940 to step S950, please refer to step S520 to step S530, which will not be repeated here.

[0192] The image processing method provided in the embodiment of the present application obtains the performance status of the terminal device through the server; the server generates information to be processed through the digital human model according to the performance status, and the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; the server performs interpolation processing on at least one frame of video image with at least one frame of video image and optical flow information; the terminal device performs interpolation processing on at least one frame of video image according to at least one frame of video image and optical flow information; the terminal device obtains the target video image sequence after interpolation processing. Therefore, the server can generate different numbers of video images and optical flow information and send them to the terminal device through different digital human models according to different performance states of the terminal device, and the terminal device obtains the corresponding target video image sequence according to the video image and optical flow information, so as to save the computing resources of the server, thereby reducing the burden on the server system.

[0193] As an example, see Fig.11 , Fig.11A schematic diagram of a process for executing an image processing method provided by an exemplary embodiment of the present application is shown. Among them, the default reply video image sequence can be a total of nine frames. The server 110 inputs the interactive information input by the user into the digital human model, and then obtains the first frame of video image (frame 1, corresponding to the first frame of video image in the default reply video image sequence), the second frame of video image (frame 3, corresponding to the third frame of video image in the default reply video image sequence), the third frame of video image (frame 5, corresponding to the fifth frame of video image in the default reply video image sequence), the fourth frame of video image (frame 7, corresponding to the seventh frame of video image in the default reply video image sequence), the fifth frame of video image (frame 9, corresponding to the ninth frame of video image in the default reply video image sequence) and the optical flow information between every two frames of images in the default reply video image sequence, which are optical flow 1 (optical flow information between the first frame of video image and the second frame of video image in the default reply video image sequence), optical flow 2 (optical flow information between the first frame of video image and the second frame of video image in the default reply video image sequence), and the optical flow information between every two frames of images in the default reply video image sequence. The server 110 then transmits the above 8 optical flows and 5 video images to the terminal device 120. The terminal device 120 can accept the 8 optical flows and 5 video images sent by the server 110, and then call the interpolation model to interpolate the 5 video images according to the 8 optical flow information.Among them, frame 1 is combined with optical flow 1 to generate frame 2 (the second frame of video image corresponding to the default reply video image); frame 3 is combined with optical flow 3 to generate a first transition frame 4, and frame 5 is combined with optical flow 3 to generate a second transition frame 4, and then the interpolation model can select a transition frame with higher quality (for example, the quality of the image can be judged by resolution, and the higher the resolution, the higher the image quality) from the first transition frame 4 and the second transition frame 4 as the final frame 4 (corresponding to the fourth frame of video image in the default reply video image sequence); frame 5 is combined with optical flow 5 to generate a first transition frame 6, and frame 7 is combined with optical flow 5 to generate a second transition frame 6, and then the interpolation model can select a transition frame with higher image quality from the first transition frame 6 and the second transition frame 6 as the final frame 6 (corresponding to the sixth frame of the default reply video image sequence); frame 7 and optical flow 7 can generate a first transition frame 8, and frame 9 and optical flow 7 can also generate a second transition frame 8, and then the interpolation model can select a transition frame with better quality from the first transition frame 8 and the second transition frame 8 as the final frame 8 (corresponding to the eighth frame of the default reply video image sequence). Finally, the terminal device 120 can insert frame 2 between frame 1 and frame 3; insert frame 4 between frame 3 and frame 5; insert frame 6 between frame 5 and frame 7; insert frame 8 between frame 7 and frame 9, thereby synthesizing the final video, and transmit the video to the display of the terminal device 120, or transmit the video to the display of other devices connected to the terminal device 120, so that the user can watch the video.

[0194] See also Fig.12 , Fig.12 The structural block diagram of an image processing device provided by an embodiment of the present application is shown. The image processing device 1000 is applied to a server. The image processing device 1000 may include an acquisition module 1010, a generation module 1020 and a sending module 1030. Among them:

[0195] The acquisition module 1010 is used to acquire the performance status of the terminal device.

[0196] The generating module 1020 is used to generate information to be processed through the digital human model according to the performance status, and the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed.

[0197] The sending module 1030 is used to send at least one frame of video image and optical flow information to the terminal device, so that the terminal device performs frame insertion processing on the at least one frame of video image according to the optical flow information.

[0198] Optionally, the generating module 1020 may include a first determining submodule and a first generating submodule, wherein:

[0199] The first determination submodule is used to determine a first image processing strategy and a first digital human model corresponding to the first preset state when the performance state satisfies the first preset state.

[0200] The first determination submodule is used to control the first digital human model to generate the first frame of video image to be processed and the optical flow information corresponding to each frame of video image in the target video image sequence to be processed according to the first image processing strategy, and the target video image sequence includes the first frame of video image.

[0201] Furthermore, the image processing apparatus 1000 may further include a first sample acquisition module and a first model training module, wherein:

[0202] The first sample acquisition module is used to acquire a sample training set, where the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images.

[0203] The first model training module is used to take the sample interaction information as input, take the optical flow information corresponding to the first frame of the sample video image and multiple frames of the sample video images as output, and obtain the first digital human model based on machine learning algorithm training.

[0204] Optionally, the generating module 1020 may further include a second determining submodule and a second generating submodule, wherein:

[0205] The second determination submodule is used to determine a second image processing strategy and a second digital human model corresponding to the second preset state when the performance state satisfies the second preset state.

[0206] The second generating submodule is used to control the second digital human model to generate multiple frames of video images to be processed and optical flow information respectively corresponding to the multiple frames of video images to be processed according to the second image processing strategy.

[0207] Furthermore, the image processing device 1000 may also include a second sample acquisition module and a second model training module, wherein:

[0208] The second sample acquisition module is used to acquire a sample training set, where the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images.

[0209] The second model training module is used to take the sample interaction information as input and multiple frames of sample video images and optical flow information as output, and obtain the second digital human model based on machine learning algorithm training.

[0210] Further, the generation module 1020 may also include a third determination submodule and a third generation submodule, wherein:

[0211] The third determination submodule is used to generate a default video image through a default digital human model when the performance state does not meet the target state. The default video image is a default reply video image determined by the server based on the interactive information input by the user. The target state is used to represent the minimum performance state required for the terminal device to perform the interpolation processing operation.

[0212] The third generating submodule is used to generate a default reply video according to the default video image, and send the default reply video to the client.

[0213] See also Fig.13 , Fig.13 The structural block diagram of an image processing device provided by another embodiment of the present application is shown. The image processing device 1100 is applied to a terminal device. The image processing device 1100 may include a receiving module 1110, a processing module 1120 and a synthesis module 1130. Among them:

[0214] The receiving module 1110 is used to receive at least one frame of video image to be processed and optical flow information to be processed sent by the server;

[0215] A processing module 1120, configured to perform frame interpolation processing on at least one frame of video image according to at least one frame of video image and optical flow information;

[0216] The synthesis module 1130 is used to obtain a target video image frame sequence after the frame insertion process.

[0217] Optionally, the at least one frame of video image to be processed is the first frame of video image, and the optical flow information to be processed is the optical flow information corresponding to each frame of video image in the target video image sequence to be processed. The processing module 1120 may include a first generation submodule, a second generation submodule and a synthesis submodule, wherein:

[0218] A first generating submodule is used to generate an intermediate frame video image according to a first frame video image and optical flow information corresponding to the first frame video image;

[0219] The second generation submodule is used to generate a final frame video image according to the intermediate frame video image and the optical flow information corresponding to the intermediate frame video image;

[0220] The synthesis submodule is used to synthesize a target video image sequence according to a first frame video image, an intermediate frame video image and a final frame video image.

[0221] Optionally, at least one frame of video image to be processed is a multi-frame video image, the multi-frame video image includes a plurality of groups of adjacent video frame images, each group of adjacent video frame images in the plurality of groups of adjacent video frame images includes a forward frame video image and a backward frame video image, and the optical flow information to be processed is the optical flow information corresponding to the multi-frame video image. The processing module 1120 may also include a first transition frame determination submodule, a second transition frame determination submodule, a target transition frame generation submodule, and a sequence generation submodule, wherein:

[0222] A first transition frame determination submodule, used to determine a first transition frame video image according to the forward frame video image and the optical flow information corresponding to the forward frame video image;

[0223] A second transition frame determination submodule, used to determine a second transition frame video image according to the backward frame video image and the optical flow information corresponding to the backward frame video image;

[0224] A target transition frame generation submodule, used to generate a target transition frame video image according to the first transition frame video image and the second transition frame video image;

[0225] The sequence generation submodule is used to insert the target transition frame video image between the forward frame video image and the backward frame video image to generate a target video image sequence.

[0226] Optionally, the at least one frame of video image to be processed is a default reply video generated by the server according to the interactive information input by the user, and the image processing device 1100 may further include a determination module and an output module.

[0227] A determination module is used to use a default reply video as a target video image sequence.

[0228] The output module is used to output the target video image sequence.

[0229] For all the above devices provided in the embodiments of the present application, technical personnel in the relevant field can clearly understand that each of the devices provided in the embodiments of the present application can implement the corresponding processes in the aforementioned method embodiments. The specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be elaborated here.

[0230] In the embodiments provided in the present application, the coupling, direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication coupling through some interfaces, devices or modules, and may be electrical, mechanical or other forms, and the embodiments of the present application do not impose specific limitations on this.

[0231] In addition, each functional module in the embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0232] The embodiment of the present application further provides an image processing system, which includes a server and a terminal device, wherein the server is electrically connected to the terminal device, wherein:

[0233] Server, used to obtain the performance status of the terminal device;

[0234] The server is further used to generate information to be processed through the digital human model according to the performance status, wherein the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed;

[0235] The server is further used to send at least one frame of video image and optical flow information to the terminal device;

[0236] A terminal device, configured to perform frame interpolation processing on at least one frame of video image according to at least one frame of video image and optical flow information;

[0237] The terminal device is also used to obtain the target video image sequence after the frame insertion process.

[0238] See also Fig.14 , Fig.14 The structural block diagram of an electronic device provided by another embodiment of the present application is shown. The electronic device 1200 may be an electronic device such as a terminal device and a server. The electronic device 1200 in the present application may include one or more of the following components: a processor 1210, a memory 1220, and one or more application programs, wherein the one or more application programs may be stored in the memory 1220 and configured to be executed by one or more processors 1210, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0239] The processor may include one or more processing cores. The processor 1210 uses various interfaces and lines to connect the various parts of the entire electronic device 1200, and executes various functions and processes data of the electronic device 1200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1220, and calling to run or execute data stored in the memory 1220. Optionally, the processor 1210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1210 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to handle wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1210, but may be implemented separately through a communication chip.

[0240] The memory 1220 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may store data created by the electronic device 1200 during use, etc.

[0241] See also Fig.15 , Fig.15 The structure block diagram of a computer-readable storage medium provided in another embodiment of the present application is shown. The computer-readable storage medium 1300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0242] The computer readable storage medium 1300 can be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a hard disk, or a ROM. Optionally, the computer readable storage medium 1300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1300 has storage space for program code 1310 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1310 can be compressed, for example, in an appropriate form.

[0243] In summary, the embodiments of the present application provide an image processing method, device, electronic device and storage medium. The method is applied to a server, and the method obtains the performance status of a terminal device; based on the performance status, generates information to be processed through a digital human model, and the information to be processed includes at least one frame of video image to be processed and optical flow information to be processed; and sends at least one frame of video image and optical flow information to the terminal device, so that the terminal device performs interpolation processing on at least one frame of video image according to the optical flow information. Since the data volume of the optical flow information is small, the amount of calculation required to generate the optical flow information is small, and the amount of calculation required to transmit the optical flow information is also small, so the method can save the computing resources of the server, improve the computing speed of the server, and reduce the system burden.

[0244] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: Applicable to servers, including: During the real-time interaction between the digital human and the user, the performance status of the terminal device is obtained; Determine a corresponding frame number according to the performance state; According to the interactive information input by the user, the video images of the frame number and the optical flow information to be processed are generated by the digital human model; The video images of the number of frames and the optical flow information to be processed are sent to the terminal device, so that the terminal device performs frame insertion processing on the video images of the number of frames according to the optical flow information to be processed to obtain a reply video about the interactive information.

2. The method according to claim 1, characterized in that The step of determining the corresponding number of frames according to the performance state; and generating the video images of the number of frames and the optical flow information to be processed through the digital human model according to the interaction information input by the user, includes: When the performance state satisfies a first preset state, determining a first image processing strategy and a first digital human model corresponding to the first preset state; Determine a target video image sequence to be processed according to the interactive information input by the user; According to the first image processing strategy, the first digital human model is controlled to generate a first frame of video image to be processed and optical flow information corresponding to each frame of video image in a target video image sequence to be processed, wherein the target video image sequence includes the first frame of video image.

3. The method according to claim 2, characterized in that When the performance state satisfies a first preset state, before determining a first image processing strategy and a first digital human model corresponding to the first preset state, the method further includes: Acquire a sample training set, wherein the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images respectively; The sample interaction information is used as input, and the optical flow information corresponding to the first frame of the sample video image and the multiple frames of the sample video images are used as output, and the first digital human model is obtained based on machine learning algorithm training.

4. The method according to claim 1, characterized in that The step of determining the corresponding number of frames according to the performance state; and generating the video images of the number of frames and the optical flow information to be processed through the digital human model according to the interaction information input by the user, includes: When the performance state satisfies a second preset state, determining a second image processing strategy and a second digital human model corresponding to the second preset state; According to the second image processing strategy and the interactive information input by the user, the second digital human model is controlled to generate multiple frames of video images to be processed and the optical flow information respectively corresponding to the multiple frames of video images to be processed.

5. The method according to claim 4, characterized in that Before determining a second image processing strategy and a second digital human model corresponding to a second preset state when the performance state satisfies the second preset state, the method further includes: Acquire a sample training set, wherein the sample training set includes sample interaction information, multiple frames of sample video images corresponding to the sample interaction information, and optical flow information corresponding to the multiple frames of sample video images respectively; The sample interaction information is used as input, the multiple frames of sample video images and the optical flow information are used as output, and the second digital human model is obtained through training based on a machine learning algorithm.

6. The method according to claim 1, characterized in that After obtaining the performance status of the terminal device, the method further includes: When the performance state does not meet the target state, a default video image is generated by a default digital human model, wherein the default video image is a default reply video image determined by the server according to the interactive information input by the user, and the target state is used to represent the minimum performance state required for the terminal device to perform the frame insertion processing operation; A default reply video is generated according to the default video image, and the default reply video is sent to the terminal device.

7. An image processing method, characterized in that: Applied to terminal equipment, including: Sending the interactive information input by the user to the server; Receive a video image with a frame number corresponding to the interaction information and its own performance status and optical flow information to be processed sent by the server; According to the video images of the number of frames corresponding to the own performance state and the optical flow information to be processed, interpolating the video images of the number of frames corresponding to the own performance state; Obtain a reply video regarding the interactive information after frame insertion processing.

8. The method according to claim 7, characterized in that The video image with the number of frames corresponding to the own performance state is the first frame of the video image in the target video image sequence, the optical flow information to be processed is the optical flow information corresponding to each frame of the video image in the target video image sequence, and the interpolation processing of the video image with the number of frames corresponding to the own performance state according to the video image with the number of frames corresponding to the own performance state and the optical flow information to be processed includes: Generate an intermediate frame video image according to the first frame video image and the optical flow information; Generate a final frame video image according to the intermediate frame video image and the optical flow information corresponding to the intermediate frame video image; A target video image sequence is synthesized according to the first frame video image, the intermediate frame video image and the final frame video image.

9. The method according to claim 7, characterized in that: The video images of the number of frames corresponding to the own performance state are multiple video images in a target video image sequence, the multiple video images include multiple groups of adjacent video frames, each group of adjacent video images in the multiple groups of adjacent video frames includes a forward frame video image and a backward frame video image, the optical flow information to be processed is the optical flow information corresponding to the multiple frames of video images, and the interpolation processing of the video images of the number of frames corresponding to the own performance state according to the video images of the number of frames corresponding to the own performance state and the optical flow information to be processed includes: Determine a first transition frame video image according to the forward frame video image and the optical flow information corresponding to the forward frame video image; Determine a second transition frame video image according to the backward frame video image and the optical flow information corresponding to the backward frame video image; Generate a target transition frame video image according to the first transition frame video image and the second transition frame video image; The target transition frame video image is inserted between the forward frame video image and the backward frame video image to generate a target video image sequence.

10. The method according to claim 7, characterized in that The method further comprises: Obtaining a default reply video generated by the server according to the interactive information input by the user; Using the default reply video as a target video image sequence; Output the target video image sequence.

11. An image processing method, characterized in that: Applied to an image processing system, the image processing system comprises a server and a terminal device, the server is electrically connected to the terminal device, and the method comprises: The terminal device sends the interaction information input by the user to the server; The server obtains the performance status of the terminal device; The server determines the corresponding number of frames according to the performance status; The server generates the video images of the frame number and the optical flow information to be processed through the digital human model based on the interactive information input by the user; The server sends the video images of the number of frames and the optical flow information to be processed to the terminal device; The terminal device performs frame insertion processing on the video image with the number of frames corresponding to the performance state of the terminal device according to the video image with the number of frames corresponding to the performance state and the optical flow information to be processed; The terminal device obtains a reply video regarding the interactive information after frame insertion processing.

12. An image processing device, characterized in that: Applicable to servers, including: The acquisition module is used to obtain the performance status of the terminal device during the real-time interaction between the digital human and the user; A generation module is used to determine the corresponding number of frames according to the performance state; and to generate video images of the number of frames and optical flow information to be processed through a digital human model according to the interaction information input by the user; A sending module is used to send the video images of the frame number and the optical flow information to be processed to the terminal device, so that the terminal device performs frame interpolation processing on the video images of the frame number according to the optical flow information to be processed to obtain a reply video about the interactive information.

13. An image processing device, characterized in that: Applied to terminal equipment, including: A receiving module, configured to send the interaction information input by the user to the server; and receive the video image with the number of frames corresponding to the interaction information and its own performance status and the optical flow information to be processed sent by the server; A processing module, configured to perform frame insertion processing on the video images of the number of frames corresponding to the own performance state according to the video images of the number of frames corresponding to the own performance state and the optical flow information to be processed; A synthesis module is used to obtain a reply video about the interactive information after frame insertion processing.

14. An electronic device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 11.

Images