Image processing method and image processing device
By determining the size of the clipping frame of each frame image in the video based on the size of the predetermined image, the problem of poor image replacement effect in the prior art is solved, and the replacement effect with closer image size is achieved, and the authenticity of image replacement in the video is improved. and visual.
Patent Information
- Application Number
- CN202110558580.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-05-21
AI Technical Summary
When the existing image processing method replaces the original image in the video with other images, the replacement effect is poor, mainly because the change in the image size in the video is not very close.
By obtaining the image size of the predetermined image, the size of the clipping frame of each frame in the video is determined, and the original image is captured based on the clipping frame is generated to generate a target image sequence, so that the captured original image is closer to the predetermined image size, thereby improving the image replacement effect .
It effectively improves the image replacement effect, making the image size after replacement closer, and enhances the authenticity and visual sense of image replacement in the video.
Smart Images

Figure CN113222810B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to an image processing method and an image processing device. Background Art
[0002] With the increasing popularity of the Internet and computer technology, more and more users choose to watch online videos through the Internet. In order to increase the fun of the video, the online video playback platform can replace the original image in the video with other images (for example, cartoon images). However, the existing image processing methods do not have a good replacement effect when replacing the original image in the video with other images. Summary of the invention
[0003] In view of this, the purpose of an embodiment of the present invention is to provide an image processing method and an image processing device, which are used to determine the capture frame size of each frame image in a video according to the image size of a predetermined image, so as to capture the original image in each frame image, so that the captured original image is closer to the size of the predetermined image, thereby improving the image replacement effect.
[0004] According to a first aspect of an embodiment of the present invention, there is provided an image processing method, the method comprising:
[0005] Acquire a first image sequence, wherein the first image sequence includes a plurality of first images, and each of the first images includes an original image;
[0006] acquiring a second image, wherein the second image includes a predetermined image;
[0007] Determining the size of the capture frame corresponding to each of the first images according to the size of the second image, and capturing the corresponding first image based on each of the capture frames to obtain a third image sequence, wherein the third image sequence includes a plurality of third images, and each of the third images includes the original image;
[0008] According to the second image and each of the third images, a target image sequence is obtained based on a predetermined image processing model, each target image in the target image sequence includes a target image, and the target image is the predetermined image having the posture of the original image.
[0009] According to a second aspect of an embodiment of the present invention, there is provided an image processing apparatus, the apparatus comprising:
[0010] A first sequence acquisition unit, used for acquiring a first image sequence, wherein the first image sequence includes a plurality of first images, and each of the first images includes an original image;
[0011] A second sequence acquisition unit, used for acquiring a second image sequence, wherein the second image sequence includes at least one second image, and each of the second images includes a predetermined image;
[0012] a third sequence acquisition unit, configured to determine the size of a clipping frame corresponding to each of the second images according to the size of the first image, and clip the corresponding first image based on each of the clipping frames to acquire a third image sequence;
[0013] The fourth sequence acquisition unit is used to obtain a target image sequence based on each of the second images and each of the third images in the third image sequence based on a predetermined image processing model, wherein each of the target images in the target image sequence includes a target image, and the target image is the original image having the posture of the predetermined image.
[0014] According to a third aspect of an embodiment of the present invention, there is provided a computer-readable storage medium storing computer program instructions, wherein the computer program instructions implement the method as described in the first aspect when executed by a processor.
[0015] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, and wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.
[0016] The embodiment of the present invention obtains a first image sequence including an original image and a second image including a predetermined image, and determines the size of the cropped frame corresponding to each first image in the first image sequence according to the size of the second image, and then crops the corresponding first image based on each cropped frame to obtain a third image, thereby obtaining a target image sequence including a predetermined image with the posture of the original image based on the second image and each third image and a predetermined image processing model. The embodiment of the present invention determines the cropped frame size of each frame image in the video according to the image size of the predetermined image, so as to crop the original image in each frame image, so that the cropped original image is closer in size to the predetermined image, thereby effectively improving the image replacement effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
[0018] Figure 1 is a schematic diagram of an interface of an online teaching activity according to an embodiment of the present invention;
[0019] Figure 2 is a flowchart of an image processing method according to a first embodiment of the present invention;
[0020] Figure 3 is a flow chart of determining a capture frame of a first image in an optional implementation manner of the first embodiment of the present invention;
[0021] Figure 4 is a schematic diagram of key points of the eye in an optional implementation manner of an embodiment of the present invention;
[0022] Figure 5 is a schematic diagram of determining a capture frame corresponding to a first image in an optional manner of the first embodiment of the present invention;
[0023] Figure 6 is a flowchart of acquiring a third image sequence in an optional implementation manner of the first embodiment of the present invention;
[0024] Figure 7 is a schematic diagram of the positions of the target interception area and the non-target interception area according to the first embodiment of the present invention;
[0025] Figure 8 is a schematic diagram of obtaining a target image according to a third image and a second image according to the first embodiment of the present invention;
[0026] Fig. 9 is a schematic diagram of acquiring a target image in an optional implementation manner of the first embodiment of the present invention;
[0027] Fig.10 is a schematic diagram of an image processing device according to a second embodiment of the present invention;
[0028] Fig.11 is a schematic diagram of an electronic device according to a third embodiment of the present invention. DETAILED DESCRIPTION
[0029] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.
[0030] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.
[0031] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.
[0032] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.
[0033] In the embodiment of the present invention, the first image sequence is an image sequence recorded in an online teaching activity and includes the teacher's face. However, it is easy for those skilled in the art to understand that the first image sequence in the embodiment of the present invention can also be any image sequence including at least one image (including people, animals, etc.), and the embodiment of the present invention does not limit this.
[0034] With the increasing popularity of the Internet and computer technology, more and more users choose to watch online videos through the Internet. Online teaching activities allow teachers to teach learners online, greatly improving the flexibility of the teaching process, and are therefore welcomed by more and more teachers and learners. In order to make the learning process more interesting, when playing pre-recorded teaching videos or teaching videos recorded in real time during the teaching process, the image of the teacher can be replaced with other images, such as cartoon images, authorized real-life images, etc., by means of expression transfer, action transfer, etc.
[0035] Figure 1 Schematic diagram of the interface of online teaching activities in an embodiment of the present invention. Figure 1 As shown, when the learner is conducting an online teaching activity, the terminal interface can display a window 11 for displaying the teaching content of the online teaching activity, a window 12 for displaying the collected facial image of the learner, and a window 13 for displaying the collected facial image of the teacher and at least one of the body movements of the teaching video. In order to increase the learning interest of the learner, the facial image of the teacher and at least one of the body movements in the teaching video can be combined with a predetermined image to generate a teaching video including the facial image of the target image and at least one of the body movements (that is, the processed teaching video is generated by at least one of the expression migration and the movement migration), and the processed teaching video is played through the window 13. However, in the process of recording the teaching video, the teacher usually does not stay in the same position for a long time, so the size of the teacher presented in the teaching video is always changing. When the existing image processing method performs expression migration and / or movement migration, it usually takes less consideration of the size of the teacher in the video, so when the original image in the video (that is, the image of the teacher) is replaced with other images, the replacement effect is not good.
[0036] Figure 2 FIG. 1 is a flowchart of an image processing method according to a first embodiment of the present invention. Figure 2 As shown, the method of this embodiment includes the following steps:
[0037] Step S100: acquiring a first image sequence.
[0038] In this embodiment, the first image sequence is also a teaching video. The first image sequence includes a plurality of first images acquired according to a predetermined period, and each first image includes an original image, that is, an image of a teacher. The purpose of this embodiment is to transfer the facial expression of the teacher, so the first image at least includes the face of the teacher.
[0039] In one possible scenario, an image acquisition device may be used to acquire in real time an image sequence of a teacher in at least one historical online teaching activity, or an image sequence of a teacher in any other situation, and each image sequence may be stored in a database according to the teaching content corresponding to each image sequence. In subsequent online teaching activities, the server may obtain the corresponding first image sequence from the database according to the teaching content of the ongoing online teaching activity.
[0040] In another possible scenario, an image acquisition device may be used to acquire in real time an image sequence of a teacher in an ongoing online teaching activity, and the acquired image sequence may be used as the first image sequence.
[0041] Step S200, acquiring a second image.
[0042] In this embodiment, the second image includes a predetermined image. The predetermined image is used to represent the image of the non-teacher, for example, a cartoon image, an authorized, real person image, etc. The predetermined image can be selected by the learner. Optionally, the terminal can respond to the learner's selection operation for the predetermined image and send a selection request for the predetermined image to the server, so that the server can determine the second image including the predetermined image after receiving the request.
[0043] Step S300, determining the size of the capture frame corresponding to each first image according to the size of the second image, and capturing the corresponding first image based on each capture frame to obtain a third image sequence.
[0044] Optionally, before this step, in order to ensure the migration effect of expression migration (that is, image replacement effect), the server can pre-crop the size of the second image so that the face of the predetermined image can be located in the middle position of the second image, and the proportion of the face of the predetermined image in the cropped second image is greater than the predetermined proportion.
[0045] In this embodiment, the second image is a square image, and the width and height of the second image may be the same or different. For ease of description, this embodiment is described by taking the case where the width and height of the second image are the same as an example. In order to make the size of the captured original image closer to the size of the predetermined image and further improve the image replacement effect, the server determines the size of the capture frame corresponding to each first image based on the size of the second image. It is easy to understand that the third image sequence also includes multiple third images, and each third image includes the original image.
[0046] Figure 3 FIG. 1 is a flow chart of determining a capture frame of a first image in an optional implementation of the first embodiment of the present invention. Figure 3 As shown, in an optional implementation of this embodiment, step S300 may include the following steps:
[0047] Step S310, determining a first ratio.
[0048] In this step, the server can perform key point detection on the second image in various existing ways to determine the eye key point positions of the predetermined image, and determine the eye distance of the predetermined image according to the eye key point positions of the predetermined image, and then determine the ratio of the eye distance of the predetermined image to the width of the second image as the first ratio. The eye distance can be the outer width of the two eyes, the inner width of the two eyes, the width of the palpebral fissure (that is, the straight line distance between the outer corner point of the eye and the inner corner point of the eye of the same eye), or the pupil-eye distance. In this embodiment, the inner width of the two eyes is selected as the eye distance.
[0049] Optionally, the server can use Dlib to determine the position of the eye key points of the predetermined image by means of key point detection, and then determine the eye distance of the predetermined image according to the position of the inner eye corner points of the two eyes. Dlib is a C++ open source toolkit that includes machine learning algorithms. In Dlib, the facial features and contours are identified by 68 key points, of which key points 37 to key points 48 are eye key points in Dlib. Figure 4 It is a schematic diagram of eye key points in an optional implementation manner of an embodiment of the present invention. Figure 4 The image shown is a human eye image, and key points 37 to 48 are eye key points. The server can calculate the distance between the inner corner points of the two eyes, that is, the distance between key point 40 and key point 43 as the eye distance.
[0050] Step S320: for each first image, determine the width of the corresponding capture frame based on the eye distance of the corresponding original image and the first ratio.
[0051] In this step, the server can also perform key point detection on each first image in various existing ways, determine the eye key point positions of the original image in each first image, and determine the eye distance of the original image in each first image according to the eye key point positions of the original image in each first image. Furthermore, for each first image, the server can determine the width of the corresponding capture frame according to the eye distance of the corresponding original image and the first ratio. Specifically, for each first image, the server can determine the ratio of the eye distance of the corresponding original image to the first ratio as the width of the corresponding capture frame.
[0052] Figure 5 FIG. 1 is a schematic diagram of determining a capture frame corresponding to a first image in an optional manner of the first embodiment of the present invention. Figure 5 As shown, image 51 is the second image, and the width of image 51 is w1. After determining the ratio of the eye distance d1 of the predetermined image in the second image to the width w1 of the second image, the server can determine image 52 according to the eye distance d2 of the original image, that is, the width w2 of the interception frame 53 corresponding to the first image = d2*w1 / d1.
[0053] Step S330, determining a second ratio.
[0054] In this step, the server can perform key point detection on the second image by various existing methods to determine the eye key point positions of the predetermined image, and determine the eye center point of the predetermined image according to the eye key point positions of the predetermined image, and then determine the shortest distance between the eye center point of the predetermined image and the upper edge of the second image as the first distance, thereby determining the ratio of the first distance to the height of the second image as the second ratio. The eye center point can be the midpoint of the outer corners of the two eyes, the midpoint of the inner corners of the two eyes, and the midpoint of the pupils of the two eyes. In this embodiment, the midpoint of the inner corners of the two eyes is selected as the eye center point.
[0055] Still Figure 4 After determining that the inner corners of the two eyes are key points 40 and 43, the server can calculate the midpoint P between key points 40 and 43 as the eye center point of the predetermined image, and the distance between midpoint P and key point 40 and the distance between midpoint P and key point 43 are both d.
[0056] It is easy to understand that in an optional implementation of this embodiment, step S310 and step S330 can be executed simultaneously or successively, and this embodiment does not limit this.
[0057] Step S340: for each first image, determine the height of the corresponding capture frame based on the corresponding second distance and the second ratio.
[0058] In this step, the server can also perform key point detection on each first image in various existing ways, determine the eye key point position of the original image in each first image, and determine the eye center point of the original image in each first image according to the eye key point position of the original image in each first image. Furthermore, for each first image, the server can determine that the shortest distance between the eye center point of the corresponding original image and the upper edge of the first image is the second distance, and determine the width of the corresponding capture frame according to the corresponding second distance and the second ratio. Specifically, for each first image, the server can determine the ratio of the corresponding second distance to the second ratio as the height of the corresponding capture frame.
[0059] Still Figure 5 The following is an example of the captured frame shown in the figure. Figure 5 As shown, image 51 is the second image, and the height of image 51 is also w1, that is, image 51 is a square image with a side length of w1. After determining the ratio of the shortest distance y1 between the center point of the eye of the predetermined image in the second image and the upper edge of the second image to the width w1 of the second image, the server can determine image 52 according to the shortest distance y2 between the center point of the eye of the original image and the upper edge of the second image, that is, the width w2 of the interception frame 53 corresponding to the first image = y2*w1 / y1.
[0060] After determining the size of the capture frame corresponding to each first image, the server may capture the corresponding first image based on each capture frame according to the order of the first images in the first image sequence, and obtain the third image corresponding to each first image, thereby obtaining a third image sequence.
[0061] Figure 6 FIG. 4 is a flow chart of obtaining a third image sequence in an optional implementation of the first embodiment of the present invention. Figure 6 As shown, in an optional implementation of this embodiment, step S300 may also include the following steps:
[0062] Step S350: for each first image, determine a corresponding clipping area based on the position of the eye center point of the corresponding original image, the width of the corresponding clipping frame, the corresponding second distance, and the height of the clipping frame.
[0063] In this embodiment, the clipping area is used to represent the position of the clipping frame relative to the corresponding clipping frame relative to the corresponding first image. The server can determine the center position of the clipping area as the position of the eye center point of the original image in the first image. Figure 5The interception box shown is used as an example. The coordinates of the center point of the eye of the original image in the first image are (x, y), and the server can determine that the width range of the interception area is between [x-w2 / 2, x+w2 / 2], and the height range is between [y-w2*y1 / w1, y+w2*(1-y1 / w1)].
[0064] Step S360, determining the current first image.
[0065] In this step, the server may determine each first image as the current first image according to the order of each first image in the first image sequence.
[0066] Step S3A, determining whether the current first image is the first image in the first image sequence.
[0067] If so, execute step S370; if not, execute step S380.
[0068] Step S370: intercept the first image according to the interception area to obtain a corresponding third image.
[0069] If the current first image is the first image in the first image sequence, the server may directly capture the first image according to the capture area corresponding to the first image to obtain a third image corresponding to the first image.
[0070] Step S380, determining the intersection-over-union ratio of the target clipping area and the non-target clipping area.
[0071] In this embodiment, the non-target captured area is the captured area corresponding to the previous first image of the current first image, and the intersection ratio is the ratio of the overlapping range of the target captured area and the non-target captured area (that is, the intersection) and the overall range of the target captured area and the non-target captured area (that is, the union).
[0072] Figure 7 Schematic diagram of the positions of the target interception area and the non-target interception area according to the first embodiment of the present invention. Figure 7 As shown, the clipping area 71 is a non-target clipping area, the clipping area 72 is a target clipping area, wherein area A3 is the overlapping range of the clipping area 71 and the clipping area 72, area A1 is the remaining range after removing area A3 from the clipping area 71, and area A2 is the remaining range after removing area A3 from the clipping area 72. The server can determine that the intersection of the clipping area 71 and the clipping area 72 is area A3, and determine that the union of the clipping area 71 and the clipping area 72 is area A1+area A2+area A3, so that the intersection-union ratio of the clipping area 71 and the clipping area 72 can be determined as area A3 / (area A1+area A2+area A3).
[0073] Step S390, in response to the intersection-over-union ratio being less than a predetermined threshold, updating the clipping area of the current first image to a non-target clipping area, and clipping the current first image according to the non-target clipping area to obtain a corresponding third image.
[0074] If the intersection-and-union ratio is less than a predetermined threshold, it means that the movement of the original image is not large. Therefore, when the intersection-and-union ratio of the target interception area and the non-target interception area is less than a predetermined threshold, the server can update the interception area of the current first image to the non-target interception area, and intercept the current first image according to the non-target interception area to obtain the third image corresponding to the current first image. In this way, the original image can be made to appear to be in a moving state, so that the target image can be made to appear to be in a moving state later, effectively improving the authenticity of the teaching video and improving the learner's perception.
[0075] Step S400: obtaining a target image sequence based on a predetermined image processing model according to the second image and each third image.
[0076] In this embodiment, the second image is used as the source image of the image processing model, and each third image is used as the driving image of the image processing model. The server can form an image pair with each third image and the second image, and then input each image pair into the image processing model in the order of each third image to obtain a target image sequence composed of multiple target images. Each target image includes a target image, and the target image is a predetermined image with the posture of the original image. Optionally, the posture can include at least one of the actions, facial expressions, etc. of the original image. It is easy to understand that the number of target images in the target image sequence is the same as the number of third images in the third image sequence.
[0077] Figure 8 FIG. 1 is a schematic diagram of obtaining a target image according to the third image and the second image in the first embodiment of the present invention. Figure 8 As shown, image P1 is a driving image, i.e., the third image, and the image in image P1 is the original image, and the facial expression of the original image is 81. Image P2 is a source image, i.e., the second image, and the image in image P2 is a predetermined image, and the facial expression of the predetermined image is 82. After image P1 and image P2 are combined into an image pair and input into the image processing model, image P3, i.e., the target image, can be obtained. The image in image P3 is the target image, and the facial expression of the target image is facial expression 83, that is, facial expression 83 is the facial expression of the original image, and the facial features are the facial features of the predetermined image.
[0078] In this embodiment, the image processing model belongs to a type of video synthesis model, and can be specifically FOMMIA (First Order Motion Model for Image Animation). FOMMIA mainly includes two modules, namely a motion estimation module and an image generation module. The motion estimation module is used to predict the dense motion field from a frame of the driving video (i.e., the driving image) D to the source image S. The dense motion field maps each pixel in D to S, which can be represented by the function T S←D :R 2 →R 2 It represents the reverse optical flow. The motion prediction module consists of two submodules, namely the key point detector and the dense motion network.
[0079] Optionally, the server can train the first-order motion model of image animation based on the method described in "First Order Motion Model for Image Animation, Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, NicuSebe, Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI), 2020". Specifically, in the training samples used to train the first-order motion model of image animation, the images included in each training image are objects of the same category as the original image and the predetermined image. For example, if it is used to achieve action transfer, the images included in the training images are all performing the same or different actions; if it is used to achieve expression transfer, the images included in the training images all have the same or different expressions.
[0080] Fig. 9 FIG. 1 is a schematic diagram of obtaining a target image in an optional implementation of the first embodiment of the present invention. Fig. 9 As shown, image 91 is a predetermined image, and image 92 is an original image. After the server combines a driving image including image 91 (i.e., the third image) and a source image including image 92 (i.e., the second image) into an image pair, the server processes the image pair using a first-order motion model of the image animation to obtain a target image including image 93. Image 93 is also the target image, and image 93 is image 91 having the posture of image 92.
[0081] This embodiment obtains a first image sequence including an original image and a second image including a predetermined image, and determines the size of the cropped frame corresponding to each first image in the first image sequence according to the size of the second image, and then crops the corresponding first image based on each cropped frame to obtain a third image, thereby obtaining a target image sequence including a predetermined image with the posture of the original image based on the second image and each third image and a predetermined image processing model. This embodiment determines the cropped frame size of each frame image in the video according to the image size of the predetermined image, so as to crop the original image in each frame image, so that the cropped original image is closer in size to the predetermined image, thereby effectively improving the image replacement effect.
[0082] Fig.10 Schematic diagram of an image processing device according to a second embodiment of the present invention. Fig.10 As shown, the device of this embodiment includes a first sequence acquisition unit 101 , a second image acquisition unit 102 , a third sequence acquisition unit 103 and a target sequence acquisition unit 104 .
[0083] The first sequence acquisition unit 101 is used to acquire a first image sequence, wherein the first image sequence includes a plurality of first images, and each of the first images includes an original image. The second image acquisition unit 102 is used to acquire a second image, and each of the second images includes a predetermined image. The third sequence acquisition unit 103 is used to determine the size of a capture frame corresponding to each of the first images according to the size of the second image, and to capture the corresponding first image based on each of the capture frames to acquire a third image sequence, wherein the third image sequence includes a plurality of third images, and each of the third images includes the original image. The target sequence acquisition unit 104 is used to obtain a target image sequence based on the second image and each of the third images based on a predetermined image processing model, wherein each of the target images in the target image sequence includes a target image, and the target image is the predetermined image having the posture of the original image.
[0084] This embodiment obtains a first image sequence including an original image and a second image including a predetermined image, and determines the size of the cropped frame corresponding to each first image in the first image sequence according to the size of the second image, and then crops the corresponding first image based on each cropped frame to obtain a third image, thereby obtaining a target image sequence including a predetermined image with the posture of the original image based on the second image and each third image and a predetermined image processing model. This embodiment determines the cropped frame size of each frame image in the video according to the image size of the predetermined image, so as to crop the original image in each frame image, so that the cropped original image is closer in size to the predetermined image, thereby effectively improving the image replacement effect.
[0085] Fig.11is a schematic diagram of an electronic device according to a third embodiment of the present invention. Fig.11 The electronic device shown is a general data processing device, which can be specifically the first terminal, the second terminal or the server of the embodiment of the present invention, and includes a general computer hardware structure, which at least includes a processor 111 and a memory 112. The processor 111 and the memory 112 are connected via a bus 113. The memory 112 is suitable for storing instructions or programs executable by the processor 111. The processor 111 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 111 executes the commands stored in the memory 112, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 113 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 114 and the display device and the input / output (I / O) device 115. The input / output (I / O) device 115 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output (I / O) device 115 is connected to the system via an input / output (I / O) controller 116.
[0086] The memory 112 may store software components such as an operating system, a communication module, an interaction module, and an application program. Each of the modules and applications described above corresponds to a set of executable program instructions that implement one or more functions and methods described in the embodiments of the invention.
[0087] The flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the present invention. It should be understood that each block of the flowchart and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that (via the processor of a computer or other programmable data processing device) instructions are created to implement the device for the function / action specified in the flowchart and / or block diagram block or block.
[0088] At the same time, as will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the following forms: a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.) or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit", "module" or "system". In addition, aspects of the present invention may take the following forms: a computer program product implemented in one or more computer-readable media, the computer-readable media having a computer-readable program code implemented thereon.
[0089] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.
[0090] A computer readable signal medium may include a propagated data signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.
[0091] Computer program code for performing operations for various aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, PHP, Python, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server as a stand-alone software package. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a first image sequence, wherein the first image sequence includes a plurality of first images, and each of the first images includes an original image; acquiring a second image, wherein the second image includes a predetermined image; Generate a clipping frame adapted to each first image according to the eye distance, the eye center point and the size of the second image of the predetermined image, and determine a clipping area based on the eye center point of the original image, the corresponding clipping frame size and the second distance to obtain a third image sequence, wherein the third image sequence includes a plurality of third images, and each of the third images includes the original image; According to the second image and each of the third images, a target image sequence is obtained based on a predetermined image processing model, each target image in the target image sequence includes a target image, and the target image is the predetermined image having the posture of the original image; The step of generating a capture frame adapted to each first image according to the eye distance, the eye center point and the size of the second image of the predetermined image comprises: Determining a first ratio, the first ratio being a ratio of an eye distance of the predetermined image to a width of the second image, the eye distance of the predetermined image being determined based on key point detection; For each of the first images, determining a width of the corresponding interception frame based on the eye distance of the corresponding original image and the first ratio; Determine a second ratio, where the second ratio is a ratio of a first distance to a height of the second image, where the first distance is a shortest distance between an eye center point of the predetermined image and an upper edge of the second image, where the eye center point of the predetermined image is determined based on key point detection; For each of the first images, determining the height of the corresponding interception frame based on the corresponding second distance and the second ratio, wherein the second distance is the shortest distance between the center point of the eye of the original image and the upper edge of the first image; The step of determining the interception area based on the eye center point of the original image, the corresponding interception frame size, and the second distance to obtain the third image sequence includes: For each of the first images, determining a corresponding clipping area based on the position of the eye center point of the corresponding original image, the width of the corresponding clipping frame, the corresponding second distance, and the height of the clipping frame, wherein the clipping area is used to represent the position of the corresponding clipping frame relative to the corresponding first image; Determine the current first image; In response to the current first image being the first image in the first image sequence, intercepting the first image according to the interception area to obtain the corresponding third image; In response to the current first image not being the first image in the first image sequence, determining an intersection-over-union ratio of a target clipping area and a non-target clipping area, wherein the target clipping area is a clipping area corresponding to the current first image, and the non-target clipping area is a clipping area corresponding to a previous first image; In response to the intersection-over-union ratio being less than a predetermined threshold, the clipped area of the current first image is updated to the non-target clipped area, and the current first image is clipped according to the non-target clipped area to obtain the corresponding third image.
2. The method according to claim 1, characterized in that The acquiring of the second image comprises: In response to receiving a selection request for the predetermined image, the second image is acquired.
3. The method according to claim 1, characterized in that The image processing model is a first-order motion model of image animation.
4. An image processing device, characterized in that: The device comprises: A first sequence acquisition unit, used for acquiring a first image sequence, wherein the first image sequence includes a plurality of first images, and each of the first images includes an original image; A second image acquisition unit, used for acquiring second images, each of which includes a predetermined image; a third sequence acquisition unit, configured to generate a clipping frame adapted to each first image according to the eye distance, the eye center point and the size of the second image of the predetermined image, and determine a clipping area based on the eye center point of the original image, the corresponding clipping frame size and the second distance, so as to acquire a third image sequence, wherein the third image sequence includes a plurality of third images, and each of the third images includes the original image; a target sequence acquisition unit, configured to obtain a target image sequence based on a predetermined image processing model according to the second image and each of the third images, wherein each of the target images in the target image sequence includes a target image, and the target image is the predetermined image having the posture of the original image; Wherein, the third sequence acquisition unit is further used for: Determining a first ratio, the first ratio being a ratio of an eye distance of the predetermined image to a width of the second image, the eye distance of the predetermined image being determined based on key point detection; For each of the first images, determining a width of the corresponding interception frame based on the eye distance of the corresponding original image and the first ratio; Determine a second ratio, where the second ratio is a ratio of a first distance to a height of the second image, where the first distance is a shortest distance between an eye center point of the predetermined image and an upper edge of the second image, where the eye center point of the predetermined image is determined based on key point detection; For each of the first images, determining the height of the corresponding interception frame based on the corresponding second distance and the second ratio, wherein the second distance is the shortest distance between the center point of the eye of the original image and the upper edge of the first image; For each of the first images, determining a corresponding clipping area based on the position of the eye center point of the corresponding original image, the width of the corresponding clipping frame, the corresponding second distance, and the height of the clipping frame, wherein the clipping area is used to represent the position of the corresponding clipping frame relative to the corresponding first image; Determine the current first image; In response to the current first image being the first image in the first image sequence, intercepting the first image according to the interception area to obtain the corresponding third image; In response to the current first image not being the first image in the first image sequence, determining an intersection-over-union ratio of a target clipping area and a non-target clipping area, wherein the target clipping area is a clipping area corresponding to the current first image, and the non-target clipping area is a clipping area corresponding to a previous first image; In response to the intersection-over-union ratio being less than a predetermined threshold, the clipped area of the current first image is updated to the non-target clipped area, and the current first image is clipped according to the non-target clipped area to obtain the corresponding third image.
5. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 3 when executed by a processor.
6. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 3.
