Image processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an image processing method and device, electronic equipment and storage medium, the method comprising: when it is detected that a current to-be-processed video frame includes a display object, obtaining a region image of a first region; rendering the region image into a second region corresponding to the display object to obtain a special effect video frame corresponding to the current to-be-processed video frame, and displaying; based on the special effect video frame, determining a target special effect video. Embodiments of the present disclosure solve the problem that the special effect content in the prior art is relatively single and the content is poor, thereby causing poor user experience, and achieve rich special effect content and strong association with users, thereby improving the effect of user experience.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of internet technology, more and more applications have entered users' lives, especially a series of software that can shoot short videos, which are very popular among users.

[0003] In existing technologies, related application software can provide users with various image processing functions, allowing an image to present a different visual effect after processing. However, although it also applies special effects to the user, the processing result is not integrated with the content displayed in the current video frame, resulting in a lack of interactivity between the special effects and the user, thus degrading the user experience. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, electronic device, and storage medium that enables the generation of corresponding special effects video content based on the display information of the display object, thereby improving the interactivity and fun between the special effects video content and the user, and ultimately enhancing the user experience.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method, including:

[0006] When a display object is detected in the current video frame to be processed, the region image of the first region is obtained;

[0007] The image of the region is rendered into a second region corresponding to the display object to obtain a special effects video frame corresponding to the current video frame to be processed, and then displayed.

[0008] The target special effects video is determined based on the special effects video frames.

[0009] Secondly, embodiments of this disclosure also provide an image processing apparatus, comprising:

[0010] The image acquisition module is used to acquire a region image of a first region when a display object is detected in the current video frame to be processed;

[0011] An image rendering module is used to render the image of the region to a second region corresponding to the display object, to obtain a special effects video frame corresponding to the current video frame to be processed, and to display it;

[0012] The video determination module is used to determine the target special effects video based on special effects video frames.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any of the embodiments of this disclosure.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any of the embodiments of this disclosure.

[0018] The technical solution of this disclosure embodiment, when detecting that the current video frame to be processed includes a display object, obtains a region image of a first region, and then renders the region image into a second region corresponding to the display object to obtain a corresponding special effects video frame. This realizes that the content of the special effects display changes based on the changes in the region image in the display interface, improves the correlation between the display content and the special effects, and enhances the user's interest in participating in the creation of special effects, thereby improving the user experience. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram illustrating a special effect provided in an embodiment of the present disclosure;

[0022] Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should also be noted that the modifications of "a" and "a plurality of" mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] Before introducing this technical solution, the application scenarios of the embodiments of this disclosure can be illustrated by example. In any software capable of image capture, users can capture videos or images based on the corresponding software. To enhance the interest and richness of the content, special effects are usually applied to the videos or images. For example, the technical solutions provided by the embodiments of this disclosure can be integrated into existing video live streaming, video conferencing, or video conversation software.

[0031] It should be noted that, based on human body segmentation technology, the local information corresponding to the displayed object can be determined. For example, the local information can be the user's exposed torso, a region, the head region, etc.

[0032] It should also be noted that the apparatus for executing the image processing method provided in this embodiment can be integrated into application software that supports special effects video processing functions, and this software can be installed on an electronic device, optionally a mobile terminal or a PC. The application software can be a type of software for image / video processing; specific application software will not be detailed here, as long as it can perform image / video processing. Alternatively, it can be a specially developed application program to add and display special effects, or it can be integrated into a corresponding page, allowing users to process special effects videos through the integrated page on a PC.

[0033] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the situation where the display information of the display object generates corresponding special effects videos. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, such as a mobile terminal, a PC, or a server.

[0034] like Figure 1 As shown, the method includes:

[0035] S110. When a display object is detected in the current video frame to be processed, the region image of the first region is obtained.

[0036] It should be noted that the triggering time for acquiring the region image can be when a display object is detected in the frame; it can also be when the user triggers a special effects prop while shooting a special effects video, and the region image is extracted as long as the display object is detected in the current video frame; or it can be acquired when a special effects wake word or body movement is detected.

[0037] In practical applications, if special effects are triggered, the sequentially captured video frames to be processed can be treated with special effects to obtain the corresponding special effects video frame. The processing method for each video frame is the same. Here, we will take the processing of one video frame as an example, and the currently described video frame can be used as the current video frame to be processed.

[0038] The displayed object can be a user, an animal, etc. The displayed object can be a pre-defined specific object or any object. For example, if the goal is to obtain a special effects video corresponding to user A, an image corresponding to user A can be uploaded beforehand for feature recognition and recording. When user A is detected in the current video frame to be processed, the region image is acquired and special effects processing is performed. If special effects processing is to be performed on all users, the displayed object does not need to be defined; special effects processing can be performed as long as the displayed interface includes a displayed object. Alternatively, if the goal is to process objects of a specific type, the object type can be defined. For example, if the goal is to perform special effects processing on a pet, the object type can be defined as "pet." When the displayed object in the current video frame to be processed is detected as a pet object, a region image is acquired. The region image can be an image corresponding to any region in the video frame to be processed, a pre-defined specific region, or a region image corresponding to a specific part of the displayed object. For example, four pixels can be pre-defined, and the rectangular area corresponding to the four pixels can be used as the first region; or, the facial area of ​​the displayed object can be used as the first region. Accordingly, the image within the first region is the region image.

[0039] Specifically, it can capture video frames to be processed in real time, and when it detects that the current video frame to be processed includes a display object, it can acquire a region image of the first region to determine the corresponding special effects video frame based on the region image.

[0040] In this embodiment, obtaining the region image of the first region can be achieved by: using a key point recognition algorithm to determine the first region and extracting the region image of the first region.

[0041] Keypoint recognition algorithms can include skeletal point recognition algorithms and facial keypoint recognition algorithms. Skeletal point recognition algorithms can identify skeletal points on a user's body; facial keypoint recognition algorithms can acquire key points on a facial image. The reason and benefit of identifying key points is that corresponding regions can be identified, thereby enabling effective image processing.

[0042] S120. Render the region image into the second region corresponding to the display object to obtain the special effects video frame corresponding to the current video frame to be processed, and display it.

[0043] In this embodiment, the second region can be determined and then rendered before the region image is rendered into the second region.

[0044] Optionally, a skeletal point recognition algorithm is used to determine the calibration key points, and the region to be fused is determined based on the calibration key points; the second region is determined based on the region to be fused and the masking region corresponding to the display object.

[0045] The key points used for calibration can be pre-defined key points, such as the left and right shoulder bone points and the center point of the clavicle. Based on the calibrated key points and a pre-defined width, a rectangular area can be determined, and this rectangular area is used as the area to be blended. The upper garment area can be used as the masking area. Of course, if the user is wearing a skirt, then the masking area corresponds to the skirt. Based on the intersection of the area to be blended and the masking area, a second area can be determined.

[0046] Specifically, a skeletal point recognition algorithm can be used to determine the skeletal points of the left and right shoulders and the center point of the clavicle. After determining these three points, a rectangular area to be merged can be defined based on a pre-set width. Simultaneously, a segmentation algorithm can be used to segment the user's various regions, such as the face, hair, upper garment, lower garment, and other areas not obscured by clothing. Different colors or fill lines can be used to fill different regions. If the masked area is the upper garment area, the intersection of the rectangular area and the upper garment area can be determined, and this intersection area can be used as the second region.

[0047] In this embodiment, the advantage of using the above method to determine the second region is that it can accurately determine the image rendering area, so that the rendered image fits the second region better, thereby improving the realism of the special effects image.

[0048] Based on the skeletal point recognition algorithm, target key points corresponding to the pose information are determined; and an updated region to be fused is obtained based on the target key points and the calibration key points. In addition to the above technical solution, the step of using the skeletal point recognition algorithm to determine calibration key points and determining the region to be fused based on the calibration key points includes: determining target key points corresponding to the pose information based on the skeletal point recognition algorithm; and obtaining an updated region to be fused based on the target key points and the calibration key points.

[0049] The displayed object can be a user. The posture information can be the current posture of the displayed object, such as an open-arms posture or an embracing posture. Different methods can be used to determine the region to be fused for different user postures. Target key points can be key points identified by a skeletal point recognition algorithm. Target key points can be elbow key points. For example, a human skeletal point algorithm can be used to obtain the coordinates and confidence scores of the corresponding points. Based on the confidence score, the information corresponding to the point can be determined. The confidence score characterizes whether the current point is the corresponding point; optionally, the higher the confidence score, the more reliable the identified point, and vice versa. If the posture information is an open-arms posture, key points on the elbows can be identified as target key points. Based on the key points on the elbows, the center point of the clavicle, and the points on the shoulders, the region to be fused can be determined.

[0050] After determining the region to be merged, image rendering can be performed based on the region to be merged. Optionally, the scaling ratio can be determined based on the size information of the first region and the region to be merged. Based on the scaling ratio, the image of the region located in the first region is rendered to the second region.

[0051] Specifically, based on the length and width information of the first region and the size information of the region to be fused, optionally, the size information can be length and width information. Based on the length and width information, the ratio of the long side and the ratio of the wide side can be determined as the scaling ratio. Based on the scaling ratio, the region image in the first region can be rendered into the second region.

[0052] The method for rendering the region image of the first region to the second region can be as follows: Align the vertices of the first region and the region to be merged to obtain UV textures for alignment. Next, sample the pixels in the region image and render them to the second region so that the image displayed in the second region matches the region image.

[0053] In this embodiment, the target texture coordinates of the region image corresponding to the second region are determined based on the vertex texture coordinates of the first region and the scaling ratio, and then rendered.

[0054] Specifically, based on the vertex texture coordinates and scaling ratio of the first region, the texture coordinates of each texture coordinate in the region image can be determined to correspond to the texture coordinates of the region to be merged. Further, based on the texture coordinates of the region to be merged and the masking region, the target texture coordinates located in the second region can be determined. Sampling based on the target texture coordinates allows the region image to be sampled to obtain the region image rendered to the second region.

[0055] Based on the above technical solution, when the pose of the target object differs, the size information of the determined fusion region will also differ. If the region image corresponds to a user's head image, to avoid displaying more pixels other than the head region in the second region, the size information of the updated fusion region can be determined by comparing the scaling ratio of the fusion region before and after the update with that of the first region, and this ratio can be used as the magnification ratio. Based on this magnification ratio, the image of the second region can be magnified. The magnified region can be a facial image.

[0056] The advantage of rendering images in the above manner is that the second area mainly displays facial images, which may directly meet the user's eyes, achieving an effect where facial images and clothing textures are matched.

[0057] For example, see Figure 2 When the display interface detects that a display object, user A, is included, the technical solution provided in this embodiment can be executed to determine that the region image is a facial image. At this time, the computer can use a skeletal point recognition algorithm to collect elbow key points and pre-set calibration key points. Based on the calibration key points and a preset length, the region to be fused is determined. Based on the region to be fused and the facial image region, a scaling ratio can be determined. Based on the scaling ratio, the vertex texture coordinates of the facial image are determined to correspond to the target vertex texture coordinates of the region to be fused. Next, based on the region to be fused and the target key points, a magnification ratio in the horizontal direction relative to the ground can be determined, and based on this magnification ratio, the target vertex texture coordinates corresponding to the vertex texture coordinates are determined. Further, based on the upper garment masking region and the region to be fused, a second region is determined. The region image can be rendered to the second region based on the vertex texture coordinates, the corresponding target vertex texture coordinates, and the second region. A specific effect illustration can be found in [reference needed]. Figure 2 As shown.

[0058] S130. Based on the special effects video frames, determine the target special effects video.

[0059] Specifically, the target special effects video is obtained by splicing together the generation timestamps of each special effects video frame.

[0060] The technical solution of this disclosure embodiment, when detecting that the current video frame to be processed includes a display object, obtains a region image of a first region, and then renders the region image into a second region corresponding to the display object to obtain a corresponding special effects video frame. This realizes that the content of the special effects display changes based on the changes in the region image in the display interface, improves the correlation between the display content and the special effects, and enhances the user's interest in participating in the creation of special effects, thereby improving the user experience.

[0061] Figure 3This is a schematic flowchart of an image processing method provided by an embodiment of the present disclosure. Based on the foregoing embodiments, a regional image can be rendered into a corresponding second region to obtain a corresponding special effects video. For specific implementation details, please refer to the detailed description of this technical solution. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0062] It should be noted that the number of displayed objects can be one or more. Different numbers of displayed objects can result in different rendered video effects. Optionally, the region images can be rendered separately onto the second region corresponding to each displayed object. That is, the second region can be the upper garment area or the full body area of ​​the displayed object. The region images are then rendered separately onto the second region of each displayed object.

[0063] like Figure 3 As shown, the method includes:

[0064] S210. When it is detected that the current video frame to be processed includes a display object, the first region is determined according to the number of objects in the display object.

[0065] The number of objects can be the number of displayed objects. The first and second regions can be determined based on the number of displayed objects. Optionally, the first region can be a facial image of a displayed object. A keypoint recognition algorithm can be used to extract the facial image of each displayed object separately. The second region can be the upper garment or full-body area of ​​each displayed object.

[0066] Optionally, the first region can be determined based on the number of displayed objects, which can be achieved by obtaining a region image within the first region of the target displayed objects.

[0067] The target display object is the region from which its facial image needs to be extracted. For example, if there are five display objects in the display interface, and display object A is pre-labeled as the target display object, a key point recognition algorithm can be used to determine the target display object A, and the region corresponding to the facial image of display object A can be used as the first region. Accordingly, the facial image within the first region is used as the region image.

[0068] This can be understood as follows: a target display object can be pre-defined. If the display interface includes this target display object, it indicates that special effects processing is needed for the content in the display interface. In this case, the facial image corresponding to the target display can be obtained and used as the region image. This achieves the effect of customizing the content rendered to the second region.

[0069] Another way to determine the region image is to obtain the region image of the first region on at least two display objects.

[0070] This can be understood as follows: as long as the display interface detects that there are display objects, the facial images of each display object can be extracted and the extracted facial images can be used as region images.

[0071] S220. Render the image of the region into the corresponding second region according to the preset rules.

[0072] The preset rule could be to render the region images separately into the second region corresponding to each display object. Alternatively, it could be to display the region images in the second regions of the at least two display objects; or, to display the region images correspondingly in the second region of the respective display object.

[0073] This can be understood as follows: Individual area images can be rendered into the second area corresponding to each display object. Alternatively, if there are multiple area images, they can be stitched together to obtain a final image, which is then rendered into the second area of ​​each display object. Or, the facial images of each display object can be rendered into the corresponding second area of ​​that display object. For example, the area image of display object A can be displayed within the second area of ​​display object A. The second area can be the ultimately determined upper garment area, pants area, skirt area, or other pre-defined areas associated with the display object.

[0074] The above method enables the second region image to be customized, or the image rendered to the second region to be associated with the display information of all display objects, thereby improving the interactivity between users.

[0075] S230. Based on the special effects video frames, determine the target special effects video.

[0076] The technical solution of this disclosure embodiment, when detecting that the current video frame to be processed includes a display object, obtains a region image of a first region, and then renders the region image into a second region corresponding to the display object to obtain a corresponding special effects video frame. This realizes that the content of the special effects display changes based on the changes in the region image in the display interface, improves the correlation between the display content and the special effects, and enhances the user's interest in participating in the creation of special effects, thereby improving the user experience.

[0077] Based on the above technical solution, there may be a problem of the regional image having a deflection angle. In order to make the regional image more compatible with the second region, or to obtain a more realistic effect, it can be that: if the regional image has a deflection angle, the regional image is adjusted based on the deflection angle to obtain the regional image rendered into the second region.

[0078] The deflection angle can be determined based on the world coordinate system. The inverse of the deflection angle can be the adjustment angle of the region image, which is used to render the region image into the second region. Alternatively, the deflection angle corresponding to the second region can be determined, and the adjustment angle can be determined based on the deflection angle of the region image and the region deflection angle. The region image can then be rendered into the second region based on the adjustment angle. The advantage of using the above methods is that they improve the fit between the facial image and the second region, thus enhancing the realism of the facial image rendering when rendering the facial image onto clothing.

[0079] It should be noted that the regional image in the second region changes based on changes in the facial information of the displayed object. The specific display information for each frame can be determined by repeatedly executing the above steps, thereby obtaining the special effects video. Furthermore, the advantage of using the above method to determine the special effects video frames is that it enables interactivity between the special effects content and the displayed object, thereby improving the user experience.

[0080] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the device includes: an image acquisition module 310, an image rendering module 320, and a video determination module 330.

[0081] The image acquisition module 310 is used to acquire a region image of a first region when a display object is detected in the current video frame to be processed; the image rendering module 320 is used to render the region image to a second region corresponding to the display object to obtain a special effects video frame corresponding to the current video frame to be processed, and display it; the video determination module 330 is used to determine a target special effects video based on the special effects video frame.

[0082] Based on the above technical solution, the image acquisition module is further used to determine the first region and extract the region image of the first region using a key point recognition algorithm.

[0083] Based on the above technical solution, the image rendering module includes:

[0084] The fusion region determination unit is used to determine the fusion region corresponding to the display object based on the posture information;

[0085] The second region determination unit is used to determine the second region based on the region to be merged and the masking region corresponding to the display object.

[0086] Based on the above technical solution, the integrated region determination unit includes:

[0087] The key point determination subunit is used to determine the target key points based on the posture information of the object to be displayed;

[0088] The region to be merged subunit is used to determine the region to be merged corresponding to the display object based on the target key points and pre-marked key points.

[0089] Based on the above technical solution, the image rendering module includes:

[0090] The scaling unit is used to determine the scaling ratio based on the size information of the first region and the region to be merged;

[0091] The second region determination unit is used to render the region image located in the first region to the second region based on the scaling ratio.

[0092] Based on the above technical solution, the second region determination unit is used to determine the target texture coordinates of the region image corresponding to the second region according to the vertex texture coordinates of the first region and the scaling ratio, and then render it.

[0093] Based on the above technical solution, the second region determination unit is used to enlarge the target sub-region in the region image and render the enlarged target sub-region image to the second region if the scaling ratio is greater than a preset scaling ratio threshold.

[0094] Based on the above technical solution, the number of display objects includes at least one, and the image rendering module is used for:

[0095] The image of the region is rendered into the second region corresponding to each display object.

[0096] Based on the above technical solution, the device further includes:

[0097] Based on the number of objects in the display object, the first region and the second region are determined, and the region image is rendered into the corresponding second region according to a preset rule.

[0098] Based on the above technical solution, the number of display objects includes at least two. According to the number of display objects, the image rendering module is used to obtain a region image within a first region of the target display object; and render the region image to a second region corresponding to each display object.

[0099] Based on the above technical solution, the number of display objects includes at least two. According to the number of display objects, the image rendering module is used to obtain region images of the first area on the at least two display objects; display the region images in the second area of ​​the at least two display objects; or, display the region images in the second area of ​​the corresponding display object.

[0100] Based on the above technical solution, the device further includes: a deflection angle correction module, used to adjust the regional image based on the deflection angle if a deflection angle is detected in the regional image, so as to obtain a regional image rendered into the second region.

[0101] Based on the above technical solution, the first region includes a preset region in the current video frame to be processed or the facial region of the target display object, and the second region is the torso region of the target display object.

[0102] The technical solution of this disclosure embodiment, when detecting that the current video frame to be processed includes a display object, obtains a region image of a first region, and then renders the region image into a second region corresponding to the display object to obtain a corresponding special effects video frame. This realizes that the content of the special effects display changes based on the changes in the region image in the display interface, improves the correlation between the display content and the special effects, and enhances the user's interest in participating in the creation of special effects, thereby improving the user experience.

[0103] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0104] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0105] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 5 The diagram below shows the structure of the terminal device or server 400. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0106] like Figure 5 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, pattern processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 406 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of electronic device 400. The processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. An edit / output (I / O) interface 405 is also connected to bus 404.

[0107] Typically, the following devices can be connected to I / O interface 405: editing devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0108] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 406, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0109] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0110] The electronic device provided in this embodiment and the image processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0111] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the image processing method provided in the above embodiments.

[0112] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0115] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0116] When a display object is detected in the current video frame to be processed, the region image of the first region is obtained;

[0117] The image of the region is rendered into a second region corresponding to the display object to obtain a special effects video frame corresponding to the current video frame to be processed, and then displayed.

[0118] The target special effects video is determined based on each special effects video frame.

[0119] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0122] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] According to one or more embodiments of this disclosure, [Example 1] provides an image processing method, the method comprising:

[0125] When a display object is detected in the current video frame to be processed, the region image of the first region is obtained;

[0126] The image of the region is rendered into a second region corresponding to the display object to obtain a special effects video frame corresponding to the current video frame to be processed, and then displayed.

[0127] The target special effects video is determined based on each special effects video frame.

[0128] According to one or more embodiments of this disclosure, [Example 2] provides an image processing method, which further includes:

[0129] Optionally, obtaining the region image of the first region includes:

[0130] A key point recognition algorithm is used to determine the first region and extract the region image of the first region.

[0131] According to one or more embodiments of this disclosure, [Example 3] provides an image processing method, which further includes:

[0132] Optionally, before rendering the region image into the second region corresponding to the display object, the method further includes:

[0133] Based on the posture information, determine the region to be merged corresponding to the display object;

[0134] The second region is determined based on the region to be merged and the masking region corresponding to the display object.

[0135] According to one or more embodiments of this disclosure, [Example 4] provides an image processing method, which further includes:

[0136] Optionally, based on the pose information of the display object, determining the region to be merged corresponding to the display object includes:

[0137] Based on the posture information of the object to be displayed, determine the target key points;

[0138] Based on the target key points and pre-marked key points, determine the region to be merged corresponding to the display object.

[0139] According to one or more embodiments of this disclosure, [Example 5] provides an image processing method, which further includes:

[0140] Optionally, rendering the region image into a second region corresponding to the display object to obtain a special effects video frame corresponding to the current video frame to be processed includes:

[0141] The scaling ratio is determined based on the size information of the first region and the region to be merged;

[0142] Based on the scaling ratio, the region image located in the first region is rendered to the second region.

[0143] According to one or more embodiments of this disclosure, [Example Six] provides an image processing method, which further includes:

[0144] Optionally, rendering the region image located within the first region to the second region based on the scaling ratio includes:

[0145] Based on the vertex texture coordinates of the first region and the scaling ratio, the target texture coordinates of the region image corresponding to the second region are determined and rendered.

[0146] According to one or more embodiments of this disclosure, [Example Seven] provides an image processing method, which further includes:

[0147] Optionally, rendering the region image located within the first region to the second region based on the scaling ratio includes:

[0148] If the scaling ratio is greater than a preset scaling ratio threshold, the target sub-region in the region image is enlarged, and the enlarged target sub-region image is rendered into the second region.

[0149] According to one or more embodiments of this disclosure, [Example Eight] provides an image processing method, which further includes:

[0150] Optionally, the number of display objects includes at least one, and rendering the region image within a second region corresponding to the display object includes:

[0151] The image of the region is rendered into the second region corresponding to each display object.

[0152] According to one or more embodiments of this disclosure, [Example Nine] provides an image processing method, which further includes:

[0153] Optionally, the first region and the second region are determined based on the number of objects in the display object, and the region image is rendered into the corresponding second region according to a preset rule.

[0154] According to one or more embodiments of this disclosure, [Example 10] provides an image processing method, which further includes:

[0155] Optionally, the number of display objects includes at least two, and the step of determining the first region and the second region based on the number of display objects, and rendering the region image to the corresponding second region according to a preset rule, includes:

[0156] Obtain the region image within the first area of ​​the target display object;

[0157] The image of the region is rendered into the second region corresponding to each display object.

[0158] According to one or more embodiments of this disclosure, [Example 11] provides an image processing method, which further includes:

[0159] Optionally, the number of display objects includes at least two, and the step of determining the first region and the second region based on the number of display objects, and rendering the region image to the corresponding second region according to a preset rule, includes:

[0160] Obtain the region image of the first region on at least two display objects;

[0161] The images of the regions are all displayed in the second region of the at least two display objects; or...

[0162] The image of the region is displayed in the second region of the corresponding display object.

[0163] According to one or more embodiments of this disclosure, [Example Twelve] provides an image processing method, which further includes:

[0164] Optionally, if a deflection angle is detected in the region image, the region image is adjusted based on the deflection angle to obtain a region image rendered into the second region.

[0165] According to one or more embodiments of this disclosure, [Example Thirteen] provides an image processing method, which further includes:

[0166] Optionally, the first region includes a preset region in the current video frame to be processed or the facial region of the target display object, and the second region is the torso region of the target display object. According to one or more embodiments of this disclosure, [Example Fourteen] provides an image processing apparatus, the apparatus comprising:

[0167] The image acquisition module is used to acquire a region image of a first region when a display object is detected in the current video frame to be processed;

[0168] An image rendering module is used to render the image of the region to a second region corresponding to the display object, to obtain a special effects video frame corresponding to the current video frame to be processed, and to display it;

[0169] The video determination module is used to determine the target special effects video based on each special effects video frame.

[0170] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0171] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0172] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: When it is detected that the display object is included in the current to-be-processed video frame, an area image of a first area is acquired; The area image is rendered into a second area corresponding to the display object to obtain a special effect video frame corresponding to the current to-be-processed video frame, and the special effect video frame is displayed; Based on the special effect video frame, a target special effect video is determined; The first area includes a face area of the display object in the current to-be-processed video frame, and the second area is a torso area of the display object.

2. The method of claim 1, wherein, The acquisition of the area image of the first area comprises the following steps: A key point recognition algorithm is used to determine the first area and extract the area image of the first area.

3. The method of claim 1, wherein, Before the rendering of the area image into the second area corresponding to the display object, the following steps are further included: A skeleton point recognition algorithm is used to determine a calibration key point, and a to-be-fused area is determined according to the calibration key point; The second area is determined according to the to-be-fused area and a mask area corresponding to the display object.

4. The method of claim 3, wherein, The determination of the calibration key point by the skeleton point recognition algorithm and the determination of the to-be-fused area according to the calibration key point comprise the following steps: A target key point corresponding to posture information is determined based on the skeleton point recognition algorithm; and an updated to-be-fused area is obtained according to the target key point and the calibration key point.

5. The method of claim 3, wherein, The rendering of the area image into the second area corresponding to the display object to obtain the special effect video frame corresponding to the current to-be-processed video frame comprises the following steps: A scaling ratio is determined according to size information of the first area and the to-be-fused area; The area image is rendered into the second area based on the scaling ratio.

6. The method of claim 5, wherein, The rendering of the area image into the second area based on the scaling ratio comprises the following steps: Target texture coordinates of the area image corresponding to the second area are determined according to vertex texture coordinates of the first area and the scaling ratio, so that the area image is rendered into the second area based on the vertex texture coordinates and the corresponding target texture coordinates.

7. The method of claim 4, wherein, Further comprising the following steps: An amplification ratio is determined according to the to-be-fused area before updating and the updated to-be-fused area; The image of the second area is amplified based on the amplification ratio.

8. The method of claim 1, wherein, The number of display objects includes at least one, and the rendering of the area image into the second area corresponding to the display object comprises the following steps: The area image is rendered into the second area corresponding to at least one display object respectively.

9. The method of claim 1, wherein, Further comprising the following steps: The first area is determined according to the number of display objects, and the area image is rendered into the corresponding second area according to a preset rule.

10. The method of claim 9, wherein, The determination of the first area according to the number of display objects and the rendering of the area image into the corresponding second area according to the preset rule comprise the following steps: An area image in the first area of a target display object is acquired; The area image is rendered into the second area corresponding to at least one display object respectively.

11. The method of claim 9, wherein, The determination of the first area according to the number of display objects and the rendering of the area image into the corresponding second area according to the preset rule comprise the following steps: acquire a region image of a first region on at least two display objects; display the region image in a second region of the at least two display objects; or, display the region image in a second region of a corresponding display object.

12. The method of claim 1, wherein, Further comprising: if a deflection angle of the region image is detected, adjusting the region image based on the deflection angle to obtain a region image rendered into the second region.

13. An image processing apparatus characterized by comprising: Comprise: an image acquisition module, configured to acquire a region image of a first region when it is detected that a display object is included in a current to-be-processed video frame; an image rendering module, configured to render the region image into a second region corresponding to the display object to obtain a special effect video frame corresponding to the current to-be-processed video frame, and display the special effect video frame; a video determination module, configured to determine a target special effect video based on each special effect video frame; wherein the first region includes a face region of the display object in the current to-be-processed video frame, and the second region is a torso region of the display object.

14. An electronic device, comprising: The electronic device comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as claimed in any one of claims 1-12.

15. A storage medium containing computer-executable instructions for performing the image processing method as claimed in any one of claims 1-12 when executed by a computer processor.

Images