Image processing method and apparatus, electronic device and storage medium

Abstract

This disclosure provides an image processing method and apparatus, an electronic device, and a storage medium. The method includes: identifying a target image frame to determine the position of a first identification object and the state of a second identification object; determining a target cropping position, which is the center position of cropping when cropping the target image frame, based on the position of the first identification object; determining a target cropping size, which is the cropping size when cropping the target image frame, based on the state of the second identification object; and cropping an image from the target image frame based on the target cropping position and target cropping size, and scaling it to a preset size. By implementing the technical proposals of the embodiments of this disclosure, it is possible to avoid the cumbersome manual lens movement process, enrich the appeal of the screen, and improve the user experience.

Description

Technical Field

[0001] This application claims priority based on Chinese Patent Application No. 202310646038.5 filed on June 1, 2023, and hereby incorporates by reference in its entirety all the content disclosed in the above-mentioned Chinese patent application as part of this application.

[0002] [Technical Field] Embodiments of the present disclosure relate to an image processing method and apparatus, an electronic device, and a storage medium.

Background Art

[0003] In the video generation process, usually, different lens movement methods are adopted to process videos. For example, when shooting a dance video, different lens movement methods can be adopted according to the movements of the limbs, so as to greatly enhance the expressiveness and charm of the video.

[0004] In related art solutions, in order to add a lens movement effect to a video, one method is to move the shooting device using a hand-held holder during shooting to achieve the lens movement effect. Another method is to post-edit an existing video to give the generated video a lens movement effect. However, realizing the lens movement effect by moving a hand-held holder requires the shooter to have a certain level of lens movement skills to meet the requirements, and the cost of realization is high. On the other hand, post-editing usually needs to be manually edited and completed, and the operation process is complex and time-consuming.

Summary of the Invention

[0005] The present disclosure provides an image processing method and apparatus, an electronic device, and a storage medium, which can avoid a complicated manual lens movement process, enrich the charm of the screen, and improve the user experience.

[0006] In a first aspect, embodiments of the present disclosure provide an image processing method, and the method includes: By identifying the target image frame, the position of the first identified object and the state of the second identified object are determined. Based on the position of the first identification object, the target cropping position, which is the center position of cropping when cropping is performed on the target image frame, is determined, and based on the state of the second identification object, the target cropping size, which is the cropping size when cropping is performed on the target image frame, is determined. This includes cropping an image from the target image frame based on the target cropping position and the target cropping size, and scaling it to a preset size.

[0007] In a second aspect, embodiments of the present disclosure further provide an image processing apparatus, which includes: An image frame identification module for determining the position of a first identified object and the state of a second identified object by identifying a target image frame, A cropping position and cropping size determination module for determining the target cropping position, which is the cropping center position when cropping is performed on the target image frame, based on the position of the first identification object, and for determining the target cropping size, which is the cropping size when cropping is performed on the target image frame, based on the state of the second identification object, The system includes an image cropping module for cropping an image from a target image frame based on the target cropping position and the target cropping size, and scaling it to a preset size.

[0008] In a third aspect, embodiments of the present disclosure further provide electronic devices, said electronic devices, One or more processors, Includes a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the image processing method described in any one embodiment of the present disclosure.

[0009] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, wherein the computer-readable medium stores the computer instructions, and the computer-executable instructions, when executed by a computer processor, are for performing an image processing method described in any one embodiment of the present disclosure. [Brief explanation of the drawing]

[0010] [Figure 1] This is a flowchart of the image processing method according to the embodiments of this disclosure. [Figure 2a] This is a flowchart of another image processing method according to an embodiment of the present disclosure. [Figure 2b] This is a flowchart of an image processing method according to an embodiment of the present disclosure. [Figure 3] This is a flowchart of yet another image processing method according to an embodiment of the present disclosure. [Figure 4] This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure. [Figure 5] This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0011] The above and other features, advantages, and aspects of each embodiment of this disclosure will become more apparent by referring to the embodiments for carrying out the invention as shown in the drawings below. In all drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and elements are not necessarily drawn to scale. The embodiments of this disclosure will be described in more detail below with reference to the drawings. While the drawings show several embodiments of this disclosure, it should be understood that this disclosure is achievable in various ways and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to allow for a more thorough and complete understanding of this disclosure. It should be understood that the drawings and embodiments of this disclosure are for illustrative purposes only and do not limit the scope of protection of this disclosure.

[0012] It should be understood that the steps described in the embodiments of the method of this disclosure may be performed in a different order and / or in parallel. Furthermore, embodiments of the method may include additional steps and / or the described steps may be omitted. The scope of this disclosure is not limited in this respect.

[0013] As used herein, the term “including” and its variations are non-restrictive “including,” meaning “including, but not limited to, ….” The term “based on” means “based at least partially on.” The term “one embodiment” means “at least one embodiment,” the term “another embodiment” means “at least one other embodiment,” and the term “several embodiments” means “at least several embodiments.” Relevant definitions of other terms are given below.

[0014] Furthermore, the concepts of "first," "second," etc., as used in this disclosure are merely for the purpose of distinguishing different devices, modules, or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules, or units.

[0015] Furthermore, the modifications “one” and “multiple” as used in this disclosure are illustrative and not restrictive, and should be understood as “one or more” unless otherwise explicitly stated in the context, as would be understood by those skilled in the art.

[0016] The names of messages or information that interact between multiple devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0017] As can be understood, before using the technical solutions disclosed by each embodiment of the present disclosure, the user should be informed of the types, scope of use, usage scenarios, etc. of personal information related to the present disclosure in an appropriate manner in accordance with relevant laws and regulations, and permission should be obtained from the user.

[0018] For example, in response to receiving a voluntary request from the user, prompt information is sent to the user so as to clearly prompt the user that the operation of requesting execution requires the acquisition and use of the user's personal information. Thereby, the user can independently select whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operation of the technical solution of the present disclosure based on the prompt information.

[0019] As a selective but non-limiting implementation form, the method of sending prompt information to the user in response to receiving a voluntary request from the user may be, for example, in the form of a pop-up window, and the prompt information may be presented in text in the pop-up window. In addition, in the pop-up window, a selection control for allowing the user to select whether to "agree" or "disagree" to provide personal information to the electronic device may be further added.

[0020] As can be understood, the above notification and the process of obtaining user permission are only schematic and do not limit the implementation forms of the present disclosure. Other methods that meet relevant laws and regulations are also applicable to the implementation forms of the present disclosure.

[0021] As can be understood, the data related to the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0022] FIG. 1 is a flowchart of an image processing method according to an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to a scene for generating a lens movement video. The method is executable by an image processing apparatus, and the apparatus can be implemented in the form of software and / or hardware, and optionally, can be implemented by an electronic device, which may be a mobile terminal, a PC, a server, or the like. As shown in FIG. 1, the method includes S110 to S130.

[0023] In S110, by identifying a target image frame, the position of the first identification object and the state of the second identification object are determined.

[0024] The position of the first identification object can be determined by the position of the target part corresponding to the first identification object identified from the target image frame. The target part includes at least one of the head position, hand position, foot position, hip position, and shoulder position of the first identification object. The state of the second identification object may be the open state or closed state of the hand of the second identification object identified from the target image frame. By identifying the target image frame to determine the position of the first identification object and the state of the second identification object, it is possible to determine the cropping position and cropping size of the target image frame, and provide a reliable data basis for realizing the effect of simulating the movement of the lens.

[0025] In S120, based on the position of the first identification object, a target cropping position, which is the cropping center position when cropping the target image frame, is determined, and based on the state of the second identification object, a target cropping size, which is the cropping size when cropping the target image frame, is determined.

[0026] In this proposed technology, a target image frame can be cropped using a cropping box, and the cropping center position may be the position of the center of the cropping box within the target image frame. This proposed technology obtains the target cropping position and target cropping size from the target image frame, directionally crops an image of a fixed size at the target cropping position, and continuously determines a new cropping position and cropping size from the image frame following the target image frame. By continuously displaying cropped images from temporally adjacent image frames, it is possible to simulate lens movement effects such as changes in position and lens movement.

[0027] In S130, the image is cropped from the target image frame based on the target cropping position and target cropping size, and then scaled to a preset size.

[0028] In this proposed technology, after determining the target cropping position and target cropping size, an image of the target cropping size is cropped at the target cropping position within the target image frame. Furthermore, depending on the relationship between the target cropping size and the preset size, each cropped image can be reduced or enlarged to the preset size. By adjusting the cropped images from temporally adjacent image frames to the preset size and displaying them continuously, it is possible to simulate changes in position and lens movement, thereby simulating lens movement effects such as zooming at a given position.

[0029] The technical solution of the embodiments of this disclosure identifies a target image frame, thereby determining the position of a first identification object and the state of a second identification object. Based on the position of the first identification object, it determines the target cropping position, which is the center position of cropping when cropping the target image frame. Based on the state of the second identification object, it determines the target cropping size, which is the cropping size when cropping the target image frame. Based on the target cropping position and target cropping size, it crops the image from the target image frame and scales it to a preset size. By determining the cropping position and cropping size of the image frame and cropping a cropped image from the image frame that simulates the lens movement effect, it is possible to avoid the cumbersome manual lens movement process, enrich the appeal of the screen, and improve the user experience. It solves the problem that the lens movement effect is not ideal due to technical or conditional limitations, and the problem that the operation process becomes complicated and time-consuming when simulating lens movement by screen editing.

[0030] Figure 2a is a flowchart of another image processing method according to an embodiment of the present disclosure, which further optimizes the above-mentioned embodiments and can be combined with each of the selectable technical options in one or more of the above embodiments. As shown in Figure 2a, the image processing method according to an embodiment of the present disclosure may include the following steps S210 to S250.

[0031] In S210, the position of the target region obtained by keypoint detection for the target region within the target image frame is identified from the target image frame.

[0032] Herein, the proposed technology can identify the position of a target part from a target image frame based on a general keypoint detection model, and the position of the target part may be at least one of the head position, hand position, foot position, groin position, and shoulder position of the identified object in the target image frame.

[0033] As a selective but non-limiting implementation, the above-mentioned identification of the position of the target portion from the target image frame may further include, but is not limited to, a process of predicting the position and confidence level of the target portion in the target image frame after completing keypoint detection for the target portion in the previous image frame, wherein the previous image frame and the target image frame are two temporally adjacent image frames.

[0034] Here, as shown in Figure 2b, this proposed technology employs a general keypoint detection model to sequentially detect keypoints for each image frame in accordance with the chronological order of the image frames. Specifically, this proposed technology employs a keypoint detection model to detect the position of the target area in the image frame preceding the target image frame, predict the confidence level for each target area's position, and determine the position of the target area in the image frame whose confidence level satisfies the requirements, thereby completing keypoint detection for the target area in the previous image frame. Subsequently, it is possible to detect the position of the target area in the target image frame, predict the confidence level for each target area's position, determine the position of the target area in the target image frame whose confidence level satisfies the requirements, and determine the reliable target area's position in each image frame, thereby providing a reliable data base for determining the cropping position of each subsequent image frame.

[0035] In S220, the position of the first identification object is determined based on the position of the target area. The position of the first identification object is determined based on the position of the hand portion and the center of the body of the first identification object among the positions of the target area. The center of the body is determined based on the positions of other parts other than the hand portion.

[0036] Here, the first identification object may be an object appearing in the target image frame, and there is at least one first identification object. For each first identification object, the present invention can determine the position of the first identification object based on the position of the target area determined in the steps described above. The position of the first identification object can be determined by the position of the hand or the center of the body of the first identification object among the positions of the target area. The center of the body can be determined by the position of the head, feet, hips, and shoulders, or other positions of the target area other than the hand. This makes it possible to provide a reliable data source for the subsequent step of determining the target cropping position in the target image frame.

[0037] As a selective but non-limiting implementation, the position of the first identification object includes a first reference center position and a second reference center position in the target image frame, wherein the first reference center position is a hand center position determined based on the hand position among the positions of the target part, and the second reference center position is a center position determined based on the positions of at least two other parts of the target part other than the hand position, or The position of the first identification object includes a second reference center position in the target image frame, the second reference center position being a center position determined based on the positions of at least two other parts of the target area other than the hand position.

[0038] Here, as shown in Figure 2b, if the hand position is included among the reliable target area positions detected in the target image frame of this proposed technology, the position of the first identified object may include a hand center position determined based on the hand position in the target image frame, i.e., a first reference center position, and a center position determined based on the positions of at least two other target areas other than the hand position, i.e., a second reference center position.

[0039] In this proposed technology, if the hand position is not included among the reliable target area positions detected in the target image frame, the position of the first identified object may include a central position determined based on the positions of at least two other target areas other than the hand position, i.e., a second reference center position. This makes it possible to provide a reliable data base for determining the target cropping position in the target image frame.

[0040] As a selective but non-restrictive implementation, the first reference center position is the position of the center point of the line connecting the left and right hands in the target image frame, which is determined based on the positions of the left and right hands of the first identified object among the positions of the target portion. The position of the target portion includes at least one of the positions of the center point of the line connecting the head and hands in the target image frame, which is determined based on the position of the head and the first reference position, which is the position of the left hand or the right hand of the first identification object.

[0041] In this proposed technology, if the position of the target portion in the target image frame includes the position of the left hand and the position of the right hand of the first identification object, the first reference center position may be the position of the center point of the line connecting the left and right hands, which is determined by the position of the left hand and the position of the right hand of the first identification object.

[0042] In this proposed technology, if the position of the target area in the target image frame includes only the position of the left hand or the right hand of the first identified object, and the position of the target area includes the position of the head of the first identified object, the first reference center position may be the center point of the line connecting the head and the hand, determined by the position of the left hand and the head of the first identified object. Alternatively, the first reference center position may be the center point of the line connecting the head and the hand, determined by the position of the right hand and the head of the first identified object. This makes it possible to provide a reliable data base for determining the target cropping position in the target image frame.

[0043] As a selective but non-limiting implementation, steps A1 to A2 may be further included before determining the position of the first identified object based on the position of the target portion, but are not limited to these steps.

[0044] Step A1: Determine the difference in hand position between the target image frame and the second reference image frame, the difference in hand position being determined based on the difference between the hand position in the target image frame and the hand position in the second reference image frame, the second reference image frame being an adjacent image frame that is temporally positioned before the target image frame.

[0045] In this proposed technology, the position of the hand in the target image frame and the position of the hand in an adjacent image frame that is temporally preceding the target image frame are determined, and then the difference between the position of the hand in the target image frame and the position of the hand in the second reference image frame is determined. The ratio of this difference to the playback time interval between two temporally adjacent image frames can then be used as the difference between the hand positions in the target image frame and the second reference image frame.

[0046] Step A2: Based on the difference in hand position, adjust the hand position of the first identification object among the target area positions.

[0047] Here, this proposed technology can adaptively adjust the position of the hand of the first identified object among the positions of the target part in the target image frame based on the difference in hand position, thereby bringing the hand position of the first identified object closer in two adjacent image frames. This enables a smooth transition of the movement of the first identified object in the cropped images of two adjacent image frames and avoids jitter.

[0048] As a selective but non-limiting implementation, adjusting the hand position of the first identified object among the target part positions based on the difference in the hand position may further include, but is not limited to, steps B1 to B2.

[0049] Step B1: If the difference in hand position is greater than the threshold for the difference in preset positions, the hand position of the first identified object among the positions of the target area is adjusted by taking a weighted average of the hand position in the target image frame and the hand position in the second reference image frame.

[0050] Here, assuming that the hand position in the target image frame is (x1, y1) and the hand position in the second reference image frame is (x2, y2), if the difference in hand positions is greater than the threshold for the difference in preset positions, it indicates that the distance between the hand positions in two adjacent image frames is too large, and the likelihood of jitter occurring during video output increases. In this proposed technique, the weighted average of the hand position in the target image frame and the hand position in the second reference image frame ((x1+x2) / 2, (y1+y2) / 2) can be used to adjust the hand position of the first identified object among the positions of the target area.

[0051] Step B2: If the difference in hand position is less than or equal to the threshold of the difference in preset positions, the hand position of the first identification object among the target area positions is not adjusted.

[0052] If the difference in hand position is below the threshold for the difference in preset positions, it indicates that the distance between the hand positions in two adjacent image frames is small, reducing the likelihood of jitter during video output, and eliminating the need to adjust the hand position of the first identified object among the target area positions. This enables a smooth transition of the movement of the first identified object in the cropped images of two adjacent image frames, thus avoiding jitter.

[0053] In S230, the state of the second identification object is determined based on the position of the target area, and the state of the second identification object is determined based on the positions of the left and right hands of the second identification object among the positions of the target area.

[0054] The second identification object may be an object appearing in the target image frame, and there is at least one second identification object. For each second identification object, the present invention can determine the state of the second identification object based on the position of the left hand and the position of the right hand of the second identification object, which are among the target area positions determined in the steps described above. This makes it possible to provide a reliable data source for the subsequent step of determining the target cropping size in the target image frame.

[0055] As a selective but non-limiting implementation, determining the state of a second identification object based on the position of the target portion may further include, but is not limited to, steps C1 to C2.

[0056] Step C1: From the positions of the target area, a second reference position of the second identification object is determined, and the second reference position includes the position of the left hand and the position of the right hand.

[0057] Here, this proposed technology can determine the positions of the left and right hands of a second identification object, i.e., the second reference position, from among the positions of the target portion of the target image frame.

[0058] Step C2: Based on the second reference position, the state of the second identification object is determined, and the state of the second identification object is determined based on the distance between the left and right hand positions of the second identification object.

[0059] In this proposed technology, the distance between the left and right hands of a second identification object is determined by the positions of the left and right hands, and then the state of the second identification object can be determined based on this distance. This provides a reliable data base for determining the target cropping size of the target image frame.

[0060] In S240, the target cropping position is determined based on the position of the first identified object, and the target cropping size is determined based on the state of the second identified object.

[0061] As a selective but non-limiting implementation, determining the target cropping position based on the position of the first identified object may further include, but is not limited to, steps D1-D2.

[0062] Step D1: Determine the relative displacement between the first reference center position and the second reference center position of the first identification object.

[0063] In this proposed technology, the relative displacement can be determined by the first and second reference center positions of the first identification object, and this relative displacement includes the offset distance and offset direction of the first reference center position relative to the second reference center position.

[0064] Step D2: Determine the target cropping position corresponding to the target image frame based on the relative displacement.

[0065] Here, this proposed technology can determine the target cropping position corresponding to the target image frame based on relative displacement. It can provide a reliable data base for determining the cropping position in the target image frame.

[0066] As a selective but non-limiting implementation, determining the target cropping position corresponding to the target image frame in accordance with the relative displacement may further include, but is not limited to, steps E1 to E2.

[0067] Step E1: If the relative displacement is detected to be greater than the preset displacement threshold, the target cropping position corresponding to the target image frame is determined based on the reference cropping position of the target image frame and the relative displacement.

[0068] Here, the preset displacement threshold can be adaptively determined according to the size of the first identified object. In this proposed technique, if the relative displacement is greater than the preset displacement threshold, the target cropping position corresponding to the target image frame is determined based on the reference cropping position of the target image frame and the relative displacement. For example, the distance of movement is determined based on the relative displacement, and then, after moving by the distance in the direction of the relative displacement at the reference cropping position of the target image frame, the target cropping position corresponding to the target image frame is obtained.

[0069] The reference cropping position of the target image frame includes at least one of the following: the center point of the line connecting the left hand position and the right hand position in the first reference image frame; the center point of the line connecting the head position and the hand position in the first reference image frame; and the cropping center position adopted when cropping the image from the second reference image frame. The first reference image frame is an image frame that is temporally located before the target image frame, and the second reference image frame is an adjacent image frame that is temporally located before the target image frame. It is possible to determine the cropping position of the target image frame based on the cropping position of the image frame preceding the target image frame, thereby achieving a smooth transition of the operation of the first identified object in the cropped images of two adjacent image frames and avoiding jitter.

[0070] Step E2: If it is detected that the relative displacement is less than or equal to the preset displacement threshold, the reference cropping position of the target image frame is determined as the target cropping position corresponding to the target image frame.

[0071] Here, the reference cropping position of the target image frame includes at least one of the following: the center point position of the line connecting the left and right hand positions in the first reference image frame; the center point position of the line connecting the head position and the hand position in the first reference image frame; and the cropping center position adopted when cropping an image from the second reference image frame, wherein the first reference image frame is an image frame located temporally before the target image frame, and the second reference image frame is an adjacent image frame located temporally before the target image frame.

[0072] Specifically, in this proposed technology, if a relative displacement is detected to be below a preset displacement threshold, the reference cropping position of the target image frame can be set to the target cropping position corresponding to the target image frame. This enables a smooth transition of the movement of the first identification object in the cropped images of two adjacent image frames, thereby avoiding jitter.

[0073] As a selective but non-limiting implementation, determining the target cropping position based on the position of the first identification object may further include, but are not limited to, the process of determining the second reference center position as the target cropping position corresponding to the target image frame.

[0074] In this proposed technology, if it is determined that a reliable hand position is not included among the target part positions of the first identified object in the target image frame, the second reference center position can be determined as the target cropping position corresponding to the target image frame. This provides a reliable data base for realizing the effect of simulating lens movement.

[0075] As a selective but non-limiting implementation, determining the target cropping size based on the state of the second identified object may further include, but is not limited to, steps F1-F2.

[0076] Step F1: If the distance between the left and right hands, as indicated by the state of the second identification object, is less than a preset distance threshold, the cropping size to be used when cropping the image in the target image frame is adjusted from the first cropping size to the second cropping size, where the first cropping size is greater than the second cropping size.

[0077] In this proposed technology, the distance between the left and right hands, as indicated by the state of the second identification object, can be determined based on the positional difference between the left and right hand positions of the second identification object, which are among the target area positions determined in the steps described above. In this proposed technology, if the distance between the left and right hands is determined to be less than a preset distance threshold, the cropping size to be adopted when cropping the image in the target image frame is adjusted from the first cropping size to the second cropping size. The first cropping size may be the cropping size adopted by the image frame that was temporally adjacent to the target image frame. The first cropping size should be larger than the second cropping size. The difference between the first cropping size and the second cropping size is the preset cropping step size. Gradual zooming of the lens can be achieved, and jitter in lens motion videos can be avoided.

[0078] Step F2: If the distance between the left and right hands, as indicated by the state of the second identification object, is greater than or equal to a preset distance threshold, the cropping size to be adopted when cropping the image in the image frame is adjusted from the first cropping size to the third cropping size, where the third cropping size is greater than the first cropping size.

[0079] In this proposed technology, when the distance between the left and right hands is determined to be greater than or equal to a preset distance threshold, the cropping size to be used when cropping the image in the target image frame is adjusted from a first cropping size to a third cropping size. The third cropping size should be larger than the first cropping size, and the difference between the third cropping size and the first cropping size is the preset cropping step size. This enables progressive zooming of the lens and avoids jitter in lens motion videos.

[0080] As a selective but non-restrictive implementation, the adjustment ratio of the cropping size used when cropping an image in two temporally adjacent image frames is a preset ratio value, and this preset ratio value determines the scaling speed when the images cropped in correspondence between the two adjacent image frames are displayed consecutively.

[0081] In this proposed technology, the ratio of the cropping sizes of two temporally adjacent image frames can be set to a preset ratio value. This preset ratio value determines the scaling speed when the images cropped in correspondence between two adjacent image frames are displayed sequentially. This allows for adjustment of the lens motion effect when image frames are played back sequentially.

[0082] In S250, the image is cropped from the target image frame based on the target cropping position and target cropping size, and then scaled to a preset size.

[0083] As a selective but non-limiting implementation, the process may further include, but is not limited to, the process of generating a target image sequence based on images cropped from different image frames, after cropping an image from the target image frame and scaling it to a preset size, wherein the images cropped from different image frames in the target image sequence are the same size when displayed sequentially.

[0084] Specifically, as shown in Figure 2b, this proposed technology allows for the formation of a target image sequence by cropping images from different image frames in chronological order, and when each image in the target image sequence is displayed sequentially, it should be the same size, which may be a preset size or another size. It is possible to generate a video with a lens motion effect based on the cropped image of each image frame.

[0085] The technical solution of the embodiments of this disclosure identifies the position of a target area from a target image frame, the position of the target area is obtained by detecting keypoints relative to the target area in the target image frame, the position of a first identification object is determined based on the position of the target area, the position of the first identification object is determined based on the position of the hand and the center of the body of the first identification object among the positions of the target area, the center of the body is determined based on the positions of other parts other than the hand, the state of a second identification object is determined based on the position of the target area, the state of the second identification object is determined based on the positions of the left and right hands of the second identification object among the positions of the target area, the target cropping position, which is the center of cropping when cropping is performed on the target image frame, is determined based on the position of the first identification object, the target cropping size, which is the cropping size when cropping is performed on the target image frame, is determined based on the state of the second identification object, the image is cropped from the target image frame based on the target cropping position and target cropping size and scaled to a preset size. By determining the cropping position and size of the image frame and cropping a cropped image from the image frame to simulate the lens movement effect, it is possible to avoid the cumbersome manual lens movement process, enriching the visual appeal of the screen and improving the user experience. This solves the problem of lens movement effects not being ideal due to technical or conditional limitations, and the problem of the operation process becoming complex and time-consuming when simulating lens movement through screen editing.

[0086] Figure 3 is a flowchart of yet another image processing method according to an embodiment of the present disclosure, which further optimizes the above-mentioned embodiments and can be combined with each of the selectable technical options in one or more of the above embodiments. As shown in Figure 3, the image processing method according to an embodiment of the present disclosure may include the following steps S310 to S350.

[0087] In S310, the position of the first identified object and the state of the second identified object are determined by identifying the target image frame.

[0088] In S320, the target cropping position is determined based on the position of the first identified object, and the target cropping size is determined based on the state of the second identified object.

[0089] In S330, it is determined whether the cropping region in the target image frame exceeds the boundary of the target image frame. If it does, S340 is executed; otherwise, S350 is executed.

[0090] Here, the cropping region in the target image frame is determined based on the target cropping position and the target cropping size.

[0091] Here, as shown in Figure 2b, in this proposed technology, the cropping region of the target image frame is determined based on the target cropping position and target cropping size. Then, it is possible to determine whether the cropping region of the target image frame extends beyond the boundary of the target image frame by comparing the position coordinates of the cropping region with the position coordinates of the boundary of the target image frame. Alternatively, it is possible to determine whether the cropping region of the target image frame extends beyond the boundary of the target image frame by checking whether the area of ​​the common portion between the cropping region and the target image frame matches the area of ​​the cropping region.

[0092] In S340, the target cropping position in the target image frame is adjusted, and based on the target cropping position and target cropping size at the end of the adjustment, the image is cropped from the target image frame, and the cropping region corresponding to the image cropped based on the target cropping position at the end of the adjustment is modified so that it no longer exceeds the boundary of the target image frame.

[0093] In this proposed technology, if it is determined that the cropping region of the target image frame exceeds the boundary of the target image frame, a black border should be generated after cropping the target image frame to correct the value that exceeds the boundary of the target image frame. This allows for adjustment of the target cropping position in the target image frame. For example, x and y in the cropping box (x, y, w, h) determined by the target cropping position and target cropping size can be restricted, while w and h can be kept unchanged. The target cropping position and target cropping size at the end of the adjustment are obtained, and the image is cropped from the target image frame based on the target cropping position and target cropping size at the end of the adjustment. The area of ​​the intersection of the cropping region corresponding to the cropped image and the target image frame is exactly the area of ​​the cropping region. That is, the cropping region corresponding to the image cropped based on the target cropping position at the end of the adjustment is modified so that it no longer exceeds the boundary of the target image frame. This ensures the integrity of the cropped image and provides a reliable data source for displaying lens motion videos.

[0094] In S350, the image is cropped from the target image frame based on the target cropping position and target cropping size, and then scaled to a preset size.

[0095] In this proposed technology, if it is determined that the cropping region in the target image frame does not extend beyond the boundary of the target image frame, the image can be directly cropped from the target image frame based on the target cropping position and target cropping size, and the resulting image can be scaled to a preset size. This provides a reliable data base for realizing the effect of simulating lens movement when cropped images in temporally adjacent image frames are displayed sequentially.

[0096] As a selective but non-limiting implementation, steps G1 to G2 may be further included before cropping the image from the target image frame based on the target cropping position and the target cropping size, but are not limited to these.

[0097] Step G1: Determine the completeness of the face in the cropping region of the target image frame based on the target cropping position, and the completeness of the face is used to represent the proportion of the face located in the cropping region relative to a complete face.

[0098] Here, as shown in Figure 2b, in this proposed technology, before cropping the image from the target image frame based on the target cropping position and target cropping size, it is necessary to further verify the proportion of faces relative to the complete face in the cropped area determined based on the target cropping position and target cropping size, i.e., the completeness of the face. This can provide a reliable data base to ensure that as many faces as possible are included in the cropped area.

[0099] Step G2: If the integrity of the face is less than the preset integrity threshold, adjust the target cropping position on the target image frame so that the integrity of the face does not fall below the preset integrity threshold.

[0100] In this proposed technology, if the completeness of a face is determined to be less than a preset completeness threshold, the region of the face that falls within the cropping area is indicated as incomplete, resulting in a poor appearance due to face cropping and affecting the display of lens motion effects. To prevent the completeness of a face from falling below the preset completeness threshold, the target cropping position on the target image frame is adaptively adjusted, thereby ensuring that the face is within the lens. This provides a reliable data base for ensuring the display of lens motion effects.

[0101] As a selective but non-limiting implementation, steps H1-H2 may be further included before cropping the image from the target image frame based on the target cropping position and target cropping size, but are not limited to these steps.

[0102] Step H1: Determine the difference in cropping positions between the target image frame and the second reference image frame. This difference in cropping positions is determined based on the difference between the cropping center position when cropping the target image frame and the cropping center position when cropping the second reference image frame, where the second reference image frame is an adjacent image frame that is temporally located before the target image frame.

[0103] Here, as shown in Figure 2b, in this proposed technology, the cropping center position in the target image frame and the cropping center position in an adjacent image frame (i.e., a second reference image frame) that is temporally prior to the target image frame can be determined, and based on the cropping position of the adjacent image frame, the difference in cropping positions between the target image frame and the second reference image frame can be determined.

[0104] Step H2: Based on the difference in the cropping positions, adjust the cropping center position used when cropping the target image frame.

[0105] Specifically, in this proposed technology, if it is determined that the difference in cropping positions exceeds the preset movement displacement, it indicates that the difference in cropping center positions of adjacent image frames is too large, resulting in jitter in the final lens motion video. A weighted average of the cropping center position when cropping the second reference image frame and the cropping center position when cropping the target image frame is calculated to obtain a new cropping center position for the target image frame. Based on this new cropping center position, the cropping center position when cropping the target image frame can be adjusted, reducing the movement speed of the cropping box and preventing jitter in the lens motion video. In this proposed technology, if it is determined that the difference in cropping positions does not exceed the preset movement displacement, there is no need to adjust the cropping center position.

[0106] The technical invention of the embodiment of this disclosure determines the position of a first identification object and the state of a second identification object by identifying a target image frame; determines the target cropping position, which is the center position of cropping when cropping is performed on the target image frame, based on the position of the first identification object; determines the target cropping size, which is the size of cropping when cropping is performed on the target image frame, based on the state of the second identification object; determines whether the cropping region in the target image frame exceeds the boundary of the target image frame; the cropping region in the target image frame is determined based on the target cropping position and target cropping size; if it exceeds the boundary of the target image frame, the target cropping position in the target image frame is adjusted; the image is cropped from the target image frame based on the target cropping position at the end of the adjustment, so that the cropping region corresponding to the image cropped based on the target cropping position at the end of the adjustment no longer exceeds the boundary of the target image frame; if it does not exceed the boundary of the target image frame, the image is cropped from the target image frame based on the target cropping position and target cropping size and scaled to a preset size. By determining the cropping position and size of the image frame and cropping a cropped image from the image frame to simulate the lens movement effect, it is possible to avoid the cumbersome manual lens movement process, enriching the visual appeal of the screen and improving the user experience. This solves the problem of lens movement effects not being ideal due to technical or conditional limitations, and the problem of the operation process becoming complex and time-consuming when simulating lens movement through screen editing.

[0107] Figure 4 is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure, and as shown in Figure 4, the apparatus is An image frame identification module 410 for determining the position of a first identification object and the state of a second identification object by identifying a target image frame, A cropping position and cropping size determination module 420 for determining the target cropping position, which is the cropping center position when cropping is performed on the target image frame, based on the position of the first identification object, and for determining the target cropping size, which is the cropping size when cropping is performed on the target image frame, based on the state of the second identification object, The system includes an image cropping module 430 for cropping an image from a target image frame and scaling it to a preset size based on the target cropping position and the target cropping size.

[0108] In one selectable technical example of the embodiments of the present disclosure, the apparatus optionally further includes a target image sequence generation module for generating a target image sequence based on images cropped from different image frames, after cropping an image from the target image frame and scaling it to a preset size, wherein the images cropped from different image frames in the target image sequence are the same size when displayed sequentially.

[0109] In one selectable technical example of the embodiments of the present disclosure, the image frame identification module 410 optionally includes: a target area position identification unit for identifying the position of a target area obtained from a target image frame based on detecting keypoints for a target area in the target image frame; a first identification object position determination unit for determining the position of a first identification object based on the position of the target area, wherein the position of the first identification object is determined based on the hand position and body center position of the first identification object among the positions of the target area, and the body center position is determined based on the positions of other parts other than the hand position; and a second identification object state determination unit for determining the state of a second identification object based on the position of the target area, which is determined based on the left and right hand positions of the second identification object among the positions of the target area.

[0110] In one selectable technical example of the embodiments of the present disclosure, the target location identification unit is, specifically, for predicting the location and confidence of a target location in a target image frame after keypoint detection for the target location in the previous image frame has been completed, wherein the previous image frame and the target image frame are two temporally adjacent image frames.

[0111] In one selectable technical example of the embodiments of the present disclosure, the location of the first identification object includes a first reference center location and a second reference center location in the target image frame, wherein the first reference center location is a hand center location determined based on the hand location of the target portion, and the second reference center location is a center location determined based on the locations of at least two other portions of the target portion other than the hand location, or the location of the first identification object includes a second reference center location in the target image frame, wherein the second reference center location is a center location determined based on the locations of at least two other portions of the target portion other than the hand location.

[0112] In one selectable technical example of the embodiments of the present disclosure, the first reference center position is selectable to include at least one of the following: the position of the center point of a line connecting the left and right hands in the target image frame, which is determined based on the position of the left hand and the position of the right hand of the first identified object among the positions of the target portion; and the position of the center point of a line connecting the head and hands in the target image frame, which is determined based on the first reference position and the head position of the first identified object among the positions of the target portion, wherein the first reference position is the position of the left hand or the position of the right hand.

[0113] In one selectable technical example of the embodiments of the present disclosure, the cropping position and cropping size determination module 420 optionally includes a relative displacement determination module for determining the relative displacement between the first reference center position and the second reference center position of the position of the first identified object, and a target cropping position determination unit for determining the target cropping position corresponding to the target image frame based on the relative displacement.

[0114] In one selectable technical example of the embodiments of the present disclosure, the target cropping position determination unit is selected to determine a target cropping position corresponding to the target image frame based on the reference cropping position of the target image frame and the relative displacement when it is detected that the relative displacement is greater than a preset displacement threshold, and to determine the reference cropping position of the target image frame as the target cropping position corresponding to the target image frame when it is detected that the relative displacement is less than or equal to a preset displacement threshold. Here, the reference cropping position of the target image frame includes at least one of the center point position of the line connecting the left and right hand positions in a first reference image frame, the center point position of the line connecting the head position and hand position in a first reference image frame, and a cropping center position adopted when cropping an image from a second reference image frame, wherein the first reference image frame is an image frame located temporally before the target image frame, and the second reference image frame is an adjacent image frame located temporally before the target image frame.

[0115] In one selectable technical example of the embodiments of the present disclosure, the cropping position and cropping size determination module 420 is optionally used to determine the second reference center position as a target cropping position corresponding to the target image frame.

[0116] In one selectable technical example of the embodiments of the present disclosure, the apparatus optionally includes a hand position difference determination unit for determining the difference between the hand position in a target image frame and a second reference image frame before determining the position of a first identified object based on the position of the target portion, wherein the hand position difference is determined based on the difference between the hand position in the target image frame and the hand position in the second reference image frame, the second reference image frame being an adjacent image frame that is temporally located before the target image frame; and a hand position adjustment unit for adjusting the hand position of the first identified object among the positions of the target portion based on the hand position difference.

[0117] In one selectable technical example of the embodiments of the present disclosure, the hand position adjustment unit is selectively used to adjust the hand position of the first identified object among the positions of the target area by taking a weighted average of the hand position in the target image frame and the hand position in the second reference image frame when the difference in the hand position is greater than a threshold of the difference in preset positions, and not to adjust the hand position of the first identified object among the positions of the target area when the difference in the hand position is less than or equal to a threshold of the difference in preset positions.

[0118] In one selectable technical example of the embodiments of the present disclosure, the second identification object state determination unit optionally includes a second reference position determination subunit for determining a second reference position of the second identification object from among the positions of the target portion, wherein the second reference position includes a left hand position and a right hand position; and a second identification object state determination subunit for determining the state of the second identification object based on the second reference position, wherein the state of the second identification object is determined based on the distance between the left and right hands of the second identification object.

[0119] In one selectable technical example of the embodiments of the present disclosure, the cropping position and cropping size determination module 420 selectively adjusts the cropping size to be adopted when cropping the image in the target image frame from a first cropping size to a second cropping size if the distance between the left and right hands, as indicated by the state of the second identification object, is less than a preset distance threshold, and the first cropping size is greater than the second cropping size, and adjusts the cropping size to be adopted when cropping the image in the image frame from a first cropping size to a third cropping size if the distance between the left and right hands, as indicated by the state of the second identification object, is greater than or equal to a preset distance threshold, and the third cropping size is used because it is greater than the first cropping size.

[0120] In one selectable technical example of the embodiments of this disclosure, the cropping size adjustment ratio used when cropping an image in two temporally adjacent image frames is a preset ratio value, and the preset ratio value determines the scaling speed when the images cropped in correspondence to the two adjacent image frames are displayed consecutively.

[0121] In one selectable technical example of the embodiments of this disclosure, the image cropping module 430 optionally includes a determination unit for determining whether a cropped region in the target image frame extends beyond the boundary of the target image frame, wherein the cropped region in the target image frame is determined based on the target cropping position and the target cropping size, The system includes a target cropping position adjustment unit that, when the boundary of the target image frame is exceeded, adjusts the target cropping position in the target image frame, crops the image from the target image frame based on the target cropping position and target cropping size at the end of the adjustment, and modifies the cropping region corresponding to the image cropped based on the target cropping position at the end of the adjustment so that it does not exceed the boundary of the target image frame; and an image cropping unit that, when the boundary of the target image frame is not exceeded, crops the image from the target image frame based on the target cropping position and target cropping size.

[0122] In one selectable technical example of the embodiments of the present disclosure, the apparatus optionally further includes a face integrity determination module for determining the integrity of faces in the cropping region of the target image frame based on the target cropping position, before cropping an image from the target image frame based on the target cropping position and the target cropping size, wherein the face integrity is used to represent the proportion of faces located in the cropping region to complete faces, and if the face integrity is less than a preset integrity threshold, the apparatus adjusts the target cropping position on the target image frame so that the face integrity does not fall below the preset integrity threshold.

[0123] In one selectable technical application of the embodiments of the present disclosure, the apparatus optionally includes a cropping center position adjustment module used for adjusting the cropping center position adopted when cropping the target image frame, based on the target cropping position and the target cropping size, before cropping the image from the target image frame, wherein the difference in cropping position is determined based on the difference between the cropping center position when cropping the target image frame and the cropping center position when cropping the second reference image frame, wherein the second reference image frame is an adjacent image frame that is temporally located before the target image frame, and for adjusting the cropping center position adopted when cropping the target image frame based on the difference in cropping position.

[0124] The image processing apparatus according to the embodiments of the present disclosure can perform the image processing method according to any embodiment of the present disclosure and has corresponding functional modules and beneficial effects for performing the method.

[0125] It should be noted that the units and modules included in the above-mentioned device are merely categorized based on functional logic, and are not limited to the above categories as long as they can realize the corresponding function. Furthermore, the specific names of each functional unit are merely for convenience of classification and are not intended to limit the scope of protection of the embodiments of this disclosure.

[0126] Figure 5 is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Hereinafter, with reference to Figure 5, we will show a schematic diagram of the structure of an electronic device (e.g., a terminal device or server in Figure 5) 500 suitable for implementing an embodiment of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile devices such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable media players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 5 is merely an example and does not in any way limit the functions and scope of use of the embodiments of the present disclosure.

[0127] As shown in Figure 5, the electronic device 500 may include a processing unit (e.g., a central processing unit, an image processing unit, etc.) 501, which can perform various appropriate operations and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data necessary for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0128] Generally, devices such as input devices 506 including touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices 507 including liquid crystal displays (LCDs), speakers, and vibrators; storage devices 508 including magnetic tape and hard disks; and communication devices 509 can be connected to the I / O interface 505. The communication device 509 can enable the electronic device 500 to exchange data with other devices via wireless or wired communication. Figure 5 shows an electronic device 500 with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have been included as alternatives.

[0129] In particular, according to embodiments of the present disclosure, the process described with reference to the flowchart above may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product which includes a computer program contained on a non-temporary computer-readable medium, the computer program which includes program code for performing the method shown in the flowchart. In such embodiments, the computer program may be downloaded and installed from a network by a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-described functions, limited to the method of embodiments of the present disclosure, are performed.

[0130] The names of messages or information that interact between multiple devices in embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0131] The electronic devices according to the embodiments of this disclosure belong to the same inventive concept as the image processing method according to the above embodiments, and technical details not described in detail in these embodiments can be referenced to the above embodiments, and these embodiments have the same beneficial effects as the above embodiments.

[0132] The embodiments of this disclosure provide a computer storage medium in which a computer program is stored, and when the program is executed by a processor, the image processing method according to the embodiments is realized.

[0133] The computer-readable media described in this disclosure may be computer-readable signal media, computer-readable storage media, or any combination thereof. Computer-readable storage media may be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable pail computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable pail compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, computer-readable storage media may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, computer-readable signal media may include data signals propagated in the baseband or as part of a carrier wave, which contain computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, and such computer-readable signal medium can transmit, propagate, or transmit programs used by or in combination with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted on any suitable medium, including but not limited to electric wires, optical cables, RF (radio frequency), or any suitable combination thereof.

[0134] In some embodiments, clients and servers can communicate using any currently known or future-to-be-developed network protocol, such as HTTP (HyperText Transfer Protocol), and can interconnect (e.g., communication networks) to communicate with digital data in any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), and networks that are currently known or future-to-be-developed.

[0135] The computer-readable medium described above may be included in the electronic device described above, or it may exist independently without being incorporated into the electronic device.

[0136] The computer-readable medium contains one or more programs, and when these programs are executed by the electronic device, the electronic device identifies a target image frame to determine the position of a first identified object and the state of a second identified object. Based on the position of the first identification object, the target cropping position, which is the cropping center position when cropping is performed on the target image frame, is determined, and based on the state of the second identification object, the target cropping size, which is the cropping size when cropping is performed on the target image frame, is determined. Based on the target cropping position and target cropping size, the image is cropped from the target image frame and scaled to a preset size.

[0137] Computer program code for performing the operations of the Disclosure can be organized in one or more programming languages ​​or a combination thereof, and such programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, and also include ordinary procedural programming languages ​​such as the C language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a standalone software package, partially on the user's computer and partially on a remote computer, or fully on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, connected via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the drawings illustrate the implementable architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code containing one or more executable instructions for implementing a given logical function. In some alternative implementations, the functions attached to a block may be implemented in a different order than that shown in the drawings. For example, two consecutively shown blocks may actually be executed essentially in parallel, or in reverse order depending on the function. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be implemented in a dedicated hardware-based system that performs a given function or operation, or in a combination of dedicated hardware and computer instructions.

[0139] The units described in the embodiments of this disclosure may be implemented in software form or in hardware form. Herein, the names of the units are not limited in some cases; for example, the first acquisition unit may be described as "a unit that acquires at least two Internet Protocol addresses."

[0140] In this specification, the functions described above can be performed by at least partially one or more hardware logic components. For example, typical types of hardware logic components that can be used include, but are not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), and composite programmable logic circuits (CPLDs).

[0141] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or stores a program used in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the above contents. More specific examples of machine-readable storage media include electrical connections by one or more leads, portable pail computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable pail compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above contents.

[0142] According to one or more embodiments of this disclosure, Example 1 provides an image processing method, the method is By identifying the target image frame, the position of the first identified object and the state of the second identified object are determined. Based on the position of the first identification object, the target cropping position, which is the center position of cropping when cropping is performed on the target image frame, is determined, and based on the state of the second identification object, the target cropping size, which is the cropping size when cropping is performed on the target image frame, is determined. This includes cropping an image from the target image frame based on the target cropping position and the target cropping size, and scaling it to a preset size.

[0143] Example 2 is the same as the method in Example 1, and after cropping the image from the target image frame and scaling it to a preset size, The method further includes generating a target image sequence based on images cropped from different image frames, wherein the cropped images from different image frames in the target image sequence are the same size when displayed sequentially.

[0144] Example 3 is the method described in Example 1, and the position of the first identified object and the state of the second identified object are determined by identifying the target image frame as described above. The position of the target region is identified from the aforementioned target image frame based on the detection of keypoints for the target region within the target image frame, The position of the first identification object is determined based on the position of the target area, wherein the position of the first identification object is determined based on the position of the hand portion and the center position of the body of the first identification object among the positions of the target area, and the center position of the body is determined based on the positions of other parts other than the hand portion. This includes determining the state of the second identification object based on the position of the target portion, and based on the positions of the left and right hands of the second identification object among the positions of the target portion.

[0145] Example 4 is the method described in Example 3, and the above-mentioned method of identifying the position of the target area from the target image frame is The process further includes, after keypoint detection for the target region in the previous image frame is completed, predicting the position and confidence level for the target region in the target image frame, wherein the previous image frame and the target image frame are two temporally adjacent image frames.

[0146] Example 5 is the method described in Example 3, wherein the position of the first identification object includes a first reference center position and a second reference center position in the target image frame, the first reference center position being a hand center position determined based on the hand position among the positions of the target part, and the second reference center position being a center position determined based on the positions of at least two other parts of the target part other than the hand position, or The position of the first identification object includes a second reference center position in the target image frame, the second reference center position being a center position determined based on the positions of at least two other parts of the target area other than the hand position.

[0147] Example 6 is the method described in Example 5, and the first reference center position is The position of the center point of the line connecting the left and right hands in the target image frame, which is determined based on the position of the left hand and the position of the right hand of the first identification object, among the positions of the target area, The location of the target portion includes at least one of the following: the position of the center point of the line connecting the head and the hand in the target image frame, which is determined based on the first reference position and head position of the first identification object, wherein the first reference position is the position of the left hand or the position of the right hand.

[0148] Example 7 is the method described in Example 5, and determining the target cropping position based on the position of the first identified object is To determine the relative displacement between the first reference center position and the second reference center position of the first identification object, This includes determining the target cropping position corresponding to the target image frame based on the relative displacement.

[0149] Example 8 is the method described in Example 7, and determining the target cropping position corresponding to the target image frame based on the relative displacement is If the relative displacement is detected to be greater than the preset displacement threshold, the target cropping position corresponding to the target image frame is determined based on the reference cropping position of the target image frame and the relative displacement. If it is detected that the relative displacement is less than or equal to a preset displacement threshold, the reference cropping position of the target image frame is determined to be the target cropping position corresponding to the target image frame, and the above is included in the above. Here, the reference cropping position of the target image frame includes at least one of the following: the center point position of the line connecting the left and right hand positions in the first reference image frame; the center point position of the line connecting the head position and the hand position in the first reference image frame; and the cropping center position adopted when cropping an image from the second reference image frame, wherein the first reference image frame is an image frame located temporally before the target image frame, and the second reference image frame is an adjacent image frame located temporally before the target image frame.

[0150] Example 9 is the method described in Example 5, and determining the target cropping position based on the position of the first identified object is This includes determining the second reference center position as the target cropping position corresponding to the target image frame.

[0151] Example 10 is the method described in Example 3, and before determining the position of the first identification object based on the position of the target portion, The difference in hand position is determined based on the difference between the hand position in the target image frame and the hand position in the second reference image frame, which is an adjacent image frame located temporally before the target image frame. The method further includes adjusting the hand position of the first identification object among the positions of the target area based on the difference in hand position.

[0152] Example 11 is the method described in Example 10, and adjusting the hand position of the first identification object among the target part positions based on the difference in hand position is, If the difference in hand position is greater than the threshold for the difference in preset positions, the hand position of the first identified object among the positions of the target area is adjusted by adopting a weighted average of the hand position in the target image frame and the hand position in the second reference image frame. This includes not adjusting the hand position of the first identification object among the target area positions if the difference in hand position is less than or equal to a threshold of the difference in preset positions.

[0153] Example 12 is the method described in Example 3, and determining the state of the second identification object based on the position of the target portion is The process involves determining a second reference position of a second identification object from among the positions of the target area, wherein the second reference position includes the position of the left hand and the position of the right hand. The method includes determining the state of a second identification object based on the second reference position, wherein the state of the second identification object is determined based on the distance between the left hand position and the right hand position of the second identification object.

[0154] Example 13 is the method described in Example 12, and determining the target cropping size based on the state of the second identified object is If the distance between the left and right hands, as indicated by the state of the second identification object, is less than a preset distance threshold, the cropping size to be adopted when cropping the image in the target image frame is adjusted from the first cropping size to the second cropping size, wherein the first cropping size is larger than the second cropping size. If the distance between the left and right hands, as indicated by the state of the second identification object, is greater than or equal to a preset distance threshold, the cropping size to be adopted when cropping the image in the image frame is adjusted from a first cropping size to a third cropping size, wherein the third cropping size is greater than the first cropping size.

[0155] Example 14 is the same as the method in Example 13, and the adjustment ratio of the cropping size used when cropping an image in two temporally adjacent image frames is a preset ratio value, and the preset ratio value determines the scaling speed when the images cropped in correspondence to the two adjacent image frames are displayed in succession.

[0156] Example 15 is the method described in Example 1, and cropping an image from the target image frame based on the target cropping position and the target cropping size is as follows: The process involves determining whether the cropping region in the target image frame extends beyond the boundary of the target image frame, wherein the cropping region in the target image frame is determined based on the target cropping position and the target cropping size. If the boundary of the target image frame is exceeded, the target cropping position in the target image frame is adjusted, and based on the target cropping position and target cropping size at the end of the adjustment, the image is cropped from the target image frame, and the cropping region corresponding to the image cropped based on the target cropping position at the end of the adjustment is modified so that it does not exceed the boundary of the target image frame. If the boundary of the target image frame is not exceeded, the image is cropped from the target image frame based on the target cropping position and the target cropping size.

[0157] Example 16 is the same as the method described in Example 1 or Example 15, and before cropping the image from the target image frame based on the target cropping position and the target cropping size, The integrity of faces in the cropping region of the target image frame is determined based on the target cropping position, and the integrity of faces is used to represent the proportion of faces located in the cropping region relative to complete faces. If the integrity of the face is less than a preset integrity threshold, the method further includes adjusting the target cropping position on the target image frame so that the integrity of the face does not fall below a preset integrity threshold.

[0158] Example 17 is the same as the method described in Example 1 or Example 15, and before cropping the image from the target image frame based on the target cropping position and the target cropping size, The difference in cropping positions is determined based on the difference between the cropping center position when cropping the target image frame and the cropping center position when cropping the second reference image frame, between the target image frame and a second reference image frame which is an adjacent image frame located temporally before the target image frame. The further includes adjusting the cropping center position used when performing cropping on the target image frame based on the difference in the aforementioned cropping positions.

[0159] According to one or more embodiments of the present disclosure, Example 18 further provides an image processing apparatus, the image processing apparatus, An image frame identification module for determining the position of a first identified object and the state of a second identified object by identifying a target image frame, A cropping position and cropping size determination module for determining the target cropping position, which is the cropping center position when cropping is performed on the target image frame, based on the position of the first identification object, and for determining the target cropping size, which is the cropping size when cropping is performed on the target image frame, based on the state of the second identification object, The system includes an image cropping module for cropping an image from a target image frame and scaling it to a preset size based on the target cropping position and the target cropping size.

[0160] According to one or more embodiments of the present disclosure, Example 19 further provides an electronic device, the electronic device being: One or more processors, Includes a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the image processing method described in any one of Examples 1 to 17.

[0161] According to one or more embodiments of the present disclosure, Example 20 further provides a storage medium that, when executed by a computer processor, includes computer-executable instructions for performing the image processing method described in any one of Examples 1 to 17.

[0162] The foregoing description is merely an explanation of preferred embodiments and the technical principles used in this disclosure. As will be understood by those skilled in the art, the scope of the disclosure is not limited to technical solutions consisting of specific combinations of the above technical features, but should also encompass other technical solutions formed by any combination of the above technical features or their equivalents, without departing from the concept of the disclosure. For example, it should encompass technical solutions formed by substituting the above features with (but not limited to) similar functional technical features disclosed herein.

[0163] Furthermore, while each operation is described in a specific order, this should not be understood as requiring these operations to be performed in a specific or sequential order as shown. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while the above considerations include several specific implementation details, these should not be construed as limitations on the scope of this disclosure. Some features described in the context of individual embodiments may be further combined and implemented in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented in multiple embodiments individually or in any suitable subcombination.

[0164] Although this subject matter has already been described using language that specifies structural features and / or methodological operations, it should be understood that the subject matter limited in the attached claims is not limited to the specific features or operations described above. Rather, the specific features and operations described above are merely illustrative forms that realize the claims.

Claims

1. An image processing method, By identifying the target image frame, the position of the first identified object and the state of the second identified object are determined. Based on the position of the first identification object, the target cropping position, which is the cropping center position when cropping is performed on the target image frame, is determined, and based on the state of the second identification object, the target cropping size, which is the cropping size when cropping is performed on the target image frame, is determined. An image processing method comprising cropping an image from a target image frame based on the target cropping position and the target cropping size, and scaling it to a preset size.

2. After cropping the image from the target image frame and scaling it to a preset size, the method proceeds as follows: The method according to claim 1, further comprising generating a target image sequence based on images cropped from different image frames, wherein the cropped images from different image frames in the target image sequence are the same size when displayed sequentially.

3. As described above, determining the position of the first identified object and the state of the second identified object by identifying the target image frame is: The position of the target region is identified from the aforementioned target image frame based on the detection of keypoints for the target region within the target image frame, Based on the position of the target area, the position of the first identification object is determined based on the position of the hand of the first identification object among the positions of the target area, and the position of the first identification object is determined based on the central position of the body determined based on the positions of other parts other than the hand. The method according to claim 1, further comprising determining the state of a second identification object based on the position of the target portion, and the position of the left and right hands of the second identification object among the positions of the target portion.

4. Identifying the position of the target area from the aforementioned target image frame is, The method according to claim 3, comprising predicting the position and confidence level of a target region in a target image frame after keypoint detection for a target region in a previous image frame is completed, wherein the previous image frame and the target image frame are two temporally adjacent image frames.

5. The position of the first identification object includes a first reference center position and a second reference center position in the target image frame, wherein the first reference center position is a hand center position determined based on the hand position among the positions of the target portion, and the second reference center position is a center position determined based on the positions of at least two other parts of the target portion other than the hand position, or The method according to claim 3, wherein the position of the first identification object includes a second reference center position in the target image frame, the second reference center position being a center position determined based on the positions of at least two other parts of the target part other than the hand part.

6. The first reference center position is, The position of the center point of the line connecting the left and right hands in the target image frame, which is determined based on the position of the left hand and the position of the right hand of the first identification object, among the positions of the target area, The method according to claim 5, wherein the position of the center point of the line connecting the head and the hand in the target image frame, which is determined based on the first reference position and the head position of the first identification object, among the positions of the target portion, the position of the center point of the line connecting the head and the hand in the target image frame, wherein the first reference position is the position of the left hand or the position of the right hand.

7. Determining the target cropping position based on the position of the first identification object is, To determine the relative displacement between the first reference center position and the second reference center position of the first identification object, The method according to claim 5, further comprising determining a target cropping position corresponding to the target image frame based on the relative displacement.

8. Determining the target cropping position corresponding to the target image frame based on the aforementioned relative displacement is: If the relative displacement is detected to be greater than the preset displacement threshold, the target cropping position corresponding to the target image frame is determined based on the reference cropping position of the target image frame and the relative displacement. If it is detected that the relative displacement is less than or equal to a preset displacement threshold, the reference cropping position of the target image frame is determined to be the target cropping position corresponding to the target image frame, and the above is included in the above. The method according to claim 7, wherein the reference cropping position of the target image frame includes at least one of the center point position of the line connecting the left and right hand positions in the first reference image frame, the center point position of the line connecting the head position and the hand position in the first reference image frame, and the cropping center position adopted when cropping an image from the second reference image frame, wherein the first reference image frame is an image frame located temporally before the target image frame, and the second reference image frame is an adjacent image frame located temporally before the target image frame.

9. Determining the target cropping position based on the position of the first identification object is, The method according to claim 5, comprising determining the second reference center position as a target cropping position corresponding to the target image frame.

10. Before determining the position of the first identification object based on the position of the target portion, the method, The difference in hand position is determined based on the difference between the hand position in the target image frame and the hand position in the second reference image frame, which is an adjacent image frame located temporally before the target image frame. The method according to claim 3, further comprising adjusting the hand position of the first identification object among the positions of the target portion based on the difference in hand position.

11. Based on the aforementioned difference in hand position, adjusting the hand position of the first identification object among the target area positions is: If the difference in hand position is greater than the threshold for the difference in preset positions, the hand position of the first identified object among the positions of the target area is adjusted by adopting a weighted average of the hand position in the target image frame and the hand position in the second reference image frame. The method according to claim 10, further comprising: if the difference in hand position is less than or equal to a threshold of the difference in preset positions, the hand position of the first identification object among the positions of the target area is not adjusted.

12. Determining the state of the second identification object based on the position of the target portion is, From the locations of the target area, a second reference position is determined, which includes the position of the left hand and the position of the right hand of the second identification object. The method according to claim 3, further comprising determining the state of a second identification object based on the second reference position, which is determined based on the distance between the left and right hands between the left and right hand positions of the second identification object.

13. Determining the target cropping size based on the state of the second identification object is, If the distance between the left and right hands, as indicated by the state of the second identification object, is less than a preset distance threshold, the cropping size to be adopted when cropping the image in the target image frame is adjusted from the first cropping size to the second cropping size, wherein the first cropping size is larger than the second cropping size. The method according to claim 12, further comprising: if the distance between the left hand and the right hand, as indicated by the state of the second identification object, is greater than or equal to a preset distance threshold, adjusting the cropping size to be adopted when cropping the image in the image frame from a first cropping size to a third cropping size that is larger than the first cropping size.

14. The method according to claim 13, wherein the adjustment ratio of the cropping size used when cropping an image in two temporally adjacent image frames is a preset ratio value, and the preset ratio value determines the scaling speed when the images cropped in correspondence with the two adjacent image frames are displayed in succession.

15. Cropping an image from the target image frame based on the aforementioned target cropping position and target cropping size is: The process involves determining whether the cropping region in the target image frame extends beyond the boundary of the target image frame, wherein the cropping region in the target image frame is determined based on the target cropping position and the target cropping size. If the boundary of the target image frame is exceeded, the target cropping position in the target image frame is adjusted, and based on the target cropping position and target cropping size at the end of the adjustment, the image is cropped from the target image frame, and the cropping region corresponding to the image cropped based on the target cropping position at the end of the adjustment is modified so that it does not exceed the boundary of the target image frame. The method according to claim 1, comprising cropping an image from the target image frame based on the target cropping position and the target cropping size, if the boundary of the target image frame is not exceeded.

16. Before cropping the image from the target image frame based on the target cropping position and target cropping size, the method: To determine the completeness of a face used to represent the proportion of a face located in the cropping region of the target image frame relative to a complete face, based on the target cropping position, The method according to claim 1 or 15, further comprising: adjusting the target cropping position on the target image frame so that the integrity of the face does not fall below a preset integrity threshold if the integrity of the face is less than a preset integrity threshold.

17. Before cropping the image from the target image frame based on the target cropping position and target cropping size, the method: The difference in cropping positions is determined based on the difference between the cropping center position when cropping the target image frame and the cropping center position when cropping the second reference image frame, between the target image frame and a second reference image frame which is an adjacent image frame located temporally before the target image frame. The method according to claim 1 or 15, further comprising adjusting the cropping center position used when performing cropping on the target image frame based on the difference in the cropping positions.

18. An image processing device, An image frame identification module configured to determine the position of a first identification object and the state of a second identification object by identifying a target image frame, A cropping position and cropping size determination module is configured to determine the target cropping position, which is the cropping center position when cropping is performed on the target image frame, based on the position of the first identification object, and to determine the target cropping size, which is the cropping size when cropping is performed on the target image frame, based on the state of the second identification object. An image processing apparatus comprising: an image cropping module configured to crop an image from a target image frame based on the target cropping position and the target cropping size, and to scale it to a preset size; and an image processing apparatus comprising: an image cropping module configured to crop an image from the target image frame and scale it to a preset size.

19. It is an electronic device, One or more processors, Includes a memory device for storing one or more programs, An electronic device that, when the one or more programs are executed by the one or more processors, causes the one or more processors to implement the image processing method described in any one of claims 1 to 17.

20. A storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the image processing method described in any one of claims 1 to 17.

Images