An image segmentation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image segmentation method and device, an electronic device and a storage medium. The method comprises: obtaining a movement parameter corresponding to each human body frame in a first global image, the movement parameter being used to represent a movement rule or a movement trend corresponding to the human body frame; for each human body frame, scaling the human body frame according to the movement parameter to obtain a target frame corresponding to the human body frame; determining a first target image according to the target frame corresponding to each human body frame and the first global image; and performing image segmentation on the first target image to obtain a human body segmentation result of the first global image. The present disclosure improves the image segmentation effect when the human body moves.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an image segmentation method and device, an electronic device and a storage medium. BACKGROUND

[0002] Background segmentation is an important problem in the field of computer vision and smart home. The background segmentation model can be used in many fields. For example, in a family entertainment scene, when multiple people interact, the portrait and the background can be segmented based on the background segmentation model, so that the monotonous background can be replaced. When performing background segmentation, a region of interest (for example, a region containing a portrait) is usually selected from a global image as a target image, and then image segmentation is performed on the target image.

[0003] In the above image segmentation process, due to the influence of human body movement, the portrait may leave the target image or the portrait segmentation details are poor, resulting in poor image segmentation effect. Therefore, how to improve the image segmentation effect when the human body moves is a problem to be solved at present. SUMMARY

[0004] The present disclosure provides a technical solution of an image segmentation method and device, an electronic device and a storage medium.

[0005] According to an aspect of the present disclosure, an image segmentation method is provided, comprising: obtaining a movement parameter corresponding to each human body frame in a first global image, the movement parameter being used to represent a movement law or a movement trend corresponding to the human body frame; for each human body frame, scaling the human body frame according to the movement parameter to obtain a target frame corresponding to the human body frame; determining a first target image according to the target frame corresponding to each human body frame and the first global image; and performing image segmentation on the first target image to obtain a human body segmentation result of the first global image.

[0006] The image segmentation method provided by the embodiment of the present disclosure can be applied to image segmentation in a single-person scene and can also be applied to image segmentation in a multi-person scene. In the embodiment of the present disclosure, the size of the target image can be adjusted in real time according to the movement law or the movement trend of the human body, so that the target image can keep up with the movement of the human body in time, thereby reducing the probability that the portrait leaves the target image and the probability that the proportion of the pixels occupied by the portrait in the target image is too small during the movement of the human body, and effectively improving the image segmentation effect when the human body moves.

[0007] In a possible implementation, the acquiring the movement parameter corresponding to each human body bounding box in the first global image comprises: performing target detection on the first global image to obtain a target detection result of the first global image, the target detection result being used to indicate positions of human body bounding boxes included in the first global image; determining scene information of the first global image and distance information corresponding to each human body bounding box in the first global image according to the target detection result of the first global image, the scene information being used to indicate whether the first global image is a global image in a single-person scene or a global image in a multi-person scene, and the distance information being used to indicate distances between human bodies in the human body bounding boxes and a first image acquisition device, the first image acquisition device being used to acquire the first global image; for each human body bounding box, obtaining a movement parameter corresponding to the human body bounding box according to the scene information, distance information corresponding to the human body bounding box, and a first preset mapping relationship, the first preset mapping relationship being used to indicate movement parameters corresponding to human body bounding boxes in different scenes and at different distances.

[0008] In the embodiments of the present disclosure, the movement parameter corresponding to the human body bounding box in the first global image is obtained based on the scene information of the first global image and distance information corresponding to the human body bounding box in the first global image, so that a condition for scaling of the human body bounding box is provided, and the image segmentation effect when the human body moves is improved.

[0009] In a possible implementation, the method further comprises: acquiring a first video, the first video corresponding to a first scene, the first scene being a single-person scene or a multi-person scene, and the first video being used to record movement of a single person or multiple persons in a first movement range, wherein the first movement range is at a first distance from a second image acquisition device, and the second image acquisition device is used to acquire the first video; performing limb tracking on a target person in the first video to obtain positions of a human body bounding box corresponding to the target person in each frame of image of the first video; determining a movement speed, a movement amplitude, and a second distance of the target person according to the positions of the human body bounding box corresponding to the target person in each frame of image of the first video, the second distance being used to indicate a distance between the target person and a reference position of the first video; obtaining a first movement parameter according to the movement speed and the movement amplitude of the target person; and establishing the first preset mapping relationship based on the first scene, the second distance, and the first movement parameter.

[0010] In the embodiments of the present disclosure, the scene and the mapping relationship between the position of the target person and the movement parameter are obtained by analyzing the historical image through limb tracking, thereby providing a basis for determining the movement parameter corresponding to the human body bounding box in the first global image.

[0011] In a possible implementation, the method further includes: in a case where the first scene is a single-person scene, setting a movement frequency of a person in the first video to be greater than a first frequency threshold, and a length of a projection of the first movement range in an x-axis direction of a camera coordinate system of the second image acquisition device to be greater than a first movement threshold; in a case where the first scene is a multi-person scene, setting the movement frequency of the person in the first video to be less than or equal to a second frequency threshold, and the length of the projection of the first movement range in the x-axis direction of the camera coordinate system of the second image acquisition device to be less than or equal to a second movement threshold; the second frequency threshold is less than or equal to the first frequency threshold, and the second movement threshold is less than or equal to the first movement threshold.

[0012] In the embodiments of the present disclosure, by setting a larger movement range and a movement frequency for a single-person scene, and setting a smaller movement range and a movement frequency for a multi-person scene, the movement regularity of the person is made to be consistent with the actual scene, the accuracy of the movement parameters corresponding to the human body box is improved, and the image segmentation effect when the person moves is improved.

[0013] In a possible implementation, the scaling the human body box according to the movement parameter corresponding to the human body box to obtain a target box corresponding to the human body box includes: determining a scaling coefficient of the human body box according to the movement parameter corresponding to the human body box and a second preset mapping relationship, and scaling the human body box according to the scaling coefficient to obtain the target box corresponding to the human body box, the second preset mapping relationship being used to indicate scaling coefficients corresponding to different movement parameters.

[0014] In a possible implementation, the determining the first target image according to the target box corresponding to each human body box and the first global image includes: merging the target box corresponding to each human body box to obtain a merged box, and obtaining the first target image according to the merged box and the first global image, the first target image corresponding to a merged box that can cover the target boxes corresponding to all human body boxes and has a minimum area.

[0015] In the embodiments of the present disclosure, the first global image is captured based on a merged box that can cover the target boxes corresponding to all human body boxes and has a minimum area, and then the first target image is obtained. In this way, the possibility that the portrait occupies too few pixels in the first target image can be reduced, and the image segmentation effect is improved.

[0016] In a possible implementation, the image segmentation on the first target image to obtain the human body segmentation result of the first global image comprises: acquiring a second target image corresponding to a second global image, the second global image being a previous frame image of the first global image in a video, and the second target image representing a target image used when acquiring a human body segmentation result of the second global image; determining a moving amplitude of the first target image relative to the second target image; and performing image segmentation on the first target image to obtain the human body segmentation result of the first global image in a case where the moving amplitude of the first target image relative to the second target image is greater than a first amplitude threshold.

[0017] In this way, the stability of the segmentation result is improved by keeping the position of the target image relatively stable.

[0018] In a possible implementation, the method further comprises: in a case where the moving amplitude of the first target image relative to the second target image is less than or equal to the first amplitude threshold, performing image segmentation by using the second target image to obtain the human body segmentation result of the first global image.

[0019] In a possible implementation, the image segmentation on the first target image to obtain the human body segmentation result of the first global image comprises: acquiring a second target image corresponding to a second global image, the second global image and the first global image belonging to a same video, and the second global image being a previous frame image of the first global image, and the second target image representing a target image used when acquiring a human body segmentation result of the second global image; determining a coverage rate of the first target image relative to the second target image; and performing image segmentation on the first target image to obtain the human body segmentation result of the first global image in a case where the coverage rate is less than a second amplitude threshold.

[0020] In this way, the target image can be updated in a timely manner, and the image segmentation result is improved.

[0021] In a possible implementation, the method further comprises:

[0022] In a case where the coverage rate is greater than or equal to the second amplitude threshold, performing image segmentation by using the second target image to obtain the human body segmentation result of the first global image.

[0023] According to an aspect of the present disclosure, an image segmentation apparatus is provided, comprising:

[0024] The first acquisition module is configured to acquire a moving parameter corresponding to each human body frame in a first global image, the moving parameter being used to represent a moving law or a moving trend corresponding to the human body frame.

[0025] a scaling module, configured to, for each of the human body boxes, scale the human body box according to the movement parameter obtained by the first obtaining module, to obtain a target box corresponding to the human body box;

[0026] a first determining module, configured to determine a first target image according to the target boxes corresponding to the human body boxes obtained by the scaling module and the first global image;

[0027] a first segmentation module, configured to perform image segmentation on the first target image determined by the first determining module, to obtain a human body segmentation result of the first global image.

[0028] In a possible implementation, the first obtaining module is further configured to:

[0029] perform target detection on the first global image, to obtain a target detection result of the first global image, the target detection result being used to indicate positions of human body boxes included in the first global image;

[0030] determine, according to the target detection result of the first global image, scene information of the first global image and distance information corresponding to each human body box in the first global image, the scene information being used to indicate whether the first global image is a global image in a single-person scene or a global image in a multi-person scene, and the distance information being used to indicate a distance between a human body in the human body box and a first image acquisition device, the first image acquisition device being used to acquire the first global image;

[0031] for each of the human body boxes, obtain a movement parameter corresponding to the human body box according to the scene information, the distance information corresponding to the human body box, and a first preset mapping relationship, the first preset mapping relationship being used to indicate movement parameters corresponding to human body boxes in different scenes and at different distances.

[0032] In a possible implementation, the apparatus further includes:

[0033] a second obtaining module, configured to obtain a first video, the first video corresponding to a first scene, the first scene being a single-person scene or a multi-person scene, the first video being used to record movement of a single person or multiple persons in a first movement range, wherein the first movement range is at a first distance from a second image acquisition device, and the second image acquisition device is used to acquire the first video;

[0034] a tracking module, configured to perform limb tracking on a target person in the first video, to obtain positions of a human body box corresponding to the target person in each frame of image of the first video;

[0035] a second determining module, configured to determine a moving speed, a moving amplitude and a second distance of the target person according to a position of a human body bounding box corresponding to the target person in each frame image of the first video, the second distance being used to indicate a distance between the target person and a reference position of the first video;

[0036] a third obtaining module, configured to obtain a first moving parameter according to the moving speed and the moving amplitude of the target person;

[0037] a establishing module, configured to establish the first preset mapping relationship based on the first scene, the second distance and the first moving parameter.

[0038] In a possible implementation, the apparatus further includes:

[0039] a first setting module, configured to, in a case where the first scene is a single-person scene, set a moving frequency of a person in the first video to be greater than a first frequency threshold, and a length of a projection of the first moving range in an x-axis direction of a camera coordinate system of the second image collection device to be greater than a first moving threshold;

[0040] a second setting module, configured to, in a case where the first scene is a multi-person scene, set the moving frequency of the person in the first video to be less than or equal to a second frequency threshold, and the length of the projection of the first moving range in the x-axis direction of the camera coordinate system of the second image collection device to be less than or equal to a second moving threshold;

[0041] wherein the second frequency threshold is less than or equal to the first frequency threshold, and the second moving threshold is less than or equal to the first moving threshold.

[0042] In a possible implementation, the scaling module is further configured to:

[0043] determine a scaling coefficient of the human body bounding box according to a moving parameter corresponding to the human body bounding box and a second preset mapping relationship;

[0044] scale the human body bounding box according to the scaling coefficient to obtain a target bounding box corresponding to the human body bounding box, the second preset mapping relationship being used to indicate scaling coefficients corresponding to different moving parameters.

[0045] In a possible implementation, the first determining module is further configured to:

[0046] merge the target bounding boxes corresponding to the human body bounding boxes to obtain a merged bounding box;

[0047] obtain the first target image according to the merged bounding box and the first global image;

[0048] The first target image corresponds to a merged frame with the smallest area in all merged frames capable of covering target frames corresponding to the human body frames.

[0049] In a possible implementation, the first segmentation module is further configured to:

[0050] obtain a second target image corresponding to a second global image, the second global image being a previous frame image of the first global image in a video, and the second target image representing a target image used when obtaining a human body segmentation result of the second global image;

[0051] determine a movement amplitude of the first target image relative to the second target image;

[0052] perform image segmentation on the first target image to obtain the human body segmentation result of the first global image, in a case where the movement amplitude of the first target image relative to the second target image is greater than a first amplitude threshold.

[0053] In a possible implementation, the apparatus further includes:

[0054] a second segmentation module configured to, in a case where the movement amplitude of the first target image relative to the second target image is less than or equal to the first amplitude threshold, perform image segmentation on the second target image to obtain the human body segmentation result of the first global image.

[0055] In a possible implementation, the first segmentation module is further configured to:

[0056] obtain a second target image corresponding to a second global image, the second global image and the first global image belonging to a same video, and the second global image being a previous frame image of the first global image, and the second target image representing a target image used when obtaining a human body segmentation result of the second global image;

[0057] determine a coverage rate of the first target image relative to the second target image;

[0058] perform image segmentation on the first target image to obtain the human body segmentation result of the first global image, in a case where the coverage rate is less than a second amplitude threshold.

[0059] In a possible implementation, the apparatus further includes:

[0060] a third segmentation module configured to, in a case where the coverage rate is greater than or equal to the second amplitude threshold, perform image segmentation on the second target image to obtain the human body segmentation result of the first global image.

[0061] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored by the memory to perform the above method.

[0062] According to an aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions, which when executed by a processor, implement the above method.

[0063] It should be understood that the general description above and the following detailed description below are only exemplary and explanatory, rather than restrictive, of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of the exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0064] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0065] Figure 1 A flow chart of an image segmentation method according to an embodiment of the present disclosure is shown;

[0066] Figure 2 An exemplary schematic diagram of a human body frame and a target frame in an embodiment of the present disclosure is shown;

[0067] Figure 3 An exemplary schematic diagram of a target frame and a merged frame in an embodiment of the present disclosure is shown;

[0068] Figure 4 A block diagram of an image segmentation apparatus according to an embodiment of the present disclosure is shown;

[0069] Figure 5 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown;

[0070] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0071] Various exemplary embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements or elements having a similar function. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0072] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0073] The term "and / or", as used herein, merely describes association between associated objects, and can indicate that three cases, for example, A and / or B, can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one of", as used herein, means any one of a plurality or at least two of a plurality in any combination, for example, includes at least one of A, B, and C, can mean any one or more elements selected from the set consisting of A, B, and C.

[0074] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some examples, methods, means, elements and circuits that are well known to those skilled in the art are not described in detail, in order to highlight the main ideas of the present disclosure.

[0075] In scenarios such as home entertainment, karaoke, online classes, and online meetings, it is necessary to set a background that matches the scenario to improve the sense of immersion, so the global image collected by an image collection device (for example, a camera, a video camera, a camera, a mobile phone, a tablet, or other devices with photographing or video recording functions) is segmented into a human body and a background. When performing segmentation, a target image containing a portrait (i.e., an image corresponding to a human body) is usually first selected from the global image, and then the target image is input into a background segmentation model for segmentation of the portrait and the background image. The segmentation effect of the target image is related to the proportion of pixels occupied by the portrait in the target image. If the selected target image is too small, the portrait may leave the target image when the human body moves; if the selected target image is too large, the foreground segmentation details may be poor due to the low proportion of pixels occupied by the portrait in the target image.

[0076] The image segmentation method provided by the embodiments of the present disclosure can be applied to image segmentation in a single-person scenario, and can also be applied to image segmentation in a multi-person scenario. In the embodiments of the present disclosure, the size of the target image can be adjusted in real time according to the movement rule or movement trend of the human body, so that the target image can keep up with the movement of the human body in time, thereby reducing the probability that the portrait leaves the target image and the probability that the proportion of pixels occupied by the portrait in the target image is too small during the movement of the human body, and effectively improving the image segmentation effect when the human body moves.

[0077] Figure 1A flowchart of an image segmentation method according to an embodiment of the present disclosure is shown. The image segmentation method can be performed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like. The method can be implemented by a processor invoking computer-readable instructions stored in a memory. Alternatively, the method can be performed by a server. As shown in Figure 1 The image segmentation method includes the following steps.

[0078] In step S11, a movement parameter corresponding to each human body bounding box in a first global image is obtained.

[0079] The global image is an image containing a person. The global image can contain one person or multiple persons, and the present disclosure does not limit this. The global image can be obtained by an image acquisition device acquiring images of persons in a certain space range, can be an image frame containing a person obtained from a video, or can be obtained by other means, and the present disclosure does not make specific limitations.

[0080] The first global image in step S11 can be used to represent a global image to be segmented currently. In the present disclosure, the first global image is input into a target detection model, and the target detection model can output a target detection result of the first global image. The target detection result can be used to indicate the position of the human body bounding box in the first global image. The target detection model can be a convolutional neural network model, and the structure and training process of the target detection model are not limited in the present disclosure.

[0081] It can be understood that the human body in the first global image corresponds to the human body bounding box indicated by the target detection result one by one. Therefore, when the first global image contains one human body, the movement parameter corresponding to one human body bounding box can be obtained in step S11. When the first global image contains multiple human bodies, the movement parameters corresponding to multiple human body bounding boxes can be obtained in step S11.

[0082] The movement parameter corresponding to the human body bounding box can be used to represent the movement rule or movement trend of the human body bounding box. The movement rule of the human body bounding box can be used to reflect the movement of the human body based on historical image analysis. The movement trend of the human body bounding box can be used to reflect the possible future movement of the human body.

[0083] In a possible implementation, the movement parameter of the human body frame includes, but is not limited to, a movement speed, a movement amplitude, a movement direction and other parameters for reflecting the movement of the human body. The specific process of obtaining the movement parameter corresponding to each human body frame will be described in detail in combination with possible implementations of the embodiments of the present disclosure, and will not be repeated here.

[0084] In step S12, for each human body frame, the human body frame is scaled according to the movement parameter, to obtain a target frame corresponding to the human body frame.

[0085] The movement of the human body relative to the image acquisition device can be decomposed into left-right movement and / or forward-backward movement. When the human body moves left and right relative to the image acquisition device, if the size of the target image remains unchanged, the portrait may be out of the target image. When the human body is close to the image acquisition device, the human body may walk out of the field of view of the image acquisition device. If the size of the target image remains unchanged, the target image may not contain the portrait or the contained portrait may be incomplete. When the human body is away from the image acquisition device, if the size of the target image remains unchanged, the proportion of the portrait in the target image may be too small. Considering that the movement parameter of the human body frame can represent the movement of the human body. Therefore, in the embodiments of the present disclosure, the human body frame can be scaled according to the movement parameter corresponding to the human body frame to obtain a target frame corresponding to the human body frame, which on the one hand reduces the possibility of the human body moving out of the target frame, and on the other hand reduces the possibility of the portrait occupying too small a proportion in the target frame.

[0086] In the embodiments of the present disclosure, scaling the human body frame includes contracting the human body frame or expanding the human body frame. Contracting the human body frame to obtain a target frame can increase the proportion of the portrait in the target frame, which is beneficial to improving the accuracy of the edge in subsequent image segmentation, thereby improving the image segmentation effect. Expanding the human body frame to obtain a target frame can reduce the possibility of the human body moving out of the target frame, so that the human body is still within the target frame even if there is corresponding movement in a period of time, thereby improving the image segmentation effect.

[0087] In a possible implementation, the scaling, in the step S12, of the human body frame according to the movement parameter corresponding to the human body frame to obtain a target frame corresponding to the human body frame includes: determining a scaling coefficient of the human body frame according to the movement parameter corresponding to the human body frame and a second preset mapping relationship; and scaling the human body frame according to the scaling coefficient to obtain a target frame corresponding to the human body frame.

[0088] The second preset mapping relationship can be used to indicate scaling coefficients corresponding to different movement parameters. The second preset mapping relationship can be set according to needs or experience. For example, the greater the movement amplitude or the greater the movement speed, the greater the corresponding scaling coefficient. When the movement direction is away from the image acquisition device, the scaling coefficient is less than 1, and the greater the away amplitude, the smaller the scaling coefficient. When the movement direction is close to the image acquisition device, the scaling coefficient is greater than 1, and the greater the close amplitude, the greater the scaling coefficient. Taking the movement direction away from the image acquisition device as an example, that is, in the process that the human body gradually moves away from the image acquisition device, the proportion of the portrait in the global image becomes smaller, and the human body box also needs to be correspondingly reduced, and as the human body moves from close to far from the image acquisition device, the amplitude of the reduction of the human body box also becomes larger, that is, the scaling coefficient for indicating the reduction amplitude of the human body box becomes smaller. Correspondingly, in the process that the movement direction is close to the image acquisition device, the proportion of the human body in the global image becomes larger, the human body box is expanded, and the scaling coefficient becomes larger.

[0089] The image coordinate system of the global image takes the center of the global image as the coordinate origin, the x-axis of the image coordinate system is parallel to the upper and lower edges of the global image, and the y-axis of the image coordinate system is parallel to the left and right edges of the global image. In a possible implementation, the scaling coefficient includes a scaling coefficient in the x-axis direction of the image coordinate system of the global image and a scaling coefficient in the y-axis direction of the image coordinate system of the global image. For example, in a single-person scene, the human body moves more to the left and right and less forward and backward, that is, the human body box moves more in the x-axis direction of the image coordinate system of the global image and less in the y-axis direction of the image coordinate system of the global image, and therefore, the scaling coefficient in the x-axis direction of the image coordinate system of the global image set for the human body box is relatively large, and the scaling coefficient in the y-axis direction is relatively small. In an example, the human body jumps left and right in front of the image acquisition device, at this time, the scaling coefficient in the x-axis direction of the image coordinate system of the global image is 1.2, and the scaling coefficient in the y-axis direction is 1.0.

[0090] Figure 2 An example schematic diagram of the human body box and the target box in the embodiment of the present disclosure is shown. As shown in the figure, Figure 2 the scaling coefficient in the x-axis direction of the image coordinate system of the global image is 2, and the scaling coefficient in the y-axis direction of the image coordinate system of the global image is 1.5. After scaling, in the x-axis direction of the image coordinate system of the global image, the length of the target box is 2 times the length of the human body box, and in the y-axis direction of the image coordinate system of the global image, the width of the target box is 1.5 times the width of the human body box.

[0091] In a possible implementation, the scaling coefficient includes a dilation coefficient and a contraction coefficient. When the scaling coefficient is greater than or equal to 1, the scaling coefficient can be referred to as a dilation coefficient, and the target frame can be obtained by dilating the human body frame, that is, the area of the target frame is greater than or equal to the area of the human body frame. When the scaling coefficient is less than 1, the scaling coefficient can be referred to as a contraction coefficient, and the target frame can be obtained by contracting the human body frame, that is, the area of the target frame is less than the area of the human body frame.

[0092] In step S13, the first target image is determined according to the target frame corresponding to each human body frame and the first global image.

[0093] The first target image can represent an image used for subsequent image segmentation. In the embodiment of the present disclosure, the first target image can be obtained by cropping the first global image according to the position of the target frame corresponding to each human body frame.

[0094] In a possible implementation, step S13 can include merging the target frame corresponding to each human body frame to obtain a merged frame, and obtaining the first target image according to the merged frame and the first global image.

[0095] The merged frame represents the merging result of the target frame corresponding to each human body frame. The first target image corresponds to the merged frame with the smallest area in the merged frame that can cover all target frames corresponding to the human body frames.

[0096] The step S13 will be described below in combination with a single-person scene and a multi-person scene respectively.

[0097] In the single-person scene, the movement parameter corresponding to the human body frame can be obtained in step S11, and the target frame corresponding to the human body frame can be obtained in step S12. Therefore, in step S13, the first target image can be cropped from the first global image according to the position of the human body frame in the first global image.

[0098] In the multi-person scene, the movement parameters corresponding to the multiple bounding boxes can be obtained in step S11, and the target boxes corresponding to the multiple bounding boxes can be obtained in step S12. Therefore, the target boxes corresponding to the multiple bounding boxes need to be merged to obtain a merged box in step S13, and then the first target image is cropped from the first target image according to the position of the merged box. It is considered that the proportion of the portrait in the first target image is too low, which can lead to the details of the foreground segmentation. Therefore, the area of the merged box should not be too large. Therefore, in the embodiment of the present disclosure, the first global image is collected based on the merged box with the smallest area that can cover the target boxes corresponding to all the bounding boxes, and then the first target image is obtained. In this way, the possibility that the proportion of the portrait in the first target image is too low can be reduced, which is beneficial to improve the image segmentation effect.

[0099] It should be noted that in the multi-person scene, the multiple target boxes can be merged to obtain a merged box, which is optional. That is, in the multi-person scene, after obtaining the target boxes corresponding to the multiple bounding boxes, the target images can be obtained based on the target boxes respectively, and then the image segmentation is performed on the target images respectively, and the human body segmentation result of the first global image can also be obtained.

[0100] Figure 3 An exemplary schematic diagram of the target box and the merged box in the embodiment of the present disclosure is shown. As shown in Figure 3 Three target boxes are obtained based on the first global image, and each target box corresponds to a human body. After merging the three target boxes, a merged box is obtained, which can cover all the target boxes. Taking Figure 3 the two merged boxes shown as an example, the merged box with the smallest area is selected from all the merged boxes for image cropping, and the first target image can be obtained.

[0101] In the related art, for the multi-person scene, the target image corresponding to each human body needs to be obtained based on the target box corresponding to each human body, and then the image segmentation is performed on the target images corresponding to the human bodies respectively. It can be seen that the processing of the pictures is multiplied in the multi-person scene, which causes pressure to the chip for image segmentation processing, resulting in insufficient chip computing power, slow processing speed, and inability of the user to run other functional modules in parallel on the chip, which greatly reduces the user experience.

[0102] In the embodiment of the present disclosure, all the target boxes are merged in the multi-person scene, so that a target image is obtained, and then only the image segmentation is performed on the target image. In this way, the multiple image segmentation processes in the multi-person scene are converted into a single image segmentation process, so that the resources and time consumed in the image segmentation process in the multi-person scene and the single-person scene are comparable, the efficiency is improved, the resources are saved, and the user experience is improved.

[0103] In step S14, image segmentation is performed on the first target image to obtain a human body segmentation result of the first global image.

[0104] In the embodiments of the present disclosure, after the first target image is input into the background segmentation model, a human body segmentation result of the first target image can be obtained, which indicates whether each pixel point in the first target image is a human body or a non-human body. According to the human body segmentation result of the first target image and the human body segmentation result of the first global image, which indicates whether each pixel point in the first global image is a human body or a non-human body. The background segmentation model can refer to related technologies, which will not be described here again. The background segmentation model can be a neural network model, and the embodiments of the present disclosure do not limit the structure and training process of the background segmentation model.

[0105] In the embodiments of the present disclosure, the size of the target image can be adjusted in real time according to the movement parameter of the human body, so that the target image can keep up with the movement of the human body in time, thereby reducing the probability that the human body leaves the target image and the probability that the proportion of the human image in the target image is too small during the movement of the human body, and effectively improving the image segmentation effect when the human body moves.

[0106] It is considered that whether it is a single-person scene or a multi-person scene, if a new target image is used for image segmentation every time the human body moves, the segmentation effect will be unstable. In order to keep the position of the target image relatively stable, in the embodiments of the present disclosure, the target image can be smoothed. The specific process of smoothing is described in detail below.

[0107] In a possible implementation, step S14 can include: obtaining a second target image corresponding to a second global image; determining a movement amplitude of the first target image relative to the second target image; and performing image segmentation on the first target image to obtain a human body segmentation result of the first global image, in a case where the movement amplitude of the first target image relative to the second target image is greater than a first amplitude threshold.

[0108] The second global image and the first global image belong to the same video, and the second global image is a previous frame image of the first global image. It can be understood that the size, resolution, etc. of the first global image and the second global image are the same.

[0109] The second target image represents a target image used when obtaining the human body segmentation result of the second global image. The second target image can be subjected to image segmentation processing to obtain the human body segmentation result of the second global image. The process of obtaining the second target image can refer to the process of obtaining the first target image (steps S11 to S13), which will not be described here again.

[0110] In a possible implementation, the movement amplitude of the first target image relative to the second target image can be determined according to a coordinate difference between a preset position (for example, a left lower corner vertex, a right upper corner vertex, or a center point, etc.) of the first target image and a preset position of the second target image. In an example, the coordinate of the left lower corner vertex of the first target image in the first global image is (100, 100), and the coordinate of the left lower corner vertex of the second target image in the second global image is (200, 100), and the movement amplitude of the first target image relative to the second target image is 100 pixels.

[0111] The first amplitude threshold value can be set as needed, for example, the first amplitude threshold value can be set as 50 pixels or 150 pixels, etc. In the case where the movement amplitude of the first target image relative to the second target image is greater than the first amplitude threshold value, it indicates that the human body has a large amplitude of movement, at this time, in order to improve the image segmentation effect, the first target image can be subjected to image segmentation to obtain the human body segmentation result of the first global image.

[0112] In a possible implementation, the method can further include: in the case where the movement amplitude of the first target image relative to the second target image is less than or equal to the first amplitude threshold value, performing image segmentation on the second target image to obtain the human body segmentation result of the first global image.

[0113] In the case where the movement amplitude of the first target image relative to the second target image is less than or equal to the first amplitude threshold value, it indicates that the movement amplitude of the human body is small, at this time, in order to improve the stability of the image subjected to image segmentation, the second target image can be subjected to image segmentation to obtain the human body segmentation result of the first global image.

[0114] Considering the case where the human body moves slowly, the relative movement amplitude between the target images corresponding to adjacent global images is small, which can cause the target image used for image segmentation to be updated not in time. In order to update the target image in time, in the embodiment of the present disclosure, the target image can be subjected to update processing. The specific process of the update processing is described in detail below.

[0115] In a possible implementation, step S14 can include: obtaining a second target image corresponding to a second global image; determining a coverage rate of the first target image relative to the second target image; and in the case where the coverage rate is less than a second amplitude threshold value, performing image segmentation on the first target image to obtain a human body segmentation result of the first global image.

[0116] In one example, the ratio of the overlapping area of ​​the first target image and the second target image to the area of ​​the second target image can be determined as the coverage of the first target image relative to the second target image.

[0117] The second amplitude threshold can be set as needed, for example, it can be 40% or 50%. When the coverage of the first target image relative to the second target image is less than the second amplitude threshold, it indicates that the human body has moved significantly, and the target image for image segmentation needs to be updated. Therefore, image segmentation can be performed on the first target image to obtain the human body segmentation effect of the first global image.

[0118] In one possible implementation, the method may further include: when the coverage is greater than or equal to the second amplitude threshold, performing image segmentation using the second target image to obtain the human body segmentation result of the first global image.

[0119] If the coverage of the first target image relative to the second target image is greater than or equal to the second amplitude threshold, it indicates that the arbitrary movement amplitude is small. In this case, to maintain the stability of the human figure, image segmentation can be performed on the second target image to obtain the human figure segmentation result of the first global image. In this way, the target image used for image segmentation in the previous frame is used as the target image for image segmentation in the current frame, ensuring that the target image for image segmentation is the same. Consequently, the image segmentation result will not change, and the segmented human figure will also remain unchanged, avoiding human figure shaking, maintaining human figure stability, and improving the user experience.

[0120] The following section details the process of obtaining the motion parameters corresponding to each human bounding box in the first global image. Considering that this process requires the use of a first preset mapping relationship to indicate the motion parameters corresponding to human bounding boxes at different distances and in different scenes, the process of obtaining the first preset mapping relationship will be explained first.

[0121] The first preset mapping relationship includes the scene, the distance between the person in the global image and the reference position of the video, and movement parameters. In one possible implementation, the method further includes: acquiring a first video; performing limb tracking on the target person in the first video to obtain the position of the human bounding box corresponding to the target person in each frame of the first video; determining the movement speed, movement amplitude, and second distance of the target person based on the position of the human bounding box corresponding to the target person in each frame of the first video; obtaining a first movement parameter based on the movement speed and movement amplitude of the target person; and establishing the first preset mapping relationship based on the first scene, the second distance, and the first movement parameter.

[0122] The first video corresponds to a first scene, which can be a single-person scene or a multi-person scene, and is used to record the movement of a single person or multiple persons within a first movement range, wherein the first movement range is at a first distance from the second image capture device.

[0123] In one example, after the second image capture device is set up, a person can move within a first movement range at a first distance (e.g., 1 meter, 3 meters, or 5 meters, etc.) from the second image capture device. The second image capture device can capture a moving video of the person as the first video. By using a body tracking technology to track the target person in the first video, the position of the target person's body box in each frame image of the first video can be obtained. According to the position of the target person's body box in each frame image of the first video, the moving speed and moving amplitude of the target person and the second distance are determined, thereby obtaining the first movement parameter. Based on the first scene, the second distance, and the first movement parameter, the first preset mapping relationship is established.

[0124] The second distance is used to indicate the distance between the target person and the reference position of the first video. Specifically, the distance between the preset position (e.g., the left lower corner vertex, the right upper corner vertex, or the center point, etc.) of the target person and the reference position of the first video can be determined as the second distance corresponding to the target person. The reference position of the first video can be a pre-specified position in the first video, such as the lower boundary line of the first video, or the upper boundary line of the first video, etc. Accordingly, the distance between the target person and the lower boundary line of the first video can be referred to as the second distance, or the distance between the target person and the upper boundary line of the first video. The size of the second distance can be used to represent the distance between the target person and the second image capture device. Taking the second distance as an example for indicating the distance between the target person and the lower boundary line of the first video, the larger the second distance, the closer the target person is to the second image capture device, and the smaller the second distance, the farther the target person is from the second image capture device. Taking the second distance as an example for indicating the distance between the target person and the upper boundary line of the first video, the larger the second distance, the farther the target person is from the second image capture device, and the smaller the second distance, the farther the target person is from the second image capture device.

[0125] In a possible implementation, the method further includes: in a case where the first scene is a single-person scene, setting a moving frequency of a person in the first video to be greater than a first frequency threshold, and a length of a projection of the first moving range in an x-axis direction of a camera coordinate system of the second image acquisition device to be greater than a first moving threshold; in a case where the first scene is a multi-person scene, setting the moving frequency of the person in the first video to be less than or equal to a second frequency threshold, and the length of the projection of the first moving range in the x-axis direction of the camera coordinate system of the second image acquisition device to be less than or equal to a second moving threshold; wherein the second frequency threshold is less than or equal to the first frequency threshold, and the second moving threshold is less than or equal to the first moving threshold. In an example, the first moving range can be a rectangle, and a shortest distance between a lower edge of the first moving range and the second image acquisition device can be determined as the first distance. In another example, the first moving range can be a sector ring (i.e., a part of a circular ring) with the second image acquisition device as a center, and an inner radius of the first moving range can be determined as the first distance.

[0126] It can be understood that, because the person in the single-person scene has less restrictions and a larger activity space, the moving range of the person in the single-person scene is larger, and the moving range of the person in the multi-person scene is smaller. The smaller the first distance is, the greater the moving amplitude and the moving speed are; the greater the first distance is, the smaller the moving amplitude and the moving speed are. Therefore, the moving frequency and the moving range of the person in the first video set in the single-person scene are greater than the moving frequency and the moving range of the person in the first video set in the multi-person scene. In addition, the target person in the multi-person scene can be any one or more persons in the multi-person scene.

[0127] It should be noted that the first frequency threshold, the first moving threshold, the second frequency threshold, and the second moving threshold can be set as needed, as long as the second frequency threshold is less than or equal to the first frequency threshold, and the second moving threshold is less than or equal to the first moving threshold.

[0128] So far, the first preset mapping relationship is obtained. On this basis, the process of obtaining the moving parameter corresponding to each human body box in the first global image is described.

[0129] In a possible implementation, the step S11 of obtaining the moving parameter corresponding to each human body box in the first global image can include: performing target detection on the first global image to obtain a target detection result of the first global image; determining, according to the target detection result of the first global image, scene information of the first global image and distance information corresponding to each human body box in the first global image; and for each human body box, determining, according to the scene information, the distance information corresponding to the human body box, and the first preset mapping relationship, a moving parameter corresponding to the human body box.

[0130] The target detection result can be used to indicate a position of a human body bounding box included in the first global image; the scene information can be used to indicate whether the first global image is a global image in a single-person scene or a global image in a multi-person scene; and the distance information can be used to indicate a distance between a human body in the human body bounding box and a first image acquisition device, the first image acquisition device representing an image acquisition device that acquires the first global image.

[0131] When the target detection result indicates a position of a human body bounding box, it can be determined that the first global image is a global image in a single-person scene. When the target detection result indicates positions of multiple human body bounding boxes, it can be determined that the first global image is a global image in a multi-person scene.

[0132] In one example, the distance information corresponding to the human body bounding box can be determined according to a position of the human body bounding box in the first global image. Specifically, a coordinate of a preset position (for example, a left lower corner vertex, a right upper corner vertex, or a center point, etc.) of the human body bounding box in a y-axis direction of the first global image can be determined as the distance information corresponding to the human body bounding box. Taking the coordinate system shown in FIG. 1 as an example, the smaller the coordinate value of the center point of the human body bounding box in the y-axis direction of the first global image, the closer the distance between the human body and the first image acquisition device; the larger the coordinate value of the center point of the human body bounding box in the y-axis direction of the first global image, the farther the distance between the human body and the first image acquisition device. Figure 2

[0133] In the embodiments of the present disclosure, the matching first preset mapping relationship can be found according to the scene information and the distance information corresponding to the human body bounding box, and a movement parameter in the matching first preset mapping relationship can be determined as the movement parameter corresponding to the human body bounding box.

[0134] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments without violating the principle logic. Limited to the length, the present disclosure will not be repeated. Those skilled in the art can understand that the specific execution order of each step in the above-mentioned method should be determined according to its function and possible internal logic.

[0135] In addition, the present disclosure also provides an image segmentation device, an electronic device, a computer readable storage medium, and a program, all of which can be used to implement any one of the image segmentation methods provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding description in the method part, and will not be repeated.

[0136] Figure 4 A block diagram of an image segmentation device according to an embodiment of the present disclosure is shown. As shown in FIG. 4, the device 40 includes: Figure 4

[0137] ​​The first obtaining module 41 is configured to obtain a movement parameter corresponding to each human body frame in the first global image, where the movement parameter is used to represent a movement rule or a movement trend corresponding to the human body frame.

[0138] The scaling module 42 is configured to scale, for each human body frame, the human body frame according to the movement parameter obtained by the first obtaining module 41, to obtain a target frame corresponding to the human body frame.

[0139] The first determining module 43 is configured to determine a first target image according to the target frame corresponding to each human body frame obtained by the scaling module 42 and the first global image.

[0140] The first segmentation module 44 is configured to perform image segmentation on the first target image determined by the first determining module 43, to obtain a human body segmentation result of the first global image.

[0141] In the embodiments of the present disclosure, the size of the target image can be adjusted in real time according to the movement rule or the movement trend of the human body, so that the target image can keep up with the movement of the human body in time, thereby reducing the probability that the human body image leaves the target image and the probability that the proportion of the human body image in the target image is too small during the movement of the human body, and effectively improving the image segmentation effect when the human body moves.

[0142] In a possible implementation, the first obtaining module is further configured to:

[0143] perform target detection on the first global image to obtain a target detection result of the first global image, where the target detection result is used to indicate the positions of the human body frames included in the first global image;

[0144] determine, according to the target detection result of the first global image, scene information of the first global image and distance information corresponding to each human body frame in the first global image, where the scene information is used to indicate whether the first global image is a global image in a single-person scene or a global image in a multi-person scene, and the distance information is used to indicate the distance between a human body in the human body frame and a first image acquisition device, where the first image acquisition device is used to acquire the first global image;

[0145] for each human body frame, obtain a movement parameter corresponding to the human body frame according to the scene information, the distance information corresponding to the human body frame and a first preset mapping relationship, where the first preset mapping relationship is used to indicate the movement parameter corresponding to the human body frame in different scenes and at different distances.

[0146] In a possible implementation, the apparatus further includes:

[0147] The second acquisition module is configured to acquire a first video, the first video corresponding to a first scene, the first scene being a single-person scene or a multi-person scene, and the first video being used to record movement of a single person or multiple persons in a first movement range, wherein the first movement range is at a first distance from a second image acquisition device, and the second image acquisition device is configured to acquire the first video.

[0148] The tracking module is configured to perform limb tracking on a target person in the first video to obtain positions of a human body frame corresponding to the target person in each frame image of the first video.

[0149] The second determination module is configured to determine a movement speed, a movement amplitude, and a second distance of the target person according to the positions of the human body frame corresponding to the target person in each frame image of the first video, wherein the second distance is used to indicate a distance between the target person and a reference position of the first video.

[0150] The third acquisition module is configured to obtain a first movement parameter according to the movement speed and the movement amplitude of the target person.

[0151] The establishment module is configured to establish the first preset mapping relationship based on the first scene, the second distance, and the first movement parameter.

[0152] In a possible implementation, the apparatus further includes:

[0153] The first setting module is configured to, in a case where the first scene is a single-person scene, set a movement frequency of a person in the first video to be greater than a first frequency threshold, and a length of a projection of the first movement range in an x-axis direction of a camera coordinate system of the second image acquisition device to be greater than a first movement threshold.

[0154] The second setting module is configured to, in a case where the first scene is a multi-person scene, set a movement frequency of a person in the first video to be less than or equal to a second frequency threshold, and a length of a projection of the first movement range in an x-axis direction of a camera coordinate system of the second image acquisition device to be less than or equal to a second movement threshold.

[0155] The second frequency threshold is less than or equal to the first frequency threshold, and the second movement threshold is less than or equal to the first movement threshold.

[0156] In a possible implementation, the scaling module is further configured to:

[0157] determine a scaling coefficient of the human body frame according to the movement parameter corresponding to the human body frame and a second preset mapping relationship.

[0158] scaling the human body frame according to the scaling coefficient to obtain a target frame corresponding to the human body frame, the second preset mapping relationship being used to indicate scaling coefficients corresponding to different movement parameters.

[0159] In a possible implementation, the first determining module is further configured to:

[0160] merge the target frames corresponding to the human body frames to obtain a merged frame;

[0161] obtain the first target image according to the merged frame and the first global image;

[0162] The first target image corresponds to a merged frame with the smallest area among the merged frames that can cover the target frames corresponding to all the human body frames.

[0163] In a possible implementation, the first determining module is further configured to:

[0164] obtain a second target image corresponding to a second global image, the second global image being a previous frame image of the first global image in a video, the second target image representing a target image used when obtaining a human body segmentation result of the second global image;

[0165] determine a movement amplitude of the first target image relative to the second target image;

[0166] perform image segmentation on the first target image to obtain the human body segmentation result of the first global image in a case where the movement amplitude of the first target image relative to the second target image is greater than a first amplitude threshold.

[0167] In a possible implementation, the apparatus further includes:

[0168] a second determining module, configured to perform image segmentation on the second target image to obtain the human body segmentation result of the first global image in a case where the movement amplitude of the first target image relative to the second target image is less than or equal to the first amplitude threshold.

[0169] In a possible implementation, the first determining module is further configured to:

[0170] obtain a second target image corresponding to a second global image, the second global image and the first global image belonging to a same video, and the second global image being a previous frame image of the first global image, the second target image representing a target image used when obtaining a human body segmentation result of the second global image;

[0171] determine a coverage rate of the first target image relative to the second target image;

[0172] In a case where the coverage is less than a second magnitude threshold, performing image segmentation on the first target image to obtain a human body segmentation result of the first global image.

[0173] In a possible implementation, the apparatus further includes:

[0174] In a case where the coverage is greater than or equal to the second magnitude threshold, performing image segmentation on the second target image to obtain the human body segmentation result of the first global image.

[0175] The method has specific technical correlation with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware operation efficiency or execution effect (including reducing data storage, reducing data transmission, improving hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the natural law.

[0176] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0177] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0178] The embodiments of the present disclosure also propose an electronic device, including a processor, a memory for storing processor-executable instructions, wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

[0179] The embodiments of the present disclosure also provide a computer program product, including computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0180] The electronic device can be provided as a terminal, a server or other forms of devices.

[0181] Figure 5A block diagram of an electronic device 800 is shown, according to an embodiment of the disclosure. The electronic device 800 can be, for example, a terminal device such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like.

[0182] Referring to Figure 5 The electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0183] The processing component 802 usually controls overall operations of the electronic device 800, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0184] The memory 804 is configured to store various types of data to support operations of the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0185] The power supply component 806 provides power for the various components of the electronic device 800. The power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the electronic device 800.

[0186] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. The front camera and / or the rear camera can receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0187] The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 includes a microphone (MIC) to receive an external audio signal when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker to output an audio signal.

[0188] The I / O interface 812 provides an interface for the processing component 802 and peripheral interface modules, such as a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0189] The sensor component 814 includes one or more sensors to provide various state assessments for the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the electronic device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change in position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and a temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, useful for image- and / or light- capturing applications. In some embodiments, the sensor component 814 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0190] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 2nd-Generation (2G) digital cellular technology, 3rd-Generation (3G) digital cellular technology, 4th-Generation (4G) digital cellular technology, long-term evolution (LTE) of the Universal Mobile Telecommunication System (UMTS), 5th-Generation (5G) digital cellular technology, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 can further include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0191] In an example embodiment, the electronic device 800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0192] In an example embodiment, a non-transitory computer-readable storage medium, such as the memory 804 including computer program instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to complete the above-described methods.

[0193] The present disclosure relates to the field of augmented reality, by acquiring image information of a target object in a real environment, and then detecting or recognizing the related features, states and attributes of the target object by means of various vision-related algorithms, so as to obtain an AR effect combining virtual and real objects that matches a specific application. Exemplarily, the target object can involve a face, a limb, a gesture, a motion, etc. related to a human body, or a marker, a sign, etc. related to an object, or a sand table, a display area or a display object, etc. related to a venue or a site. The vision-related algorithms can involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of an object, pose or depth detection of an object, etc. The specific application can not only involve interactive scenarios such as touring, navigation, explanation, reconstruction, virtual effect superimposed display, etc. related to a real scene or an object, but also interactive scenarios such as special effect processing related to a person, e.g. makeup beautification, limb beautification, special effect display, virtual model display, etc. The detection or recognition of the related features, states and attributes of the target object can be achieved by means of a convolutional neural network. The convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0194] Figure 6 A block diagram of an electronic device 1900 according to embodiments of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 6 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-mentioned method.

[0195] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server TM ), a graphical user interface operating system (Mac OSX TM ) developed by Apple Inc., a multi-user multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or the like.

[0196] In example embodiments, a non-transitory computer-readable storage medium, e.g., memory 1932 including computer program instructions, is also provided, which can be executed by processing component 1922 of electronic device 1900 to implement the above-described methods.

[0197] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0198] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0199] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0200] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0201] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0202] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0203] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0204] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0205] The computer program product can be embodied in particular by hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in particular as a computer storage medium, in another alternative embodiment, the computer program product is embodied in particular as a software product, such as a software development kit (SDK) or the like.

[0206] The above description of the various embodiments is intended to be illustrative in all respects, and not restrictive. The scope of the present disclosure is indicated by the appended claims, rather than the foregoing description, and all changes that come within the meaning and range of equivalents are intended to be embraced therein.

[0207] Those skilled in the art can understand that, in the above-described method of the specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible inherent logic.

[0208] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that it has entered the personal information collection range and will collect personal information. If the individual voluntarily enters the collection range, it is considered to agree to collect personal information. Or on the device for processing personal information, through the pop-up information or by asking the individual to upload his personal information, the individual's authorization is obtained under the condition that the device uses obvious mark / information to inform the individual of the personal information processing rules. The personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

[0209] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image segmentation method characterized by, The method comprises: obtaining a movement parameter corresponding to each human body frame in a first global image, the movement parameter being used to represent a movement law or a movement trend corresponding to the human body frame; scaling the human body frame according to the movement parameter to obtain a target frame corresponding to the human body frame; determining a first target image according to the target frame corresponding to each human body frame and the first global image; performing image segmentation on the first target image to obtain a human body segmentation result of the first global image; The method comprises: performing target detection on the first global image to obtain a target detection result of the first global image, the target detection result being used to indicate the position of the human body frame included in the first global image; determining scene information of the first global image and distance information corresponding to each human body frame in the first global image according to the target detection result of the first global image, the scene information being used to indicate whether the first global image is a global image in a single-person scene or a global image in a multi-person scene, and the distance information being used to indicate the distance between the human body in the human body frame and a first image acquisition device, the first image acquisition device being used to acquire the first global image; for each human body frame, obtaining the movement parameter corresponding to the human body frame according to the scene information, the distance information corresponding to the human body frame, and a first preset mapping relationship, the first preset mapping relationship being used to indicate the movement parameter corresponding to the human body frame in different scenes and at different distances.

2. The method of claim 1, wherein, The method further comprises: obtaining a first video, the first video corresponding to a first scene, the first scene being a single-person scene or a multi-person scene, and the first video being used to record the movement of a single person or multiple persons in a first movement range, wherein the first movement range is at a first distance from a second image acquisition device, and the second image acquisition device is used to acquire the first video; tracking the limbs of a target person in the first video to obtain the position of a human body frame corresponding to the target person in each frame of image of the first video; determining the movement speed, the movement amplitude, and a second distance of the target person according to the position of the human body frame corresponding to the target person in each frame of image of the first video, the second distance being used to indicate the distance between the target person and a reference position of the first video; obtaining a first movement parameter according to the movement speed and the movement amplitude of the target person; establishing the first preset mapping relationship based on the first scene, the second distance, and the first movement parameter.

3. The method of claim 2, wherein, The method further comprises: in the case where the first scene is a single-person scene, setting the movement frequency of the person in the first video to be greater than a first frequency threshold, and the length of the projection of the first movement range in the x-axis direction of the camera coordinate system of the second image acquisition device to be greater than a first movement threshold. In a case where the first scene is a multi-person scene, the moving frequency of the person in the first video is set to be less than or equal to a second frequency threshold, and a length of a projection of the first moving range in an x-axis direction of a camera coordinate system of the second image acquisition device is less than or equal to a second moving threshold; wherein the second frequency threshold is less than or equal to the first frequency threshold, and the second moving threshold is less than or equal to the first moving threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The scaling of the human body frame according to the moving parameter corresponding to the human body frame to obtain a target frame corresponding to the human body frame comprises: determining a scaling coefficient of the human body frame according to the moving parameter corresponding to the human body frame and a second preset mapping relationship; scaling the human body frame according to the scaling coefficient to obtain the target frame corresponding to the human body frame, the second preset mapping relationship being used to indicate a scaling coefficient corresponding to different moving parameters.

5. The method according to any one of claims 1 to 3, characterized in that, The determination of a first target image according to the target frame corresponding to each human body frame and the first global image comprises: merging the target frame corresponding to each human body frame to obtain a merged frame; obtaining the first target image according to the merged frame and the first global image; wherein the first target image corresponds to a merged frame with the smallest area among merged frames capable of covering the target frame corresponding to all human body frames.

6. The method according to any one of claims 1 to 3, characterized in that, The image segmentation of the first target image to obtain a human body segmentation result of the first global image comprises: obtaining a second target image corresponding to a second global image, the second global image being a previous frame image of the first global image in a video, the second target image representing a target image used when obtaining a human body segmentation result of the second global image; determining a moving amplitude of the first target image relative to the second target image; in a case where the moving amplitude of the first target image relative to the second target image is greater than a first amplitude threshold, performing image segmentation on the first target image to obtain the human body segmentation result of the first global image.

7. The method of claim 6, wherein, The method further comprises: in a case where the moving amplitude of the first target image relative to the second target image is less than or equal to the first amplitude threshold, performing image segmentation on the second target image to obtain the human body segmentation result of the first global image.

8. The method according to any one of claims 1 to 3, characterized in that, The image segmentation of the first target image to obtain a human body segmentation result of the first global image comprises: obtaining a second target image corresponding to a second global image, the second global image and the first global image belonging to a same video, and the second global image being a previous frame image of the first global image, the second target image representing a target image used when obtaining a human body segmentation result of the second global image; determining a coverage rate of the first target image relative to the second target image; in a case where the coverage rate is less than a second amplitude threshold, performing image segmentation on the first target image to obtain the human body segmentation result of the first global image.

9. The method of claim 8, wherein, The method further comprises: In a case where the coverage is greater than or equal to the second amplitude threshold, image segmentation is performed on the second target image to obtain a human body segmentation result of the first global image.

10. An image segmentation apparatus characterized by comprising: The device comprises: A first obtaining module is configured to obtain a movement parameter corresponding to each human body bounding box in a first global image, the movement parameter being used to represent a movement law or a movement trend corresponding to the human body bounding box. A scaling module is configured to, for each human body bounding box, scale the human body bounding box according to the movement parameter obtained by the first obtaining module to obtain a target bounding box corresponding to the human body bounding box. A first determining module is configured to determine a first target image according to the target bounding box corresponding to each human body bounding box obtained by the scaling module and the first global image. A first segmentation module is configured to perform image segmentation on the first target image determined by the first determining module to obtain a human body segmentation result of the first global image. The first obtaining module is further configured to: perform target detection on the first global image to obtain a target detection result of the first global image, the target detection result being used to indicate a position of a human body bounding box included in the first global image; determine, according to the target detection result of the first global image, scene information of the first global image and distance information corresponding to each human body bounding box in the first global image, the scene information being used to indicate whether the first global image is a global image in a single-person scene or a global image in a multi-person scene, and the distance information being used to indicate a distance between a human body in the human body bounding box and a first image acquisition device, the first image acquisition device being used to acquire the first global image; for each human body bounding box, obtain a movement parameter corresponding to the human body bounding box according to the scene information, the distance information corresponding to the human body bounding box, and a first preset mapping relationship, the first preset mapping relationship being used to indicate the movement parameter corresponding to the human body bounding box in different scenes and at different distances.

11. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method in any one of claims 1 to 9.

12. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method in any one of claims 1 to 9. The computer program instructions, when executed by the processor, implement the method in any one of claims 1 to 9.

Images