A human body image segmentation method and device

Abstract

The present application provides a human body image segmentation method and device, in the present application embodiment, the electronic equipment determines the candidate pixel point corresponding to the human body in the RGB image, and then obtains the first pixel point corresponding to each candidate pixel point in the first image obtained through the point cloud data mapping, and determines a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image, and determines the first gradient value corresponding to each candidate pixel point in each first preset direction. The electronic equipment determines the target pixel point corresponding to the human body according to the first gradient value corresponding to each candidate pixel point in each first preset direction. That is, the electronic equipment further verifies the candidate pixel point corresponding to the human body in the RGB image determined according to the point cloud data, avoids the over-segmentation phenomenon in the human body image segmentation process, and improves the accuracy of human body image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a human body image segmentation method and device. BACKGROUND

[0002] The current human body instance segmentation method (such as the Mask-RCNN instance segmentation network method) can segment the image of a single human body from an RGB image.

[0003] However, the quality of the input image and the algorithm itself can cause over-segmentation. That is, the existing human body instance segmentation method may incorrectly segment parts that do not belong to the human body as human body parts, resulting in poor human body image segmentation effect. SUMMARY

[0004] The present application provides a human body image segmentation method and device to solve the problem of over-segmentation and poor human body image segmentation effect in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a human body image segmentation method, which comprises the following steps:

[0006] Obtaining point cloud data collected by a laser radar sensor and an RGB image collected by an image collection device, and mapping the point cloud data to a first image consistent in size with the RGB image;

[0007] Using a preset human body instance segmentation algorithm to perform human body image segmentation on the RGB image to determine candidate pixel points corresponding to the human body in the RGB image;

[0008] For each candidate pixel point, determining a first pixel point in the first image corresponding to the candidate pixel point, and determining a preset number of second pixel points in the first image adjacent to the first pixel point in each first preset direction; and determining a first gradient value of the candidate pixel point in each first preset direction according to the pixel values of the first pixel point and the second pixel points in the first image;

[0009] Determining target pixel points corresponding to the human body according to the first gradient value of each candidate pixel point in each first preset direction.

[0010] In a second aspect, the embodiments of the present application further provide an electronic device, which comprises a processor configured to implement the steps of the human body image segmentation method as described above when executing a computer program stored in a memory.

[0011] In the embodiment of the present application, the point cloud data collected by the laser radar sensor and the RGB image collected by the image collection device are obtained, and the point cloud data is mapped into a first image consistent with the size of the RGB image; a preset human instance segmentation algorithm is used to perform human image segmentation on the RGB image to determine the candidate pixel points corresponding to the human in the RGB image; for each candidate pixel point, a first pixel point corresponding to the candidate pixel point in the first image is determined, and a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image are determined; according to the pixel values corresponding to the first pixel point and the second pixel points in the first image, a first gradient value of the candidate pixel point in each first preset direction is determined; according to the first gradient value of each candidate pixel point in each first preset direction, a target pixel point corresponding to the human is determined. In the embodiment of the present application, the electronic device determines the candidate pixel points corresponding to the human in the RGB image, and then determines the first pixel point corresponding to each candidate pixel point in the first image obtained by mapping the point cloud data, and determines a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image, to determine the first gradient value corresponding to each candidate pixel point in each first preset direction. The electronic device determines the target pixel point corresponding to the human according to the first gradient value corresponding to each candidate pixel point in each first preset direction. That is, the electronic device further verifies the candidate pixel points corresponding to the human in the RGB image determined according to the point cloud data, avoids the over-segmentation phenomenon in the human image segmentation process, and improves the accuracy of human image segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0013] Figure 1 A human image segmentation process schematic diagram provided for the embodiment of the present application;

[0014] Figure 2 A point cloud data mapping schematic diagram provided for the embodiment of the present application;

[0015] Figure 3 A schematic diagram of the second preset direction provided for the embodiment of the present application;

[0016] Figure 4 A schematic diagram of the first preset direction provided for the embodiment of the present application;

[0017] Figure 5A flowchart of a human image segmentation method provided by an embodiment of the present application is shown in FIG. 1.

[0018] Figure 6 A flowchart of a human image segmentation method provided by an embodiment of the present application is shown in FIG. 1.

[0019] Figure 7 A structure diagram of a human image segmentation device provided by an embodiment of the present application is shown in FIG. 2.

[0020] Figure 8 A structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0022] In order to improve the accuracy of human image segmentation and avoid over-segmentation in the process of human image segmentation, the present application provides a human image segmentation method and device.

[0023] In an embodiment of the present application, an electronic device obtains point cloud data collected by a laser radar sensor and an RGB image collected by an image collection device, and maps the point cloud data to a first image with the same size as the RGB image. A preset human instance segmentation algorithm is used to perform human image segmentation on the RGB image to determine candidate pixel points corresponding to a human in the RGB image. For each candidate pixel point, a first pixel point in the first image corresponding to the candidate pixel point is determined, and a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image are determined. A first gradient value of the candidate pixel point in each first preset direction is determined according to pixel values of the first pixel point and the second pixel points in the first image. A target pixel point corresponding to the human is determined according to the first gradient value of each candidate pixel point in each first preset direction.

[0024] Figure 1 A process diagram of a human image segmentation method provided by an embodiment of the present application is shown in FIG. 4, which includes the following steps.

[0025] S101: Obtain point cloud data collected by a laser radar sensor and an RGB image collected by an image collection device, and map the point cloud data to a first image with the same size as the RGB image.

[0026] The human body image segmentation method provided in the embodiments of the present application is applied to an electronic device, which can be a PC or a server.

[0027] In the embodiments of the present application, the laser radar sensor emits laser pulses, and the laser pulses are reflected back to the laser radar sensor after reaching the surface of the hand, and the laser radar sensor calculates the three-dimensional spatial position of the hand according to the interval between the emission and reception times. The laser radar sensor emits a large number of laser pulses in different directions and receives echoes, that is, three-dimensional point cloud data is formed. That is, the point cloud data is composed of a large number of three-dimensional spatial points.

[0028] In addition, in the embodiments of the present application, the RGB camera performs real-time image acquisition to obtain an RGB image containing a human body.

[0029] In addition, in the embodiments of the present application, when the RGB camera and the laser radar sensor acquire data, the RGB camera and the laser radar sensor are strictly aligned in a plane perpendicular to the sensor visual axis, that is, the point cloud points in the point cloud data and the pixel points corresponding to the point cloud points in the RGB image both indicate the same target object, and the field of view ranges of the RGB camera and the laser radar sensor are the same. The human body stands in front of the laser radar sensor and the RGB camera, rather than below. That is, the RGB camera and the laser radar sensor are approximately at a horizontal viewing angle, rather than a strict overhead viewing angle.

[0030] Since the information of each pixel point in the RGB image includes a coordinate position and a pixel value, but the coordinate position contained in the RGB image is a two-dimensional position, in order to better segment the human body, the point cloud data can be generated into a raster image and mapped to the RGB image, so that the RGB image has a dimension information parallel to the direction of the sensor visual axis, that is, the coordinate position contained in the RGB image is changed to a three-dimensional position.

[0031] Based on this, in the embodiments of the present application, after the electronic device acquires the RGB image and the point cloud data, the electronic device maps the point cloud data into a first image consistent with the size of the RGB image.

[0032] Specifically, in the embodiments of the present application, the coordinate origins of the point cloud data and the RGB image are both arranged at the lower left corner of the image, the horizontal direction to the right is the positive direction of the X axis, and the vertical direction upward is the positive direction of the Y axis. The electronic device establishes an empty raster image with the same number of pixel rows as the RGB image, the same number of pixel columns as the RGB image, and the same coordinate origin as the RGB image, wherein the initial value of the number of pixel points contained in each raster is 0. The electronic device one-to-one maps each point cloud point in the point cloud data to the empty raster image according to the plane coordinates, and determines the mapped image as the first image.

[0033] In the embodiments of the present application, the number of point cloud points corresponding to each pixel point in the first image can be 0, 1, 2 or a larger value.

[0034] Figure 2 A schematic diagram of point cloud data mapping provided by the embodiments of the present application is shown in the figure. Figure 2 As shown in the figure, the electronic device maps each point cloud point in the point cloud data to an empty point cloud grid image according to the plane coordinates.

[0035] S102: A preset human instance segmentation algorithm is used to perform human image segmentation on the RGB image, and candidate pixel points corresponding to the human in the RGB image are determined.

[0036] In the embodiments of the present application, the electronic device will use a preset human instance segmentation algorithm to perform human image segmentation on the RGB image and determine the pixel points corresponding to the human in the RGB image.

[0037] Specifically, in the embodiments of the present application, the electronic device can use a Mask-RCNN segmentation method or the like to perform human instance segmentation on the RGB image.

[0038] S103: For each candidate pixel point, a first pixel point corresponding to the candidate pixel point in the first image is determined, and a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image are determined; and a first gradient value of the candidate pixel point in each first preset direction is determined according to the pixel values corresponding to the first pixel point and the second pixel points in the first image.

[0039] In the embodiments of the present application, after the electronic device determines the candidate pixel points corresponding to the human in the RGB image, the electronic device can further verify each candidate pixel point according to the point cloud points corresponding to each pixel point in the first image, and determine the gradient value of the candidate pixel point in each first preset direction. The gradient value can distinguish the candidate pixel point from other pixel points in the corresponding first preset direction, and the greater the gradient value, the greater the probability that the candidate pixel point is the pixel point corresponding to the human.

[0040] Specifically, in the embodiments of the present application, for each candidate pixel point, the electronic device determines the location information of the candidate pixel point in the RGB image, and finds the pixel point corresponding to the location information in the first image. The electronic device determines the first pixel point corresponding to the candidate pixel point as the pixel point. The electronic device determines a preset number of second pixel points adjacent to the first pixel point in each preset direction in the first image, and determines the first gradient value of the candidate pixel point in each first preset direction according to the pixel values corresponding to the first pixel point and the preset number of second pixel points in each first preset direction.

[0041] The electronic device calculates, for each first preset direction, a difference value of pixel values of each adjacent two of the preset number of second pixel points in the first preset direction from the first pixel point, and determines an average value of the difference value as a first gradient value of the candidate pixel point corresponding to the first pixel point in the first preset direction.

[0042] For example, the electronic device determines that the pixel values of the first pixel point and the preset number of second pixel points in the first preset direction are e1, e2, …, e k The electronic device determines the gradient value as (|e1-e2|+|e2-e3|+…+|e k-1 k ) / (k-1).

[0043] It should be noted that in the embodiments of the present application, the first preset direction is set by the technician according to experience and saved in the electronic device, which is not limited herein.

[0044] S104: determining a target pixel point corresponding to the human body according to the first gradient value of each candidate pixel point in each first preset direction.

[0045] In the embodiments of the present application, after the electronic device determines the first gradient value of each candidate pixel point in each first preset direction, the electronic device can determine whether each candidate pixel point is a target pixel point corresponding to the human body according to the first gradient value of each candidate pixel point in each first preset direction.

[0046] Specifically, in the embodiments of the present application, the electronic device can save a gradient value threshold corresponding to each first preset direction. For each candidate pixel point, if there is a first gradient value greater than the corresponding gradient value threshold that meets the preset number requirement among the first gradient values of the candidate pixel point in each first preset direction, the electronic device determines that the candidate pixel point is a target pixel point corresponding to the human body.

[0047] ​In the embodiment of the present application, the electronic device determines the candidate pixel points corresponding to the human body in the RGB image, then obtains the first pixel points corresponding to each candidate pixel point in the first image through the mapping of the point cloud data, and determines the first pixel points adjacent to the first pixel points in a preset number in each first preset direction in the first image, and determines the first gradient value corresponding to each candidate pixel point in each first preset direction. The electronic device determines the target pixel points corresponding to the human body according to the first gradient value corresponding to each candidate pixel point in each first preset direction. That is, the electronic device further verifies the candidate pixel points corresponding to the human body in the RGB image according to the point cloud data, avoids the over-segmentation phenomenon in the human body image segmentation process, and improves the accuracy of human body image segmentation.

[0048] In order to determine the pixel value corresponding to each pixel point in the first image, on the basis of the above-mentioned embodiment, in the embodiment of the present application, after the point cloud data is mapped into the first image consistent with the size of the RGB image, the method further comprises:

[0049] For each pixel point in the first image, the number of point cloud points mapped to the pixel point is determined; if the number of point cloud points is non-zero, the first average value of the vertical coordinate values of the point cloud points corresponding to the pixel point is determined, and the first average value is determined as the pixel value corresponding to the pixel point;

[0050] If the number of point cloud points is zero, each third pixel point in the first image adjacent to the pixel point in each second preset direction and having a non-zero number of point cloud points is determined; the second average value corresponding to the pixel value of each third pixel point is determined, and the second average value is determined as the pixel value corresponding to the pixel point.

[0051] In the embodiment of the present application, after the point cloud data is mapped into the first image consistent with the size of the RGB image, the electronic device determines the pixel value corresponding to each pixel point in the first image based on the number of point cloud points contained by each pixel point.

[0052] Specifically, in the embodiment of the present application, for each pixel point in the first image, the electronic device determines the number of point cloud points mapped to the pixel point, which can be zero or non-zero. If the number of point cloud points corresponding to the pixel point is non-zero, the electronic device determines the first average value of the vertical coordinates of the point cloud points mapped to the pixel point, and determines the first average value as the pixel value corresponding to the pixel point. The vertical coordinate is the vertical distance from the point cloud point to the laser radar sensor, and the vertical coordinate is used to identify the front and back distance of the point cloud point.

[0053] If the electronic device determines that the number of point cloud points corresponding to the pixel point is zero, the electronic device determines the pixel value of the pixel point based on each third pixel point adjacent to the pixel point in each second preset direction and having a non-zero number of point cloud points. Specifically, in the embodiments of the present application, the electronic device determines each third pixel point adjacent to the pixel point in each second preset direction and having a non-zero number of point cloud points, and determines a second average value corresponding to the pixel value of each third pixel point. The electronic device determines the second average value as the pixel value corresponding to the pixel point.

[0054] For example, if the number n of point cloud points mapped to a certain pixel point is 1, i.e. n = 1, and the vertical coordinate value of the point cloud point is z, then the pixel value v of the pixel point is z; if n is greater than 1, i.e. n > 1, and the vertical coordinate values of the n point cloud points mapped to the pixel are z1, z2, …, zn, respectively, then the pixel value v of the pixel point is (z1+ z2+ …+ zn) / n. n n After the electronic device determines the pixel value corresponding to the pixel point in the first image and having a non-zero number of corresponding point cloud points, if the number of corresponding point cloud points of the remaining pixel points in the first image whose pixel values are not determined is zero, i.e. n = 0, then for each pixel point having a zero number of corresponding point cloud points, the electronic device obtains the pixel values v u , v d , v l , and v r of the pixel points adjacent to the pixel point in each second preset direction, and determines the first average value of v u , v d , v l , and v r as the value of the pixel point, i.e. v = (v u + v d + v l + v r ) / 4.

[0055] In the embodiments of the present application, if the number of point cloud points corresponding to the pixel points adjacent to the pixel point in a certain second preset direction is zero, the electronic device determines the pixel value of the pixel point closest to the pixel point in the second preset direction and having a non-zero number of corresponding point cloud points, and determines the pixel value of the pixel point based on the pixel value. However, in order to improve the accuracy of the calculation, in the embodiments of the present application, the electronic device needs to consider the influence of the distance between the two pixel points when calculating based on the pixel value.

[0056] Specifically, the electronic device can calculate by the following formula: v = (v1 / p1+ vd+ vl+ vr) / (3+1 / p1), where v is the pixel value of the pixel point to be calculated, v u , v​d , v l and v r are pixel values of pixel points adjacent to the pixel point and corresponding to the pixel point in each second preset direction, and v1 corresponds to a pixel point spaced from the pixel point by p pixel points.

[0057] If the number of point cloud points corresponding to the pixel points adjacent to the pixel point in two second preset directions is zero, the electronic device determines pixel values of pixel points closest to the pixel point and corresponding to the pixel point in the two second preset directions, and determines the pixel value of the pixel point according to the pixel values.

[0058] For example, the electronic device determines values v1 and v2 of pixel points closest to the pixel point and corresponding to the pixel point in the two second preset directions, and the two pixel points are spaced from the pixel point by p1 and p2 pixels respectively, and the pixel value v corresponding to the pixel point is (v1 / p1+v2 / p2+v l + v r ) / (2+1 / p1+1 / p2).

[0059] If the number of point cloud points corresponding to the pixel points adjacent to the pixel point in three second preset directions is zero, the electronic device determines pixel values of pixel points closest to the pixel point and corresponding to the pixel point in the three second preset directions, and determines the pixel value of the pixel point according to the pixel values.

[0060] For example, the electronic device determines values v1, v2 and v3 of pixel points closest to the pixel point and corresponding to the pixel point in the three second preset directions, and the three pixel points are spaced from the pixel point by p1, p2 and p3 pixels respectively, and the pixel value v corresponding to the pixel point is (v1 / p1+v2 / p2+v3 / p3+v r ) / (1+1 / p1+1 / p2+1 / p3).

[0061] In the embodiments of the present application, after the pixel value corresponding to each pixel point in the first image is determined, the pixel value indicates the vertical distance of the corresponding pixel point to the laser radar sensor, wherein the smaller the pixel value, the farther the relative distance of the pixel point to the sensor, and the larger the pixel value, the closer the relative distance of the pixel point to the sensor.

[0062] In order to better determine the pixel value corresponding to each pixel point in the first image, on the basis of the above embodiments, in the embodiments of the present application, the second preset direction includes: X-axis positive direction, X-axis negative direction, Y-axis positive direction and Y-axis negative direction.

[0063] In order to better determine the pixel value corresponding to each pixel point in the first image and improve the accuracy of the determined pixel value, in the embodiments of the present application, the second preset direction includes: the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis and the negative direction of the Y axis.

[0064] Figure 3 A schematic diagram of the second preset direction provided for the embodiments of the present application is shown in FIG. 2. Figure 3 As shown in FIG. 2, the second preset direction includes: the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis and the negative direction of the Y axis.

[0065] In order to better determine the first gradient value corresponding to each candidate pixel point, on the basis of the above embodiments, in the embodiments of the present application, the first preset direction includes: the first angle bisector direction corresponding to the positive direction of the X axis and the positive direction of the Y axis, the second angle bisector direction corresponding to the positive direction of the X axis and the negative direction of the Y axis, the third angle bisector direction corresponding to the negative direction of the X axis and the positive direction of the Y axis, the fourth angle bisector direction corresponding to the negative direction of the X axis and the negative direction of the Y axis, the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis and the negative direction of the Y axis.

[0066] In the embodiments of the present application, when the electronic device determines the preset number of second pixel points adjacent to each candidate pixel point in each first preset direction in the first image, the first preset direction is the first angle bisector direction corresponding to the positive direction of the X axis and the positive direction of the Y axis, the second angle bisector direction corresponding to the positive direction of the X axis and the negative direction of the Y axis, the third angle bisector direction corresponding to the negative direction of the X axis and the positive direction of the Y axis, the fourth angle bisector direction corresponding to the negative direction of the X axis and the negative direction of the Y axis, the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis and the negative direction of the Y axis.

[0067] Figure 4 A schematic diagram of the first preset direction provided for the embodiments of the present application is shown in FIG. 3. Figure 4 As shown in FIG. 3, the first preset direction includes: the first angle bisector direction corresponding to the positive direction of the X axis and the positive direction of the Y axis, the second angle bisector direction corresponding to the positive direction of the X axis and the negative direction of the Y axis, the third angle bisector direction corresponding to the negative direction of the X axis and the positive direction of the Y axis, the fourth angle bisector direction corresponding to the negative direction of the X axis and the negative direction of the Y axis, the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis and the negative direction of the Y axis.

[0068] In order to better determine the first gradient value corresponding to each candidate pixel point, on the basis of the above embodiments, in the embodiments of the present application, the step of determining the first gradient value of the candidate pixel point in each first preset direction according to the pixel values corresponding to the first pixel point and the second pixel point includes:

[0069] For each first preset direction, a pixel value difference value of each adjacent two pixel points in the second pixel point of the first preset direction is determined; a third mean value of each pixel value difference value is determined, and the third mean value is determined as the first gradient value of the candidate pixel point in the first preset direction.

[0070] In the embodiments of the present application, when the electronic device determines the first gradient value of the first pixel point corresponding to the candidate pixel point in each first preset direction, the electronic device determines the pixel value difference value of each adjacent two pixel points in the second pixel point of the first preset direction; a third mean value of each pixel value difference value is determined, and the third mean value is determined as the first gradient value of the candidate pixel point in the first preset direction.

[0071] For example, for each first preset direction, the electronic device obtains a preset number k of second pixel points in the first preset direction. Assuming that the first preset direction is the positive direction of the Y axis, the coordinates of the k second pixel points are (x, y+1), (x, y+2), …, (x, y+k) respectively, and the pixel values of the k second pixel points are e1, e2, …, e k Then the electronic device determines the first gradient value E of the candidate pixel point corresponding to the first pixel point as E=(|e1-e2|+|e2-e3|+……+|e k-1 k ) / (k-1).

[0072] Wherein, the electronic device can calculate the first gradient value of the candidate pixel point in each first preset direction by using the above formula.

[0073] In order to better determine the target pixel point corresponding to the human body, on the basis of the above embodiments, in the embodiments of the present application, the target pixel point corresponding to the human body is determined according to the first gradient value of each candidate pixel point in each first preset direction, comprising:

[0074] Determine each target candidate pixel point whose at least one first gradient value is greater than the corresponding first gradient value threshold;

[0075] Topologically connect each target candidate pixel point and the fourth pixel point in the RGB image except the candidate pixel point corresponding to the human body, and determine at least one connected domain;

[0076] Determine the pixel point contained in the connected domain with the largest area as the target pixel point corresponding to the human body.

[0077] ​In the embodiment of the present application, after determining the first gradient value of each candidate pixel point in each first preset direction, the electronic device determines whether each candidate pixel point is an error segmented pixel point according to the pre-stored first gradient value threshold corresponding to each first preset direction, that is, identifies the over-segmented pixel point in each candidate pixel point.

[0078] Specifically, in the embodiment of the present application, for each candidate pixel point, the electronic device compares the first gradient value of the candidate pixel point in each first preset direction with the corresponding first gradient value threshold, and if there is at least one first gradient value greater than the corresponding first gradient value threshold, the electronic device determines that the candidate pixel point is an over-segmented target candidate pixel point.

[0079] The electronic device topologically connects each target candidate pixel point and the fourth pixel point in the RGB image except the candidate pixel point corresponding to the human body to determine at least one connected domain. The electronic device determines the largest area connected domain, and determines the pixel points contained in the largest area connected domain as the target pixel points corresponding to the human body.

[0080] Wherein, in order to reduce the load pressure and the calculation pressure of the electronic device, in the embodiment of the present application, the first angle bisector direction corresponding to the X-axis positive direction and the Y-axis positive direction in the first preset direction, the second angle bisector direction corresponding to the X-axis positive direction and the Y-axis negative direction, the third angle bisector direction corresponding to the X-axis negative direction and the Y-axis positive direction, and the fourth angle bisector direction corresponding to the X-axis negative direction and the Y-axis negative direction can use the same first gradient value threshold; the X-axis positive direction, the X-axis negative direction, the Y-axis positive direction and the Y-axis negative direction can use the same first gradient value threshold.

[0081] Figure 5 A flowchart of a human body image segmentation process provided in the embodiment of the present application is shown in the figure. Figure 5 As shown in the figure, the process includes:

[0082] S501: Obtain the point cloud data collected by the laser radar sensor and the RGB image collected by the image collection device, and map the point cloud data to a first image consistent with the size of the RGB image.

[0083] S502: Using a preset human instance segmentation algorithm, perform human body image segmentation on the RGB image to determine the candidate pixel points corresponding to the human body in the RGB image.

[0084] S503: For each candidate pixel point, determine the first gradient value of the candidate pixel point in each first preset direction.

[0085] S504: Determine each target candidate pixel point with at least one first gradient value greater than the corresponding first gradient value threshold.

[0086] S505: Topologically connect each target candidate pixel point and the fourth pixel point in the RGB image except the candidate pixel point corresponding to the human body, to determine at least one connected domain.

[0087] S506: Determine the pixel points contained in the connected domain with the largest area as the target pixel points corresponding to the human body.

[0088] To improve the accuracy of human body image segmentation, on the basis of the above embodiments, in the embodiments of the present application, the method further comprises:

[0089] determining a fifth pixel point in the target pixel points as a human body edge;

[0090] For each fifth pixel point, determine a preset number of sixth pixel points in the RGB image adjacent to the fifth pixel point in each first preset direction; for each first preset direction, determine a second gradient value of the fifth pixel point in the first preset direction according to the pixel values of the fifth pixel point and the preset number of sixth pixel points in the RGB image;

[0091] According to each second gradient value of each fifth pixel point in each first preset direction, determine the target other pixel points corresponding to the human body in the other pixel points in the RGB image except the target pixel points.

[0092] In the existing human instance segmentation method, in addition to the possibility of over-segmentation of the human body by the electronic device, there may also be under-segmentation phenomenon, that is, the electronic device may also incorrectly segment the part belonging to the human body as a non-human body part.

[0093] Based on this, in order to solve the under-segmentation problem in the prior art and improve the accuracy of human body image segmentation, on the basis of the above embodiments, in the embodiments of the present application, after the electronic device determines the target pixel points corresponding to the human body, it further screens other pixel points in the RGB image except the target pixel points corresponding to the human body, to determine the target other pixel points that are under-segmented in the other pixel points.

[0094] Specifically, in the embodiments of the present application, the electronic device determines a fifth pixel point in the target pixel points as a human body edge. For each fifth pixel point, the electronic device determines a preset number of sixth pixel points in the RGB image adjacent to the fifth pixel point in each first preset direction. For each first preset direction, the electronic device determines a second gradient value of the fifth pixel point in the first preset direction according to the pixel values of the fifth pixel point and the preset number of sixth pixel points in the RGB image. The electronic device determines the target other pixel points corresponding to the human body in the other pixel points in the RGB image except the target pixel points according to each second gradient value of each fifth pixel point in each first preset direction.

[0095] In the embodiment of the present application, the method for determining the second gradient value of the fifth pixel point in each first preset direction by the electronic device is the same as the method for determining the first gradient value of the candidate pixel point in each first preset direction by the electronic device in the above-mentioned embodiments, and will not be described here.

[0096] In order to improve the accuracy of human image segmentation, on the basis of the above-mentioned embodiments, in the embodiment of the present application, the target other pixel point corresponding to the human in the other pixel points in the RGB image except the target pixel point is determined according to each second gradient value of each fifth pixel point in each first preset direction.

[0097] For each fifth pixel point, if the second gradient value of the target first preset direction corresponding to the fifth pixel point is less than the second gradient value threshold corresponding to the target first preset direction, the other pixel points adjacent to the fifth pixel point in the target first preset direction are determined as the target other pixel points corresponding to the human.

[0098] In the embodiment of the present application, after determining the second gradient value of each fifth pixel point in each first preset direction, the electronic device determines whether each candidate pixel point is an error segmented pixel point according to the pre-saved second gradient value threshold corresponding to each first preset direction, that is, identifies the under-segmented pixel point in each fifth pixel point.

[0099] Specifically, for each fifth pixel point, if the second gradient value of the target first preset direction corresponding to the fifth pixel point is less than the second gradient value threshold corresponding to the target first preset direction, the other pixel points adjacent to the fifth pixel point in the target first preset direction are determined as the target other pixel points corresponding to the human.

[0100] In the embodiment of the present application, the second gradient value threshold corresponding to each first preset direction can be the same as the first gradient value threshold corresponding to the first preset direction.

[0101] Figure 6 The flowchart of human image segmentation for under-segmentation problem provided by the embodiment of the present application is shown as follows. Figure 6 The process includes:

[0102] S601: Determine the fifth pixel point in the target pixel point which is the edge of the human.

[0103] S602: For each fifth pixel point, determine the second gradient value of the fifth pixel point in the first preset direction.

[0104] S603: For each fifth pixel point, if the second gradient value of the target first preset direction corresponding to the fifth pixel point is less than the second gradient value threshold corresponding to the target first preset direction, it is determined that the target first preset direction and other pixels adjacent to the fifth pixel point are target other pixels corresponding to the human body.

[0105] In order to realize the segmentation of the human body, on the basis of the above embodiments, in the embodiments of the present application, the method further comprises:

[0106] The target pixel point and the target other pixel point are determined and segmented in the RGB image, and a sub-RGB image obtained by segmentation is determined as an RGB image corresponding to the human body.

[0107] In the embodiments of the present application, after the electronic device determines the target pixel point and the target other pixel point corresponding to the human body, the electronic device determines and segments the target pixel point and the target other pixel point in the RGB image, and determines a sub-RGB image obtained by segmentation as an RGB image corresponding to the human body.

[0108] Figure 7 A human body image segmentation device structure schematic diagram is provided in the embodiments of the present application, and the device comprises:

[0109] The acquisition module 701 is configured to acquire point cloud data collected by a laser radar sensor and an RGB image collected by an image acquisition device, and map the point cloud data to a first image with a size consistent with that of the RGB image.

[0110] The processing module 702 is configured to perform human body image segmentation on the RGB image by using a preset human instance segmentation algorithm, to determine candidate pixel points corresponding to the human body in the RGB image; for each candidate pixel point, determine a first pixel point corresponding to the candidate pixel point in the first image, and determine a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image; determine a first gradient value of the candidate pixel point in each first preset direction according to pixel values corresponding to the first pixel point and the second pixel points in the first image; and determine a target pixel point corresponding to the human body according to the first gradient value of each candidate pixel point in each first preset direction.

[0111] In a possible implementation, the processing module 702 is further configured to determine, for each pixel point in the first image, a number of point cloud points mapped to the pixel point; determine a first average value of vertical coordinate values of the point cloud points corresponding to the pixel point if the number of point cloud points is non-zero, and determine the first average value as a pixel value corresponding to the pixel point; and determine a number of third pixel points in the first image that are adjacent to the pixel point in each second preset direction and have non-zero number of point cloud points, if the number of point cloud points is zero; determine a second average value corresponding to pixel values of the third pixel points, and determine the second average value as the pixel value corresponding to the pixel point.

[0112] In a possible implementation, the second preset direction includes an X-axis positive direction, an X-axis negative direction, a Y-axis positive direction, and a Y-axis negative direction.

[0113] In a possible implementation, the first preset direction includes a first angle bisector direction corresponding to an X-axis positive direction and a Y-axis positive direction, a second angle bisector direction corresponding to the X-axis positive direction and a Y-axis negative direction, a third angle bisector direction corresponding to an X-axis negative direction and the Y-axis positive direction, a fourth angle bisector direction corresponding to the X-axis negative direction and the Y-axis negative direction, the X-axis positive direction, the X-axis negative direction, the Y-axis positive direction, and the Y-axis negative direction.

[0114] In a possible implementation, the processing module 702 is specifically configured to determine, for each first preset direction, a pixel value difference value of each adjacent two pixel points in the second pixel points in the first preset direction; determine a third average value of each pixel value difference value, and determine the third average value as a first gradient value of the candidate pixel point in the first preset direction.

[0115] In a possible implementation, the processing module 702 is specifically configured to determine each target candidate pixel point with at least one first gradient value greater than a corresponding first gradient value threshold; topologically connect each target candidate pixel point and fourth pixel points in the RGB image except the candidate pixel points corresponding to the human body, to determine at least one connected domain; and determine pixel points included in a connected domain with the largest area as target pixel points corresponding to the human body.

[0116] In one possible implementation, the processing module 702 is further configured to: determine a fifth pixel among the target pixels that is a human body edge; for each fifth pixel, determine a preset number of sixth pixels in the RGB image that are adjacent to the fifth pixel in each first preset direction; for each first preset direction, determine a second gradient value of the fifth pixel in the first preset direction based on the pixel values ​​of the fifth pixel and the preset number of sixth pixels in the RGB image; and based on each second gradient value of the fifth pixel in each first preset direction, determine other target pixels in the RGB image other than the target pixel that correspond to the human body.

[0117] In one possible implementation, the processing module 702 is specifically used to determine, for each fifth pixel, other pixels adjacent to the fifth pixel in the first preset direction of the target as other pixels corresponding to the human body if the second gradient value of the target in the first preset direction of the target is less than the threshold value of the second gradient value corresponding to the target in the first preset direction of the target.

[0118] In one possible implementation, the processing module 702 is specifically used to determine and segment the target pixel and other target pixels in the RGB image, and to determine the segmented sub-RGB image as the RGB image corresponding to the human body.

[0119] Based on the above embodiments, this application also provides an electronic device. Figure 8 This application provides a schematic diagram of an electronic device structure, such as... Figure 8 As shown, it includes: processor 801, communication interface 802, memory 803 and communication bus 804, wherein processor 801, communication interface 802 and memory 803 communicate with each other through communication bus 804.

[0120] The memory 803 stores a computer program, which, when executed by the processor 801, causes the processor 801 to perform the steps as described in the above embodiments.

[0121] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the human image segmentation method, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated details will not be repeated.

[0122] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 802 is used for communication between the above electronic device and other devices. The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0123] The processor mentioned above can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; can also be a Digital Signal Processing (DSP) processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0124] On the basis of the above embodiments, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program executable by a processor. When the program runs on the processor, the processor executes the steps of the above embodiments.

[0125] Since the principle of solving the problem of the above computer readable storage medium is similar to the human body image segmentation method, the implementation of the above computer readable storage medium can refer to the embodiment of the method, and the repeated parts will not be described again.

[0126] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0127] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0128] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0129] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0130] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A human body image segmentation method characterized by comprising: The method comprises: acquiring point cloud data collected by a laser radar sensor and an RGB image collected by an image acquisition device, and mapping the point cloud data into a first image consistent with the size of the RGB image; using a preset human instance segmentation algorithm to perform human image segmentation on the RGB image to determine candidate pixel points corresponding to a human in the RGB image; for each candidate pixel point, determining a first pixel point corresponding to the candidate pixel point in the first image, and determining a preset number of second pixel points adjacent to the first pixel point in each first preset direction in the first image; determining a first gradient value of the candidate pixel point in each first preset direction according to pixel values corresponding to the first pixel point and the second pixel points in the first image; determining a target pixel point corresponding to the human according to the first gradient value of each candidate pixel point in each first preset direction; wherein the determining of the target pixel point corresponding to the human according to the first gradient value of each candidate pixel point in each first preset direction comprises: determining each target candidate pixel point with at least one first gradient value greater than a corresponding first gradient value threshold; topologically connecting each target candidate pixel point and fourth pixel points other than the candidate pixel points corresponding to the human in the RGB image to determine at least one connected domain; determining pixel points contained in a connected domain with the largest area as the target pixel point corresponding to the human.

2. The method of claim 1, wherein, After the mapping of the point cloud data into the first image consistent with the size of the RGB image, the method further comprises: for each pixel point in the first image, determining the number of point cloud points mapped to the pixel point; if the number of point cloud points is non-zero, determining a first mean value of the vertical coordinate values of the point cloud points corresponding to the pixel point, and determining the first mean value as the pixel value corresponding to the pixel point; if the number of point cloud points is zero, determining each third pixel point with the number of point cloud points adjacent to the pixel point in each second preset direction in the first image being non-zero; determining a second mean value corresponding to the pixel value of each third pixel point, and determining the second mean value as the pixel value corresponding to the pixel point.

3. The method of claim 2, wherein, The second preset direction comprises: an X-axis positive direction, an X-axis negative direction, a Y-axis positive direction, and a Y-axis negative direction.

4. The method of claim 1, wherein, The first preset direction comprises: a first angle bisector direction corresponding to the X-axis positive direction and the Y-axis positive direction, a second angle bisector direction corresponding to the X-axis positive direction and the Y-axis negative direction, a third angle bisector direction corresponding to the X-axis negative direction and the Y-axis positive direction, a fourth angle bisector direction corresponding to the X-axis negative direction and the Y-axis negative direction, the X-axis positive direction, the X-axis negative direction, the Y-axis positive direction, and the Y-axis negative direction.

5. The method of claim 4, wherein, The determining of the first gradient value of the candidate pixel point in each first preset direction according to the pixel values corresponding to the first pixel point and the second pixel points comprises: For each first preset direction, a pixel value difference value of each adjacent two pixel points in the second pixel point of the first preset direction is determined; a third mean value of each pixel value difference value is determined, and the third mean value is determined as a first gradient value of the candidate pixel point in the first preset direction.

6. The method of claim 1, wherein, The method further comprises: determining a fifth pixel point in the target pixel point as a human body edge; For each fifth pixel point, a preset number of sixth pixel points adjacent to the fifth pixel point in each first preset direction in the RGB image are determined; for each first preset direction, a second gradient value of the fifth pixel point in the first preset direction is determined according to pixel values of the fifth pixel point and the preset number of sixth pixel points in the RGB image; According to each second gradient value of each fifth pixel point in each first preset direction, a target other pixel point corresponding to a human body in other pixel points in the RGB image except the target pixel point is determined.

7. The method of claim 6, wherein, The method further comprises: For each fifth pixel point, if the second gradient value of the target first preset direction corresponding to the fifth pixel point is less than a second gradient value threshold corresponding to the target first preset direction, it is determined that other pixel points adjacent to the fifth pixel point in the target first preset direction are target other pixel points corresponding to a human body.

8. The method of claim 7, wherein, The method further comprises: In the RGB image, the target pixel point and the target other pixel point are determined and segmented, and a sub-RGB image segmented is determined as a human body corresponding RGB image.

9. An electronic device, comprising: The electronic device comprises a processor, and the processor is used to implement the steps of the human body image segmentation method in any one of claims 1-8 when executing a computer program stored in a memory.

Images