Electronic device, program, and information processing method

Abstract

To provide an electronic apparatus, a program, and an information processing method capable of accurately measuring a position of a pedestrian.SOLUTION: An electronic device (10) includes a disparity data processing unit (11) that generates disparity data based on image data acquired from a stereo camera, a skeleton detection unit (12) that detects a skeleton of a person included in the disparity data, a center-of-gravity recognition unit (13) that recognizes a center of gravity of the person based on the skeleton of the person, an ROI setting unit (14) that sets an ROI including the center of gravity of the person for the disparity data, and a conversion unit (15) that converts a position of the center of gravity of the person into a position in a sensor coordinate system based on the disparity data in the ROI.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an electronic device, a program, and an information processing method. [Background technology]

[0002] For example, recognizing pedestrians and measuring the distance to them is important for collision avoidance in automated delivery robots and the like. For example, Patent Document 1 and Patent Document 2 disclose technologies for tracking pedestrians and avoiding collisions with them by referencing their skeletal information. Also, for example, Non-Patent Document 1 discloses an object detection method for Autoware, an autonomous driving system. The method in Non-Patent Document 1 approximates objects detected by a camera into rectangles and assigns distance data measured by LiDAR to the rectangles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-90760 [Patent Document 2] International Publication No. 2017 / 056385 [Non-patent literature]

[0004] [Non-Patent Document 1] Tier IV, "Autonomous Driving System Construction School," [online], [Retrieved June 7, 2024], Internet (https: / / 4c281b16296b2ab02a4e0b2e3f75446d.cdnext.stream.ne.jp / randc / mirai / 1-2_Detection_LP.pdf) Summary of the Invention [Problem to be solved by the invention]

[0005] Here, because pedestrians move their limbs, there is variation in the distance data measured for the pedestrian themselves. Also, in the method of Non-Patent Document 1, if the object detection result approximated by a rectangle includes distance data measured for objects other than the pedestrian, the distance data measured for objects other than the pedestrian becomes noise, making it impossible to obtain an accurate distance to the pedestrian.

[0006] In view of the above circumstances, an object of the present disclosure is to provide an electronic device, a program, and an information processing method that can accurately measure the position of a pedestrian. [Means for solving the problem]

[0007] (1) An electronic device according to an embodiment of the present disclosure includes: a parallax data processing unit that generates parallax data based on image data acquired from the stereo camera; a skeleton detection unit that detects a skeleton of a person included in the parallax data; a center of gravity recognition unit that recognizes a center of gravity of the person based on the skeleton of the person; an ROI setting unit that sets an ROI including a center of gravity of the person for the disparity data; and a conversion unit that converts the position of the center of gravity of the person into a position in a sensor coordinate system based on the parallax data in the ROI.

[0008] (2) As one embodiment of the present disclosure, in (1), The ROI setting unit sets the ROI by multiplying each of the trunk length and trunk width calculated from the human skeleton by a predetermined coefficient.

[0009] (3) As an embodiment of the present disclosure, in (1) or (2), The ROI setting unit adjusts the edge of the ROI based on the wrist position of the person calculated from the skeleton of the person.

[0010] (4) As an embodiment of the present disclosure, in any one of (1) to (3), The conversion unit converts the position of the center of gravity of the person into a position in the sensor coordinate system based on the median value of matching points in the disparity data in the ROI.

[0011] (5) As an embodiment of the present disclosure, in any one of (1) to (4), The ROI setting unit causes a display unit to display an image including the person and the ROI superimposed on the person's torso.

[0012] (6) A program according to an embodiment of the present disclosure includes: Computer, a parallax data processing unit that generates parallax data based on image data acquired from the stereo camera; a skeleton detection unit that detects a skeleton of a person included in the parallax data; a center of gravity recognition unit that recognizes a center of gravity of the person based on the skeleton of the person; an ROI setting unit that sets an ROI including a center of gravity of the person for the disparity data; The device functions as a conversion unit that converts the position of the center of gravity of the person into a position in the sensor coordinate system based on the parallax data in the ROI.

[0013] (7) An information processing method according to an embodiment of the present disclosure includes: An information processing method executed by an electronic device, a disparity data processing step of generating disparity data based on image data acquired from the stereo camera; a skeleton detection step of detecting a skeleton of a person included in the disparity data; a center of gravity recognition step of recognizing a center of gravity of the person based on the skeleton of the person; an ROI setting step of setting an ROI including a center of gravity of the person for the disparity data; and a transformation step of transforming the position of the center of gravity of the person into a position in a sensor coordinate system based on the parallax data in the ROI. [Effects of the Invention]

[0014] According to the present disclosure, it is possible to provide an electronic device, a program, and an information processing method that can measure the position of a pedestrian with high accuracy. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an electronic device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram for explaining the problems of the conventional technology. [Figure 3] FIG. 3 is a flowchart showing the processing of an information processing method according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a flowchart showing the ROI setting process of FIG. [Figure 5] FIG. 5 is a flowchart showing the edge adjustment process of FIG. [Figure 6] FIG. 6 is a flowchart showing the disparity value reference process of FIG. [Figure 7] FIG. 7 is a diagram for explaining the conversion step. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an electronic device 10 (see FIG. 1), a program, and an information processing method according to an embodiment of the present disclosure will be described with reference to the drawings.

[0017] 1 is a diagram showing an example of the configuration of an electronic device 10 according to this embodiment. The electronic device 10 is used, for example, in an automatic delivery robot, and measures the position of a person by executing an information processing method (see FIG. 3) described below in order to avoid collisions with people. In this embodiment, the person is described as a pedestrian.

[0018] 1, electronic device 10 includes a disparity data processing unit 11, a skeleton detection unit 12, a center of gravity recognition unit 13, an ROI setting unit 14, a conversion unit 15, and a storage unit 16. The hardware configuration of electronic device 10 may be, for example, a computer. The computer may be, for example, a small computer mounted on an automatic delivery robot or the like, or may be, for example, a server computer. The components of electronic device 10 will be described in detail below.

[0019] Here, electronic device 10 may not be a single device, but may be composed of multiple devices located in multiple locations and capable of transmitting and receiving data to and from each other via a network. In other words, multiple devices connected via a network may function as electronic device 10 as a whole as shown in Fig. 1. Therefore, for example, electronic device 10 may be composed of a single computer as a hardware configuration, or may be composed of multiple computers connected via a network. When composed of multiple computers, a shared memory accessible by each computer or a storage device on the network may be used.

[0020] Generally, a computer includes, for example, a memory, a hard disk drive (storage device), a CPU (processing device), a display, etc. When the electronic device 10 is realized by a computer, a program may cause the processing device to function as a parallax data processing unit 11, a skeleton detection unit 12, a center of gravity recognition unit 13, an ROI setting unit 14, and a conversion unit 15. The storage device may function as a memory unit 16. Here, the memory unit 16 may be any other storage medium such as a RAM, a ROM, an HDD, or an SSD. The memory unit 16 may store various programs and data, such as an OS and application software, used by the electronic device 10.

[0021] The electronic device 10 according to this embodiment acquires image data of the surroundings captured by a stereo camera mounted on, for example, an automatic delivery robot, recognizes pedestrians, and calculates the pedestrian's position to determine the distance to the pedestrian. Furthermore, the electronic device 10 according to this embodiment can display a recognition result image on a display unit (display) used by, for example, an administrator who manages the automatic delivery robot. Recently, technologies for avoiding collisions with pedestrians have been proposed, but the problem with conventional technologies is that the accuracy of the calculated pedestrian position is reduced due to the presence of distance data measured for objects other than pedestrians.

[0022] FIG. 2 is a diagram illustrating the problems with the conventional technology. In the conventional technology, a rectangular area is set to include a recognized pedestrian, and the pedestrian's position is identified based on multiple measurement results (measurements for calculating distance) included in the rectangular area. Specifically, the pedestrian's position is identified using the average value of multiple measurement results. Here, as shown in the left diagram of FIG. 2, the rectangular area including the pedestrian may also include objects other than the pedestrian (a vehicle in the example of FIG. 2). In the conventional technology, the measurement results for the pedestrian and the measurement results for objects other than the pedestrian are handled together, reducing the accuracy of the calculated pedestrian position. In other words, distance data measured for objects other than the pedestrian becomes noise, reducing the accuracy of the measurement of the pedestrian's position. While it is possible to identify and remove objects other than the pedestrian from the measurement results in the rectangular area, this increases processing time, potentially making it impossible to avoid collisions with the pedestrian. Furthermore, the rectangular area in the conventional technology also includes the pedestrian's hands and feet, and the movement of the hands and feet reduces the accuracy of the calculated pedestrian position.

[0023] In contrast, the electronic device 10 according to this embodiment can improve the accuracy of the pedestrian's position by setting an ROI on the torso of a recognized pedestrian and using the measurement results included in the ROI to calculate the pedestrian's position (see the right diagram in FIG. 2). Here, the ROI (Region of Interest) is a limited area that is a part of the entire area including the measurement results and is used to calculate the pedestrian's position. The ROI may be a part of the torso, such as the head, arms, or legs. The ROI may be any part of a person, such as a car, bicycle, stroller, or motorcycle.

[0024] Referring again to FIG. 1, the components of the electronic device 10 will be described. The disparity data processing unit 11 generates disparity data based on image data acquired from the stereo camera. Here, disparity data is data indicating the difference in the positions of corresponding parts (matching points) between an image for the right eye (right camera) and an image for the left eye (left camera). The value of the disparity data for a matching point is indicated by a positive value. In other words, if there is no correspondence between the two images for a certain part (if a matching point is not identified), the value of the disparity data will be zero or less.

[0025] The skeleton detection unit 12 detects the skeleton of a person (a pedestrian in this embodiment) included in the parallax data. Here, a known method may be used to detect the skeleton. For example, as in this embodiment, the skeleton detection unit 12 may detect the coordinates of 17 points (head, shoulders, hips, both hands, wrists, feet, etc.) that are characteristic of a person's skeleton. Based on the skeleton coordinates detected by the skeleton detection unit 12, for example, the length (height) of the torso, the width of the torso, and the position of the wrists are calculated. For example, the torso may be defined such that the skeletal coordinates of both shoulders and hips are the four corners.

[0026] The center of gravity recognition unit 13 recognizes the center of gravity of a person based on the skeleton of the person detected by the skeleton detection unit 12. In this embodiment, the center of gravity recognition unit 13 identifies the torso from the skeletal coordinates of both shoulders and both hips, and recognizes the center of gravity from the length and width of the torso.

[0027] The ROI setting unit 14 sets an ROI including the center of gravity recognized by the center of gravity recognition unit 13 for the disparity data. As shown in the right diagram of FIG. 2, the ROI is set so as to be included in the torso of the person. In this embodiment, the ROI setting unit 14 sets the ROI by multiplying each of the torso length and torso width calculated from the skeleton of the person by a predetermined coefficient. The ROI setting unit 14 may also adjust the edges of the ROI based on the wrist position of the person calculated from the skeleton of the person. The ROI setting unit 14 may also cause the display unit to display a recognition result image including the person and the ROI superimposed on the torso of the person.

[0028] The conversion unit 15 converts the position of the center of gravity of the person into a position in the sensor coordinate system based on the disparity data in the ROI. That is, the conversion unit 15 converts the coordinates of the recognized center of gravity in the image coordinate system (two-dimensional UV space) into coordinates in the sensor coordinate system (three-dimensional XYZ space) in the real world. Here, a well-known method in stereo camera technology may be used for the coordinate conversion. In this embodiment, the conversion unit 15 converts the position of the center of gravity of the person into a position in the sensor coordinate system based on the median of the matching points in the disparity data in the ROI. In the example on the right side of FIG. 2, the conversion unit 15 calculates the median of the disparity data of the "measurement results used for calculation" included in the ROI and performs coordinate conversion based on the median. Here, to eliminate the influence of noise, the conversion unit 15 may convert using a representative value of the disparity data values ​​of the matching points in the ROI. Therefore, instead of the median, it is possible to use, for example, the mode.

[0029] 3 is a flowchart showing the process of the information processing method executed by the electronic device 10 according to this embodiment. The electronic device 10 can accurately measure the position of a pedestrian by executing the information processing method described below.

[0030] The parallax data processing unit 11 acquires image data from the stereo camera (step S1).

[0031] The parallax data processing unit 11 generates parallax data based on the acquired image data (step S2, parallax data processing step).

[0032] The skeleton detection unit 12 detects the skeleton of a person included in the parallax data (step S3, skeleton detection step).

[0033] The center of gravity recognition unit 13 recognizes the center of gravity of the person based on the skeleton of the person (step S4, center of gravity recognition step).

[0034] The ROI setting unit 14 executes ROI setting processing for setting an ROI including the center of gravity of the person for the disparity data (step S5, ROI setting step).Here, Fig. 4 is a flowchart showing the details of the ROI setting processing of Fig. 3.

[0035] If both the body length and the body width are equal to or less than zero (YES in step S101), the ROI setting unit 14 sets the ROI width to 0 [px] and the ROI length (height) to 0 [px] (step S102).

[0036] If both the body length and the body width are not equal to or less than zero, that is, if at least one of the body length and the body width is a positive value (NO in step S101), the ROI setting unit 14 proceeds to the processing of step S103. If the body length is equal to or less than zero (YES in step S103), the ROI setting unit 14 sets the ROI width to a value obtained by multiplying the body length by a constant A, and sets the ROI length to a value obtained by multiplying the body length by a constant B (step S104). Here, the constants A and B are examples of predetermined coefficients, and are determined from past performance data or experimental data, etc. One example of the constant A is 0.33. One example of the constant B is 1.0. Both the constants A and B may be values ​​greater than 0 and less than or equal to 1.

[0037] If the body length is not equal to or less than zero, that is, if the body length is a positive value (NO in step S103), the ROI setting unit 14 proceeds to the processing of step S105. If the body width is equal to or less than zero (YES in step S105), the ROI setting unit 14 sets the ROI width to the value obtained by multiplying the body length by a constant A, and sets the ROI length to the value obtained by multiplying the body length by a constant B (step S106).

[0038] If the trunk width is not equal to or less than zero, that is, if the trunk width is a positive value (NO in step S105), the ROI setting unit 14 proceeds to the processing of step S107. The ROI setting unit 14 sets the average trunk width as the fixed value for left-right orientation and the average trunk length as the fixed value for left-right orientation (step S107). Here, the fixed value for left-right orientation of the trunk width and the fixed value for left-right orientation of the trunk length may be determined based on the size of an average human body. An example of the fixed value for left-right orientation of the trunk width is 193 [mm]. Also, an example of the fixed value for left-right orientation of the trunk length is 485 [mm].

[0039] If (body width / body length) is smaller than (average body width / average body length) (YES in step S108), the ROI setting unit 14 sets the ROI width to the value obtained by multiplying the body length by a constant A, and sets the ROI length to the value obtained by multiplying the body length by a constant B (step S109).

[0040] If (torso width / torso length) is equal to or greater than (average torso width / average torso length) (NO in step S108), the ROI setting unit 14 proceeds to the processing of step S110. The ROI setting unit 14 sets the average torso width as the fixed value for the front-to-back orientation and the average torso length as the fixed value for the front-to-back orientation (step S110). Here, the fixed value for the torso width as the fixed value for the front-to-back orientation and the fixed value for the torso length as the fixed value for the front-to-back orientation may be determined based on the size of an average human body. An example of the fixed value for the torso width as the fixed value for the front-to-back orientation is 469 [mm]. Also, an example of the fixed value for the torso length as the fixed value for the front-to-back orientation is 485 [mm].

[0041] If (body width / body length) is greater than (average body width / average body length) (YES in step S111), the ROI setting unit 14 sets the ROI width to the value obtained by multiplying the body width by a constant B, and sets the ROI length to the value obtained by multiplying the body width by a constant A (step S112).

[0042] If (body width / body length) is equal to or less than (average body width / average body length) (NO in step S111), the ROI setting unit 14 sets the ROI width to the value obtained by multiplying the body width by a constant B, and sets the ROI length to the value obtained by multiplying the body length by a constant B (step S113).

[0043] The ROI setting unit 14 sets an ROI having the width and length (height) determined in step S102, S104, S106, S109, S112 or S113, with the center of gravity at the center (step S114).

[0044] 3 again, after the ROI setting process, the ROI setting unit 14 executes edge adjustment process to adjust the edge of the ROI based on the wrist position of the person calculated from the skeleton of the person (step S6). Here, FIG. 5 is a flowchart showing the details of the edge adjustment process of FIG. 3.

[0045] If the (torso width / torso length) is smaller than the (average torso width / average torso length) (YES in step S201), the ROI setting unit 14 proceeds to the processing of step S202. If the skeletal coordinates of the wrist exist within the ROI (YES in step S202), the ROI setting unit 14 proceeds to the processing of step S203. If the (torso width / torso length) is equal to or larger than the (average torso width / average torso length) (NO in step S201), or if the skeletal coordinates of the wrist do not exist within the ROI (NO in step S202), the ROI setting unit 14 ends the end adjustment processing.

[0046] If the skeletal coordinate of the wrist is above the center of the ROI (YES in step S203), the ROI setting unit 14 updates the upper end of the ROI to be lower than the skeletal coordinate of the wrist by a constant [px] (step S204). Here, the constant [px] is, for example, 2.

[0047] If the wrist skeletal coordinate is below the ROI center or is the same in the height direction (NO in step S203), the ROI setting unit 14 updates the bottom edge of the ROI to be above the wrist skeletal coordinate by a constant [px] (step S205).

[0048] If the skeletal coordinates of the wrist are to the left of the ROI center (YES in step S206), the ROI setting unit 14 updates the left end of the ROI to the right of the skeletal coordinates of the wrist by a constant [px] (step S207).

[0049] If the wrist skeletal coordinates are the same to the right of the ROI center in the width direction (NO in step S206), the ROI setting unit 14 updates the right end of the ROI to the left of the ROI by a constant [px] from the wrist skeletal coordinates (step S208).

[0050] 3 again, after the edge adjustment process, the conversion unit 15 executes a disparity value reference process to calculate a median value of matching points among the disparity data in the ROI (step S7). Here, FIG. 6 is a flowchart showing the details of the disparity value reference process in FIG.

[0051] The conversion unit 15 acquires the disparity values ​​in the ROI (step S301). The conversion unit 15 extracts matching points and calculates the median of the disparity values ​​at the matching points (step S302).

[0052] 3 again, after the parallax value reference process, the conversion unit 15 converts the position of the center of gravity of the person into a position in the sensor coordinate system based on the parallax data in the ROI (step S8, conversion step). As shown in Fig. 7, the coordinate conversion executed by the conversion unit 15 is to convert the coordinates of the center of gravity of the person in the image coordinate system (two-dimensional UV space) into coordinates in the sensor coordinate system (three-dimensional XYZ space) in the real world. The Z direction corresponds to the depth direction (depth direction).

[0053] 7, (cx, cy) are the coordinates of the camera principal point in the image coordinate system. Furthermore, assuming that the parallax value is d, the coordinates on the stereo camera image are (u, v), the focal length is f, and the base line length is Base, the conversion unit 15 calculates the converted coordinates (x, y, z) as follows:

[0054] The conversion unit 15 calculates z by (f×Base) / d. The conversion unit 15 also calculates x by {(u-cx)×Base} / d. The conversion unit 15 also calculates y by {(u-cy)×Base} / d.

[0055] Referring again to FIG. 3, the ROI setting unit 14 outputs a recognition result image including the person and the ROI superimposed on the person's torso, and displays it on the display unit (step S9). The recognition result image visually shows the ROI, for example, as shown in the right diagram of FIG. 2. As described above, the position of the pedestrian in the real world is calculated based on the parallax value included in the ROI. Because the ROI is set to include the pedestrian's torso, the measurement results do not include anything other than the pedestrian, and are not affected by the movement of the pedestrian's limbs.

[0056] As described above, the electronic device 10, the program, and the information processing method according to the present embodiment can accurately measure the position of a pedestrian by using the above configuration to prevent distance data measured for objects other than a pedestrian from becoming noise.

[0057] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined or divided into one. The embodiments of the present disclosure can also be realized as a storage medium on which a program executed by a processor included in an apparatus is recorded. It should be understood that these are also included within the scope of the present disclosure. [Explanation of symbols]

[0058] 10 Electronic equipment 11 Parallax data processing section 12 Skeleton detection unit 13 Center of gravity recognition unit 14 ROI setting section 15 Conversion unit 16 Memory section

Claims

1. a parallax data processing unit that generates parallax data based on image data acquired from the stereo camera; a skeleton detection unit that detects a skeleton of a person included in the parallax data; a center of gravity recognition unit that recognizes a center of gravity of the person based on the skeleton of the person; an ROI setting unit that sets an ROI including a center of gravity of the person for the disparity data; a conversion unit that converts the position of the center of gravity of the person into a position in a sensor coordinate system based on the parallax data in the ROI.

2. The electronic device according to claim 1 , wherein the ROI setting unit sets the ROI by multiplying a trunk length and a trunk width calculated from the human skeleton by predetermined coefficients.

3. The electronic device according to claim 1 , wherein the ROI setting unit adjusts an edge of the ROI based on a wrist position of the person calculated from a skeleton of the person.

4. The electronic device according to claim 1 , wherein the conversion unit converts the position of the center of gravity of the person into a position in the sensor coordinate system based on a median value of matching points in the disparity data in the ROI.

5. The electronic device according to claim 1 , wherein the ROI setting unit causes a display unit to display a recognition result image including the person and the ROI superimposed on a torso of the person.

6. Computer, a parallax data processing unit that generates parallax data based on image data acquired from the stereo camera; a skeleton detection unit that detects a skeleton of a person included in the parallax data; a center of gravity recognition unit that recognizes a center of gravity of the person based on the skeleton of the person; an ROI setting unit that sets an ROI including a center of gravity of the person for the disparity data; a program that functions as a conversion unit that converts the position of the center of gravity of the person into a position in a sensor coordinate system based on the disparity data in the ROI.

7. An information processing method executed by an electronic device, a disparity data processing step of generating disparity data based on image data acquired from the stereo camera; a skeleton detection step of detecting a skeleton of a person included in the disparity data; a center of gravity recognition step of recognizing a center of gravity of the person based on the skeleton of the person; an ROI setting step of setting an ROI including a center of gravity of the person for the disparity data; and a transformation step of transforming the position of the center of gravity of the person into a position in a sensor coordinate system based on the disparity data in the ROI.

Images