DATA PROCESSING APPARATUS, DATA PROCESSING METHOD, AND PROGRAM

Abstract

This data processing device (100) comprises an object position specifying unit (103), an object depth distance extraction unit (104), a coordinate conversion unit (105), and a label conversion unit (106). The object position specifying unit (103) specifies, on the basis of an image from a first camera, the position of an object in the image. The object depth distance extraction unit (104) extracts the depth distance from the first camera to the object. The coordinate conversion unit (105) converts the object position in the image to an object position in a world coordinate system using the depth distance. The label conversion unit (106) converts the object position in the world coordinate system to an object label in an image using the position of the first camera in the world coordinate system and imaging information used during generation of an image from sensor measurement results.

Description

[Technical field]

[0001] The present invention relates to a data processing device, a data processing method, and a program. [Background technology]

[0002] Radar-based body scanners have been introduced in airports and other locations to detect dangerous objects. In the radar system of Non-Patent Document 1, an antenna (radar 2) placed on the xy plane (panel 1 in Fig. 21) in Fig. 21(A) irradiates radio waves and measures signals reflected from objects (pedestrians). A radar image is generated based on the measured signals, and dangerous objects (targets in Fig. 21(B)) are detected from the generated radar image.

[0003] Patent Document 1 also describes that the following processing is performed when identifying objects present in a monitoring area. First, distance data to multiple objects present in the monitoring area is obtained from the measurement results of the 3D laser scanner. Next, a change area where the difference between the current distance data and the past distance data is equal to or greater than a threshold is extracted. Next, a front viewpoint image based on the current distance data and the change area is converted to an image obtained by moving the viewpoint of the 3D laser scanner, and an image is created. Then, multiple objects present in the monitoring area are identified based on the front viewpoint image and the image created by the coordinate conversion unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 142779 [Non-patent literature]

[0005] [Non-Patent Document 1] David M. Sheen, Douglas L. McMakin and Thomas E. Hall, "Three-Dimensional Millimeter-Wave Imaging for Concealed Weapon Detection," IEEE Transactions on Microwave Theory and Techniques, vol. 49, No. 9, Sep. 2001. Summary of the Invention [Problem to be solved by the invention]

[0006] The generated radar image is represented by three-dimensional voxels with x, y, and z axes in Fig. 21. Fig. 22 shows the three-dimensional radar image in Fig. 21 projected in the z direction. In object detection using machine learning, it is necessary to label the detected object in the radar image as shown in Fig. 22(A). If the shape of the detected object can be seen in the radar image as shown in Fig. 22(B), labeling is possible. On the other hand, as shown in Fig. 22(B), there are many cases where the shape of the detected object in the radar image is unclear and cannot be seen due to a difference in the posture of the detected object. This is because the clarity of the shape of the detected object depends on the size, posture, reflection intensity, etc. of the detected object. In this case, labeling becomes difficult, which induces incorrect labeling. As a result, a model with poor detection performance may be generated by learning with incorrect labels.

[0007] One of the problems that the present invention aims to solve is to improve the accuracy of labeling in images. [Means for solving the problem]

[0008] According to the present invention, an object position specifying means for specifying a position of an object in an image based on an image of a first camera; an object depth distance extraction means for extracting a depth distance from the first camera to the object; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion means for converting a position of the object in a world coordinate system into a label of the object in the image, using imaging information used to generate an image from a position of the first camera in a world coordinate system and a measurement result of a sensor; There is provided a data processing apparatus comprising:

[0009] According to the present invention, an object position specifying means for specifying a position of an object in an image based on an image of a first camera; an object depth distance extraction means for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system based on the depth distance; a label conversion means for converting an object position in a world coordinate system into a label of the object in the radar image using the position of the first camera in a world coordinate system and imaging information of a sensor; There is provided a data processing apparatus comprising:

[0010] According to the present invention, a marker position specifying means for specifying a position of a marker attached to an object in an image based on an image of a first camera as a position of the object in the image; an object depth distance extraction unit that extracts a depth distance from the first camera to an object using a radar image generated based on a radar signal generated by a sensor; a coordinate conversion unit that converts a position of the object in the image into a position of the object in a world coordinate system using a depth distance from the first camera to the object; a label conversion unit that converts the position of the object in the world coordinate system into a label of the object in the radar image using a camera position in the world coordinate system and imaging information of the sensor; There is provided a data processing apparatus comprising:

[0011] According to the present invention, a computer an object position identification process for identifying a position of an object in an image based on an image of the first camera; an object depth distance extraction process for extracting a depth distance from the first camera to the object; a coordinate transformation process for transforming a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion process for converting the position of the object in the world coordinate system into a label of the object in the image using the position of the first camera in the world coordinate system and imaging information used to generate an image from the measurement results of a sensor; A data processing method is provided for:

[0012] According to the present invention, a computer includes: An object location identification function that identifies a location of an object in an image based on an image of the first camera; an object depth distance extraction function for extracting a depth distance from the first camera to the object; a coordinate transformation function that transforms a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion function that converts the position of the object in the world coordinate system into a label of the object in the image using the position of the first camera in the world coordinate system and imaging information used to generate an image from the measurement results of a sensor; A program to hold them will be provided. Effect of the Invention

[0013] According to the present invention, the accuracy of labeling in images can be improved. [Brief description of the drawings]

[0014] The above objects, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings.

[0015] [Figure 1] FIG. 1 is a block diagram of a first embodiment. [Diagram 2] FIG. 2 is a flow chart of the first embodiment. [Diagram 3] FIG. 11 is a block diagram of a second embodiment. [Figure 4] FIG. 11 is a flow chart of the second embodiment. [Diagram 5] FIG. 13 is a block diagram of a third embodiment. [Figure 6] FIG. 13 is a flowchart of a third embodiment. [Figure 7] FIG. 13 is a block diagram of a fourth embodiment. [Figure 8] FIG. 13 is a flowchart of the fourth embodiment. [Figure 9] FIG. 13 is a block diagram of a fifth embodiment. [Figure 10] FIG. 13 is a flowchart of the fifth embodiment. [Figure 11] FIG. 13 is a block diagram of a sixth embodiment. [Figure 12] FIG. 13 is a flowchart of the sixth embodiment. [Figure 13] FIG. 13 is a block diagram of a seventh embodiment. [Figure 14] FIG. 13 is a flowchart of the seventh embodiment. [Figure 15] FIG. 13 is a block diagram of an eighth embodiment. [Figure 16] FIG. 23 is a flowchart of the eighth embodiment. [Figure 17] FIG. 13 is a block diagram of a ninth embodiment. [Figure 18] FIG. 13 is a flowchart of the ninth embodiment. [Figure 19] FIG. 13 is a block diagram of a tenth embodiment. [Figure 20] FIG. 13 is a flowchart of the tenth embodiment. [Figure 21] This figure shows an overall view of the system ((A) three-dimensional view, (B) top view). [Figure 22] FIG. 1 illustrates the labeling problem in radar images. [Diagram 23]1A and 1B are diagrams illustrating an embodiment ((A) is a three-dimensional view, (B) is a top view). [Figure 24] 1A and 1B are diagrams illustrating examples of labeling according to an embodiment ((A) labels on camera images, (B) labels on radar images). [Diagram 25] FIG. 13 is a diagram showing variations in camera position. [Figure 26] 1A to 1C are diagrams illustrating variations of a method for identifying an object position in a camera image. [Figure 27] FIG. 13 is a diagram showing an object depth distance. [Figure 28] FIG. 1 is a diagram showing a three-dimensional radar image (radar coordinate system). [Figure 29] FIG. 11 illustrates an example of the operation of a label conversion unit; [Diagram 30] FIG. 11 is a diagram illustrating an example of an alignment operation. [Diagram 31] FIG. 13 is a diagram illustrating an example of depth distance extraction. [Diagram 32] FIG. 13 is a diagram showing types of markers. [Diagram 33] FIG. 13 is a diagram showing an example of marker distortion. [Diagram 34] FIG. 2 illustrates an example of a hardware configuration of a data processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals and the description will be omitted as appropriate.

[0017] [First embodiment] [Configuration Description] A first embodiment will be described with reference to Fig. 1. The data processing device 100 includes a synchronization unit 101 that transmits a synchronization signal for synchronizing measurement timing, a first camera measurement unit 102 that instructs the first camera to capture an image, an object position identification unit 103 that identifies the position of an object in an image of the first camera (e.g., a label in an image shown in Fig. 24(A)), an object depth distance extraction unit 104 that extracts a depth distance from the first camera to an object based on the camera image, a coordinate conversion unit 105 that converts the position of the object in the image of the first camera to the position of the object in a world coordinate system based on the depth distance from the first camera to the object, a label conversion unit 106 that converts the position of the object in the world coordinate system to a label of the object in a radar image (e.g., a label in a radar image shown in Fig. 24(B)), a storage unit 107 that holds the position of the first camera and radar imaging information, a radar measurement unit 108 that performs signal measurement at the radar antenna, and an imaging unit 109 that generates a radar image from a radar measurement signal.

[0018] The data processing device 100 is also a part of a radar system. This radar system also includes a camera 20 and a radar 30 shown in Fig. 23. The camera 20 is an example of a first camera described below. Note that, as shown in (B) of Fig. 25, a plurality of cameras 20 may be provided. In this case, at least one of the plurality of cameras 20 is an example of the first camera.

[0019] The synchronization unit 101 outputs a synchronization signal to synchronize the measurement timing of the first camera measurement unit 102 and the radar measurement unit 108. The synchronization signal is output periodically, for example. If the object to be labeled moves over time, strict synchronization between the first camera and the radar is necessary, but if the object to be labeled does not move, synchronization accuracy is not important.

[0020] The first camera measurement unit 102 receives a synchronization signal from the synchronization unit 101 as an input, and outputs an image capture instruction to the first camera upon receiving the synchronization signal. In addition, the first camera measurement unit 102 outputs an image captured by the first camera to an object position identification unit 103 and an object depth distance extraction unit 104. The first camera is a camera that can calculate the distance from the first camera to an object. For example, it is a depth camera (ToF (Time-of-Flight) camera, infrared camera, stereo camera, etc.). In the following description, the image captured by the first camera is a camera with a size of w pixel ×h pixel The depth image is set to the depth of the object to be detected. The first camera is installed at a position where the first camera can capture an image of the detection target. As shown in FIG. 23, the first camera may be installed on the panel 12 on which the antenna of the radar 30 is installed, or may be placed on the walking path as shown in FIG. 25(A). The radar system according to this embodiment also operates when each of the multiple cameras 20 installed at different positions as shown in FIG. 25(B) is used as the first camera. In the example shown in FIG. 25, two panels 12 are installed on either side of the walking path. A camera 20 is installed on each of the two panels 12 facing the walking path, and a camera 20 is also installed in front of and behind the panel 12 in the traveling direction of the walking path. In the following, the camera is assumed to be located at the position shown in FIG. 23.

[0021] The object position identifying unit 103 receives an image from the first camera measurement unit 102 as an input, and outputs the position of the object in the image of the first camera to the object depth distance extraction unit 104 and the coordinate conversion unit 105. The object position may be determined to be the center position of the object as shown in FIG. 26(A), or a region (rectangle) including the object may be selected as shown in FIG. 26(B). The object position in the image identified here is expressed as (x img ,y img When an area is selected, the position of the target object may be set to four points (the four corners of a rectangle) or two points, the start point and the end point.

[0022] The object depth distance extraction unit 104 receives as input the image from the first camera measurement unit 102 and the position of the object in the image from the object position identification unit 103, and outputs the depth distance from the first camera to the object based on the image and the object position in the image to the coordinate conversion unit 105. The depth distance here refers to the distance D from the surface on which the first camera is installed to the surface on which the object is placed, as shown in FIG. 27. The distance D is the position (x img ,y img ) depth.

[0023] The coordinate conversion unit 105 receives as input the object position in the image from the object position identification unit 103 and the depth distance from the object depth distance extraction unit 104, calculates the object position in the world coordinate system based on the object position in the image and the depth distance, and outputs this object position to the label conversion unit 106. target ,Y' target ,Z' target ) has the position of the first camera as the origin, and each dimension corresponds to the x, y, and z axes in Figure 23. The focal length of the first camera in the x direction is f x , the focal length of the first camera in the y direction is f y Then, the object position in the image (x img ,y img ) and the depth distance D, the target position (X' target ,Y' target ,Z' target ) is calculated using equation (1).

[0024]

number

[0025] The label conversion unit 106 receives as input the object position in the world coordinate system from the coordinate conversion unit 105, and also receives the position of the first camera and radar imaging information (described later) from the memory unit 107, converts the object position in the world coordinate system into a label of the object in radar imaging based on the radar imaging information, and outputs the converted label to the learning unit. target,Y' target, Z' target ) is the origin of the first camera position. receive The position of the first camera in the world coordinate system when the radar position is the origin (X camera ,Y camera ,Z camera ) to calculate the target position (X target ,Y target ,Z target ) can be calculated using the following formula (2).

[0026]

number

[0027] The label conversion unit 106 derives the position of the object in radar imaging and sets it as a label based on the position of the object with the radar position as the origin and the radar imaging information received from the storage unit 107. The radar imaging information is the starting point (X init ,Y init ,Z int ) and the x, y, and z direction lengths dX, dY, and dZ per voxel in radar imaging. The position of the object in radar imaging (x target ,y target ,z target ) can be calculated using equation (3).

[0028]

number

[0029] In addition, when the object position identification unit 103 selects one object position (the center of the object) as in FIG. 26(A), the object position here is also one point, so if the size of the object is known, it may be converted into a label with the object position as the center and width and height equal to the size of the object, as in FIG. 29. When there are multiple object positions as in FIG. 26(B), the above calculation may be performed for each, and conversion into a final label may be performed based on the multiple object positions obtained. For example, if the positions (x target {1-4}, y target {1-4}, z target If you have {1-4}, set the start of the label to (min(x target {1-4}),min(y target {1-4}),min(z target {1-4})), and set the end of the label to (max(x target {1-4}),max(y target {1-4}),max(z target {1-4})) can be used.

[0030] The storage unit 107 holds the position of the first camera when the radar position is set as the origin in the world coordinate system, and the radar imaging information. The radar imaging information is the start point (X init ,Y init ,Z int ) and the lengths in the world coordinate system in the x, y, and z directions of one voxel in radar imaging (dX, dY, dZ).

[0031] The radar measurement unit 108 receives a synchronization signal from the synchronization unit 101 as an input, and instructs the antenna of the radar (for example, the above-mentioned radar 30) to perform measurement. It also outputs the measured radar signal to the imaging unit 109. That is, the imaging timing of the first camera and the measurement timing of the radar are synchronized. There are Ntx transmitting antennas, Nrx receiving antennas, and Nk frequencies to be used. A radio wave transmitted by an arbitrary transmitting antenna may be received by multiple receiving antennas. The frequency is switched at a certain frequency width, such as Stepped Frequency Continuous Wave (SWCF). Hereinafter, it is assumed that the radar signal S(it, jr, k) is a signal that is irradiated by the transmitting antenna it at the kth step frequency f(k) and measured by the receiving antenna jr.

[0032] The imaging unit 109 receives a radar signal from the radar measurement unit 108 as an input, generates a radar image, and outputs the generated radar image to the learning unit. c In the vector(v) (tor(v)), vector(v) represents the position of one voxel v in the radar image, and can be calculated from the radar signal S(it,ir,k) using the following equation (4).

[0033]

number

[0034] where c is the speed of light and i is an imaginary number and Let R be the distance from the transmitting antenna it to the receiving antenna ir via the voxel v. R is calculated using the following formula (5). Vector(Tx(it)) and vector(Rx(ir)) are the positions of the transmitting antenna it and the receiving antenna ir, respectively.

[0035]

number

[0036] 34 is a diagram showing an example of a hardware configuration of the data processing device 10. The data processing device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0037] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0038] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0039] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0040] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the data processing device 10. The processor 1020 loads each of these program modules onto the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also function as various types of storage units.

[0041] The input / output interface 1050 is an interface for connecting the data processing device 10 to various input / output devices (for example, cameras and radar).

[0042] The network interface 1060 is an interface for connecting the data processing device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting the network interface 1060 to the network may be a wireless connection or a wired connection.

[0043] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flowchart of FIG. First, synchronization processing (S101) is the operation of the synchronization unit 101 in FIG.

[0044] The camera measurement process (S102) is the operation of the first camera measurement unit 102 in Figure 1, which instructs the first camera to capture an image when a synchronization signal is received, and outputs the captured image to the object position identification unit 103 and the object depth distance extraction unit 104.

[0045] The object position identification process (S103) is the operation of object position identification unit 103 in FIG. 1, which identifies the position of the object based on the image from the first camera and outputs the object position to object depth distance extraction unit 104 and coordinate conversion unit 105.

[0046] The object depth extraction process (S104) is an operation of object depth distance extraction unit 104 in FIG. 1, which extracts the depth distance from the first camera to the object based on the object position in the image and outputs the depth distance to coordinate conversion unit 105.

[0047] The coordinate conversion process (S105) is an operation of the coordinate conversion unit 105 in Figure 1, which converts the object position in the image to the object position in a world coordinate system with the position of the first camera as the origin based on the depth distance, and outputs the object position to the label conversion unit 106. The label conversion process (S106) is an operation of the label conversion unit 106, which converts the position of an object in world coordinates with the position of the first camera as the origin into a label of the object in radar imaging, and outputs the label to the learning unit. In this conversion, the position of the first camera with the radar position as the origin and radar imaging information are used. In this embodiment, the label includes position information, indicating that an object exists at that position.

[0048] The radar measurement process (S107) is an operation of the radar measurement unit 108 in FIG. 1, which, upon receiving a synchronization signal from the synchronization unit 101, instructs the radar antenna to perform measurement and outputs the measured radar signal to the imaging unit 109.

[0049] 1, the imaging unit 109 receives a radar signal from the radar measurement unit 108, generates a radar image from the radar signal, and outputs the radar image to the learning unit. At the time of this output, the label generated in S106 is also output together with the radar image.

[0050] It should be noted that S107 and S108 are executed in parallel with S102 to S106.

[0051] [Effect description] In this embodiment, for objects whose shapes are unclear in a radar image, labeling is possible in the radar image by labeling them in the image of the first camera.

[0052] [Second embodiment] The second embodiment will be described with reference to Fig. 3. The data processing device 200 includes a synchronization unit 201 that transmits a synchronization signal for synchronizing measurement timing, a first camera measurement unit 202 that issues an image capture instruction to the first camera, an object position identification unit 203 that identifies the position of the object in the image of the first camera, an object depth distance extraction unit 204 that extracts the depth distance from the first camera to the object based on the image of the second camera, a coordinate conversion unit 205 that converts the position of the object in the image of the first camera to the position of the object in the world coordinate system based on the depth distance of the object from the first camera, a label conversion unit 206 that converts the position of the object in the world coordinate system to the label of the object in the radar image, a storage unit 207 that holds the position of the first camera and radar imaging information, a radar measurement unit 208 that performs signal measurement at the radar antenna, an imaging unit 209 that generates a radar image from the radar measurement signal, a second camera measurement unit 210 that issues an image capture instruction to the second camera, and an image position alignment unit 211 that aligns the image obtained by the first camera with the camera image obtained by the second camera.

[0053] At least a part of the area captured by the second camera overlaps with the area captured by the first camera. Therefore, the image generated by the first camera and the image generated by the second camera include the same object. In the following description, it is assumed that the first camera and the second camera are located in the same place.

[0054] In addition to the functions of the synchronization unit 101 , the synchronization unit 201 outputs a synchronization signal to the second camera measurement unit 210 .

[0055] The first camera measurement unit 202 receives a synchronization signal as an input, similar to the first camera measurement unit 102. 201and outputs an image capturing instruction to the first camera when the synchronization signal is received. The first camera measurement unit 202 also outputs the image captured by the first camera to the object position identification unit 203 and the image position alignment unit 211. However, the first camera here may be a camera that cannot measure depth. Such a camera is, for example, an RGB camera. However, the second camera is a camera that can measure depth.

[0056] The object position identifying section 203 has the same functions as the object position identifying section 103, and therefore a description thereof will be omitted.

[0057] The object depth distance extraction unit 204 receives as input the position of the object in the image of the first camera from the object position identification unit 203, and also receives the aligned image of the second camera from the image alignment unit 211. Then, the object depth distance extraction unit 204 extracts the depth distance from the second camera to the object in the same manner as the object depth distance extraction unit 104, and outputs the depth distance to the coordinate conversion unit 205. Since the aligned image of the second camera has the same angle of view as the image of the first camera, the depth of the corresponding position in the second depth image becomes the depth distance depending on the position of the object in the image of the first camera.

[0058] Coordinate conversion section 205 has the same function as coordinate conversion section 105, and therefore a description thereof will be omitted.

[0059] The label conversion unit 206 has the same functions as the label conversion unit 106, and therefore a description thereof will be omitted.

[0060] The storage unit 207 has the same functions as the storage unit 107, and therefore the description thereof will be omitted.

[0061] The radar measurement unit 208 has the same functions as the radar measurement unit 108, and therefore a description thereof will be omitted.

[0062] The imaging unit 209 has the same functions as the imaging unit 109, and therefore a description thereof will be omitted.

[0063] The second camera measurement unit 210 receives a synchronization signal from the synchronization unit 201, and outputs an image capture instruction to the second camera upon receiving the synchronization signal. That is, the image capture timing of the second camera is synchronized with the image capture timing of the first camera and the measurement timing of the radar. In addition, the second camera outputs an image captured by the second camera to the image position alignment unit 211. The second camera is a camera that can calculate the distance from the second camera to an object. It corresponds to the first camera in the first embodiment.

[0064] The image alignment unit 211 receives as input an image captured by the first camera from the first camera measurement unit 202 and an image captured by the second camera from the second camera measurement unit 210, aligns the images, and outputs the aligned image of the second camera to the object depth distance extraction unit 204. An example of alignment is shown in FIG. 30. If the size of the first camera image is w1 pixel ×h1 pixel , the size of the image from the second camera is w2 pixel ×h2 pixel In FIG. 30, the angle of view of the image from the second camera is wider. In this case, an image is generated in which the size of the image from the second camera matches the size of the image from the first camera. As a result, any position in the image selected from the image from the first camera in the figure corresponds to the same position in the image from the second camera, and the viewing angle (angle of view) in the images becomes the same. If the angle of view of the image from the second camera is narrower, no alignment is required.

[0065] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. First, synchronization processing (S201) is the operation of the synchronization unit 201 in FIG.

[0066] The camera measurement process (S202) is the operation of the first camera measurement unit 202 in Figure 3, which instructs the first camera to capture an image when a synchronization signal is received, and outputs the image captured by the first camera to the object position identification unit 203 and the image alignment unit 211.

[0067] The object position identification process (S203) is the operation of object position identification unit 203 in FIG. 3, which identifies the object position based on the image from the first camera and outputs the object position to object depth distance extraction unit 204 and coordinate conversion unit 205.

[0068] The object depth extraction process (S204) is an operation of the object depth distance extraction unit 204 in Fig. 3, which extracts the depth distance from the first camera to the object. A specific example of the process performed here is as described with reference to Fig. 3. Then, the object depth distance extraction unit 204 outputs the extracted depth distance to the coordinate conversion unit 205.

[0069] The coordinate conversion process (S205) is an operation of the coordinate conversion unit 205 in Figure 3, which converts the object position in the image to the object position in a world coordinate system with the position of the first camera as the origin based on the depth distance, and outputs the object position to the label conversion unit 206.

[0070] The label conversion process (S206) is an operation of the label conversion unit 206, which converts the position of the object in the world coordinate system with the position of the first camera as the origin into the label of the object in radar imaging based on the position of the first camera with the radar position as the origin and the radar imaging information, and outputs the label to the learning unit. Specific examples of the label are the same as those in the first embodiment.

[0071] The radar measurement process (S207) is an operation of the radar measurement unit 208 in FIG. 3, which, upon receiving a synchronization signal from the synchronization unit 201, instructs the radar antenna to perform measurement and outputs the measured radar signal to the imaging unit 209.

[0072] The imaging process (S208) is an operation of the imaging unit 209 in FIG. 3, which receives a radar signal from the radar measurement unit 108, generates a radar image from the radar signal, and outputs the radar image to the learning unit.

[0073] The camera 2 measurement process (S209) is the operation of the second camera measurement unit 210 in Figure 3, which, upon receiving a synchronization signal from the synchronization unit 201, instructs the second camera to capture an image and outputs the image captured by the second camera to the image alignment unit 211.

[0074] The position alignment process (S210) is an operation of the image position alignment unit 211 in FIG. Total The unit receives an image from the first camera from the measurement unit and an image from the second camera from the second camera measurement unit 210, aligns the image from the second camera so that the angle of view of the image from the second camera is the same as the angle of view of the image from the first camera, and outputs the aligned image from the second camera to the object depth distance extraction unit 204.

[0075] It should be noted that S209 is executed in parallel with S202, and S203 and S210 are executed in parallel. Furthermore, S207 and S208 are executed in parallel with S202 to S206, S209, and S210.

[0076] [Effect description] In this embodiment, even if the position of an object whose shape is unclear in the radar image cannot be identified in the image of the second camera, labeling in the radar image is possible as long as the position of the object can be identified in the image of the first camera.

[0077] [Third embodiment] [Configuration Description] The third embodiment will be described with reference to Fig. 5. The data processing device 300 includes a synchronization unit 301 that transmits a synchronization signal for synchronizing measurement timing, a first camera measurement unit 302 that issues an image capture instruction to the first camera, an object position identification unit 303 that identifies the position of an object in an image of the first camera, an object depth distance extraction unit 304 that extracts a depth distance from the first camera to the object based on a radar image, a coordinate conversion unit 305 that converts the position of the object in the image of the first camera to the position of the object in a world coordinate system based on the depth distance from the first camera to the object, a label conversion unit 306 that converts the position of the object in the world coordinate system to a label of the object in a radar image, a storage unit 307 that holds the position of the first camera and radar imaging information, a radar measurement unit 308 that performs signal measurement at the radar antenna, and an imaging unit 309 that generates a radar image from a radar measurement signal.

[0078] Synchronization unit 301 has the same function as synchronization unit 101, and therefore a description thereof will be omitted.

[0079] The first camera measurement unit 302 receives a synchronization signal from the synchronization unit 301 as an input, instructs the first camera to capture an image at that timing, and outputs the captured image to the object position identification unit 303. The first camera here may be a camera that cannot measure depth, for example, an RGB camera.

[0080] The object position identifying unit 303 receives the image of the first camera from the first camera measurement unit 302 , identifies the object position, and outputs the object position in the image to the coordinate conversion unit 305 .

[0081] The object depth distance extraction unit 304 receives as input a radar image from the imaging unit 309, and also receives from the storage unit 307 the position of the first camera in a world coordinate system with the radar position as the origin and radar imaging information. The object depth distance extraction unit 304 then calculates the depth distance from the first camera to the object, and outputs the depth distance to the coordinate conversion unit 305. At this time, the object depth distance extraction unit 304 uses the radar image to calculate the depth distance from the first camera to the object. For example, the object depth distance extraction unit 304 generates a two-dimensional radar image (FIG. 31) by projecting the three-dimensional radar image V in the z direction and selecting only voxels with the strongest reflection intensity. Next, the object depth distance extraction unit 304 selects an area around the object in this two-dimensional radar image (start point (xs, ys), end point (xe, ye) in the figure), and calculates a z coordinate obtained by averaging the z coordinates of voxels in this area that have a reflection intensity equal to or greater than a certain value. average For example, the object depth distance extraction unit 304 calculates the depth distance using z average and radar imaging information (the size of one voxel in the z-direction dZ and the starting point of the radar image in world coordinates Z init ) and the position of the first camera, the depth distance is output. This depth distance (D) can be calculated, for example, by the following formula (6). Note that in formula (6), it is assumed that the radar position and the first camera position are the same.

[0082]

number

[0083] [Explanation of operation] For example, in Fig. 31, the depth distance is calculated by dividing the z coordinate of the voxel closest to the radar among those with a reflection intensity above a certain value. average It may be calculated similarly using equation (6).

[0084] Coordinate conversion section 305 has the same function as coordinate conversion section 105, and therefore a description thereof will be omitted.

[0085] The label conversion unit 306 has the same function as the label conversion unit 106, and therefore a description thereof will be omitted.

[0086] Since storage unit 307 holds the same information as storage unit 107, a description thereof will be omitted.

[0087] The radar measurement unit 308 has the same function as the radar measurement unit 108, and therefore a description thereof will be omitted.

[0088] In addition to the functions of the imaging unit 109 , the imaging unit 309 outputs the generated radar image to the object depth distance extraction unit 304 .

[0089] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. The synchronization process (S301) is the same as the synchronization process (S101), so a description thereof will be omitted.

[0090] The camera measurement process (S302) is the operation of the first camera measurement unit 302 in Figure 5, which instructs the first camera to capture an image when it receives a synchronization signal from the synchronization unit 301, and outputs the image captured by the first camera to the object position identification unit 303.

[0091] The object position identification process (S303) is the operation of the object position identification unit 303 in Figure 5, which identifies the position of the object based on the image of the first camera received from the first camera measurement unit 302, and outputs the object position to the coordinate conversion unit 305.

[0092] The object depth extraction process (S304) is an operation of the object depth distance extraction unit 304 in Fig. 5, which calculates the depth distance from the first camera to the object using the radar image received from the imaging unit 309, and the position of the first camera in the world coordinate system with the radar position received from the sensor DB 312 as the origin, and radar imaging information, and outputs the depth distance to the coordinate conversion unit 305. Details of this process are as described above with reference to Fig. 5.

[0093] The coordinate conversion process (S305) is the same as the coordinate conversion process (S105), and therefore a description thereof will be omitted.

[0094] The label conversion process (S306) is the same as the label conversion process (S106), so a description thereof will be omitted.

[0095] The radar measurement process (S307) is the same as the radar measurement process (S107), and therefore a description thereof will be omitted.

[0096] The imaging process (S308) is an operation of the imaging unit 309 in FIG. 5, which receives a radar signal from the radar measurement unit 308, generates a radar image from the radar signal, and outputs the radar image to the object depth distance extraction unit 304 and the learning unit.

[0097] [Effect description] In this embodiment, even if the depth distance from the first camera to an object whose shape is unclear in the radar image is unknown, the depth distance of the object from the first camera can be calculated based on the radar image, making it possible to label the object in the radar image as long as the position of the object can be identified in the image of the first camera.

[0098] [Fourth embodiment] [Configuration Description]

[0099] The fourth embodiment will be described with reference to Fig. 7. The data processing device 400 according to this embodiment differs from the first embodiment only in the marker position specifying unit 403 and the object depth distance extracting unit 404, and only these will be described. The first camera here may be a camera that cannot measure depth, for example, an RGB camera.

[0100] The marker position identification unit 403 identifies the position of the marker from the image received from the first camera measurement unit 402 as an input, and outputs the position of the marker to the object depth distance extraction unit 404. Furthermore, the position of the marker is output to the coordinate conversion unit 405 as the position of the object. The marker here is a material that is easily visible to the first camera and easily transmits radar signals. For example, materials such as paper, wood, cloth, and plastic can be used as the marker. Also, a mark made with paint on such a material that is easily transmittable may be used as the marker. The marker is placed on the surface of the object, or in a part close to the surface, in a place that is visible from the first camera. If the object is hidden under a bag or clothing, the marker is placed on the surface of the bag or clothing that hides the object. This makes it possible to visually recognize the marker even if the object cannot be directly seen in the image of the first camera, and to roughly identify the position of the object. The marker may be attached around the center of the object, or multiple markers may be attached to surround the area where the object is located, as shown in FIG. 32. The marker may also be an AR marker. In the example of FIG. 32, the markers are lattice points, but as mentioned above, they may also be AR markers. The position of the marker in the image of the first camera may be identified by visually recognizing the marker position with the human eye, or it may be identified automatically using general image recognition techniques such as pattern matching and tracking. The shape and size of the marker are not important as long as the position of the marker can be calculated from the image of the first camera in the following calculations. In the following, the position of the marker located in the center of the lattice point markers in the image will be referred to as (x marker_c ,y marker_c ), and the positions of the four corners of the marker in the image are (x marker_i ,y marker_i )(i=1,2,3,4).

[0101] The object depth distance extraction unit 404 receives as input the image from the first camera measurement unit 402 and the position of the marker from the marker position identification unit 403, calculates the depth distance of the object from the first camera based on these, and outputs the depth distance to the coordinate conversion unit 405. Regarding the calculation method of the depth distance using a marker, if the first camera can measure the depth without a marker, the depth corresponding to the position of the marker in the image is set as the depth distance as in the first embodiment. If the first camera cannot measure the depth without a marker, as in the case of an RGB image, the depth position of the marker may be calculated from the size of the marker in the image and the positional relationship of the marker (distortion of relative position, etc.) as shown in FIG. 33, and the depth distance from the first camera to the object may be estimated. For example, in the case of an AR marker, the depth distance from the camera to the marker can be calculated even in an RGB image. An example of calculating the position of the marker is shown below. The calculation method differs depending on the type of marker and the installation conditions. A candidate position of the point located at the center of the marker in a world coordinate system with the first camera as the origin is (X' marker_c ,Y' marker_c ,Z' marker_c ), and the possible coordinates of the four corners of the marker based on the roll, pitch, and yaw rotations from the point at the center of the marker are (X' marker_i ,Y' marker_i ,Z' marker_i ) (where i = 1, 2, 3, 4). The candidate positions of the point located at the center of the marker can be arbitrarily selected from the imaging area targeted by the radar image. For example, the center points of each voxel in the entire area can be set as the point located at the center of the marker. of The marker position in the image of the first camera calculated from the coordinates of the four corners of the marker can be set as a candidate position. marker_i ,y' marker_i The marker position can be calculated, for example, from equation (7). In equation (7), f x is the focal length of the first camera in the x direction, and f y is the focal length of the first camera in the y direction.

[0102]

number

[0103] [Explanation of operation] Based on the positions of the four corners of the marker in the image obtained by the marker position specifying unit 403, the error E is calculated by equation (8). The marker position in the world coordinate system is estimated based on the error E. For example, Z' of the marker position in the world coordinate system when E is smallest is marker_c is the depth distance from the first camera to the object. Or, Z' of the four corners of the marker at this time marker_i may be the distance from the first camera to the object.

[0104]

number

[0105] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. Operations other than the marker position identification process (S403) and the object depth extraction process (S404) are the same as those in the first embodiment, and therefore will not be described.

[0106] The marker position identification process (S403) is the operation of the marker position identification unit 403 in Figure 7, which identifies the position of the marker based on the image of the first camera received from the first camera measurement unit 402, outputs the position of the marker to the object depth distance extraction unit 404, and further outputs the position of the marker to the coordinate conversion unit 405 as the position of the object.

[0107] The object depth extraction process (S404) is the operation of the object depth distance extraction unit 404 in Figure 7, which calculates the depth distance from the first camera to the object based on the image received from the first camera measurement unit 402 and the marker position from the marker position identification unit 403, and outputs the depth distance to the coordinate conversion unit 405.

[0108] [Effect description] In this embodiment, by using markers, it is possible to perform more accurate labeling in a radar image for an object whose shape is unclear in the radar image.

[0109] [Fifth embodiment] [Configuration Description]

[0110] The fifth embodiment will be described with reference to Fig. 9. The data processing device 500 according to this embodiment differs from the second embodiment only in the marker position specifying unit 503 and the object depth distance extracting unit 504, and therefore description of the other units will be omitted.

[0111] Marker position identifying section 503 has the same function as marker position identifying section 403, and therefore a description thereof will be omitted.

[0112] The object depth distance extraction unit 504 receives the marker position of the image of the first camera from the marker position identification unit 503, and also receives the aligned image of the second camera from the image alignment unit 511, and by using these, calculates the depth distance from the first camera to the object, and outputs the depth distance to the coordinate conversion unit 505. Specifically, the object depth distance extraction unit 504 uses the aligned second camera image to extract the depth at the position of the marker in the first camera image, and sets the extracted depth as the depth distance from the first camera to the object.

[0113] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. Operations other than the marker position identification process (S503) and the object depth extraction process (S504) are the same as those in the second embodiment, and therefore will not be described.

[0114] The marker position identification process (S503) is the operation of the marker position identification unit 503 in Figure 9, which identifies the position of the marker based on the image of the first camera received from the first camera measurement unit 502, outputs the position of the marker to the object depth distance extraction unit 504, and further outputs the position of the marker to the coordinate conversion unit 505 as the position of the object.

[0115] The object depth extraction process (S504) is the operation of the object depth distance extraction unit 504 in Figure 9, which calculates the depth distance from the first camera to the object using the position of the marker in the first camera image received from the marker position identification unit 503 and the aligned second camera image received from the image alignment unit 511, and outputs the depth distance to the coordinate conversion unit 505.

[0116] [Effect description] In this embodiment, by using markers, it is possible to perform more accurate labeling in a radar image for an object whose shape is unclear in the radar image.

[0117] [Sixth embodiment] [Configuration Description] The sixth embodiment will be described with reference to Fig. 11. A data processing device 600 according to this embodiment differs from the third embodiment only in a marker position specifying unit 603, and therefore descriptions of other parts will be omitted.

[0118] The marker position identifying unit 603 receives an image of the first camera from the first camera measurement unit 602 as an input, identifies the position of the marker in the first camera image, and outputs the identified position of the marker as the position of the object to the coordinate conversion unit 605. Note that the definition of the marker is the same as that described in the marker position identifying unit 403.

[0119] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. Operations other than the marker position identification process (S603) are the same as those in the third embodiment, and therefore will not be described.

[0120] The marker position identification process (603) is the operation of the marker position identification unit 603 in FIG. 11, which identifies the position of the marker based on the image of the first camera received from the first camera measurement unit 602, and outputs the position of the marker to the coordinate conversion unit 605 as the position of the target object.

[0121] [Effect description] In this embodiment, by using markers, it is possible to perform more accurate labeling in a radar image for an object whose shape is unclear in the radar image.

[0122] [Seventh embodiment] [Configuration Description] The seventh embodiment will be described with reference to Fig. 13. A data processing device 700 according to this embodiment is configured by removing the radar measurement unit 108 and the imaging unit 109 from the first embodiment. Each processing unit is the same as in the first embodiment, so the description will be omitted.

[0123] It should be noted that the storage unit 707 holds sensor imaging information instead of radar imaging information.

[0124] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. The radar measurement process (S107) and the imaging process (S108) are omitted from the operation of the first embodiment. Each process is the same as that of the first embodiment, so the description will be omitted.

[0125] [Effect description] This embodiment makes it possible to label an object even if the shape of the object is unclear in an image captured by an external sensor.

[0126] [Eighth embodiment] [Configuration Description] The eighth embodiment will be described with reference to Fig. 15. A data processing device 800 according to this embodiment is configured by removing the radar measurement unit 208 and the imaging unit 209 from the second embodiment. Each processing unit is the same as in the second embodiment, so the description will be omitted.

[0127] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. This is the same as the operation of the second embodiment except for the radar measurement process (S207) and the imaging process (S208). Each process is the same as in the second embodiment, so the description will be omitted.

[0128] [Effect description] This embodiment makes it possible to label an object even if the shape of the object is unclear in an image captured by an external sensor.

[0129] [Ninth embodiment] [Configuration Description] The ninth embodiment will be described with reference to Fig. 17. A data processing device 900 according to this embodiment is configured by removing the radar measurement unit 408 and the imaging unit 409 from the fourth embodiment. Each processing unit is the same as in the fourth embodiment, so the description will be omitted.

[0130] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. The radar measurement process (S407) and the imaging process (S408) are omitted from the operation of the fourth embodiment. Each process is the same as in the fourth embodiment, so the description will be omitted.

[0131] [Effect description] In this embodiment, the use of markers enables more accurate labeling even for objects whose shapes are unclear in images captured by an external sensor.

[0132] [Tenth embodiment] [Configuration Description] A tenth embodiment will be described with reference to Fig. 19. A data processing device 1000 according to this embodiment is configured by removing the radar measurement unit 508 and the imaging unit 509 from the fourth embodiment. Each processing unit is the same as in the fourth embodiment, so a description thereof will be omitted.

[0133] [Explanation of operation] Next, the operation of this embodiment will be described with reference to the flow chart of FIG. The radar measurement process (S507) and the imaging process (S508) are omitted from the operation of the fifth embodiment. Each process is the same as that of the fourth embodiment, so the description will be omitted.

[0134] [Effect description] In this embodiment, the use of markers enables more accurate labeling even for objects whose shapes are unclear in images captured by an external sensor.

[0135] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0136] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. In addition, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0137] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An object position identifying means for identifying a position of an object in an image based on an image of a first camera; an object depth distance extraction means for extracting a depth distance from the first camera to the object; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion means for converting a position of the object in a world coordinate system into a label of the object in the image, using imaging information used to generate an image from a position of the first camera in a world coordinate system and a measurement result of a sensor; A data processing device comprising: 2. In the data processing device according to claim 1, A data processing apparatus, wherein the imaging information includes a start point of a region of interest of an image in a world coordinate system and a length in the world coordinate system of one voxel in the image. 3. In the data processing device according to 1 or 2 above, The object depth distance extraction means is a data processing device that further uses an image generated by a second camera and including the object to extract the depth distance. 4. In the data processing device according to any one of claims 1 to 3, The object position specifying means is a data processing device that specifies the position of the object by specifying the position of a marker attached to the object. 5. In the data processing device according to claim 4, The object depth distance extraction means is a data processing device that calculates a position of the marker using a size of the marker in the image of the first camera, and extracts a depth distance from the first camera to the object based on the position of the marker. 6. In the data processing device according to any one of claims 1 to 5, The sensor performs measurements using radar, The data processing apparatus further comprises imaging means for generating a radar image based on the radar signal generated by said radar. 7. An object position identifying means for identifying a position of an object in an image based on an image of the first camera; an object depth distance extraction means for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system based on the depth distance; a label conversion means for converting an object position in a world coordinate system into a label of the object in the radar image using the position of the first camera in a world coordinate system and imaging information of a sensor; A data processing device comprising: 8. A marker position specifying means for specifying a position of a marker attached to an object in an image based on an image of a first camera as a position of the object in the image; an object depth distance extraction means for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal generated by a sensor; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system using a depth distance from the first camera to the object; a label conversion means for converting the position of the object in the world coordinate system into a label of the object in the radar image using a camera position in the world coordinate system and imaging information of the sensor; A data processing device comprising: 9. In the data processing device according to claim 8, The marker is visible to the first camera and is not visible to the radar image. 10. In the data processing device according to claim 9, A data processing device, wherein the marker is formed using at least one of paper, wood, cloth, and plastic. 11. The computer an object position identification process for identifying a position of an object in an image based on an image of the first camera; an object depth distance extraction process for extracting a depth distance from the first camera to the object; a coordinate transformation process for transforming a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion process for converting the position of the object in the world coordinate system into a label of the object in the image using the position of the first camera in the world coordinate system and imaging information used to generate an image from the measurement results of a sensor; The data processing method used. 12. The data processing method according to claim 11, A data processing method in which the imaging information includes a start point of a region of interest of an image in a world coordinate system and a length in the world coordinate system of one voxel in the image. 13. In the data processing method according to 11 or 12 above, A data processing method in which, in the object depth distance extraction process, the computer further uses an image generated by a second camera and including the object to extract the depth distance. 14. In the data processing method according to any one of claims 11 to 13, A data processing method in which, in the object position specifying process, the computer specifies the position of the object by specifying the position of a marker attached to the object. 15. The data processing method according to claim 14, A data processing method in which, in the object depth distance extraction process, the computer calculates the position of the marker using the size of the marker in the image of the first camera, and extracts the depth distance from the first camera to the object based on the position of the marker. 16. In the data processing method according to any one of claims 11 to 15, The sensor performs measurements using radar, Furthermore, the computer performs imaging processing to generate a radar image based on the radar signal generated by the radar. 17. The computer an object position identification process for identifying a position of an object in an image based on an image of the first camera; an object depth distance extraction process for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal; a coordinate transformation process for transforming a position of the object in the image into a position of the object in a world coordinate system based on the depth distance; a label conversion process for converting an object position in a world coordinate system to a label of the object in the radar image using the position of the first camera in a world coordinate system and imaging information of a sensor; The data processing method used. 18. The computer a marker position identification process for identifying a position of a marker attached to an object in an image based on an image of a first camera as a position of the object in the image; an object depth distance extraction process for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal generated by a sensor; a coordinate transformation process for transforming a position of the object in the image into a position of the object in a world coordinate system using a depth distance from the first camera to the object; a label conversion process for converting the position of the object in the world coordinate system into a label of the object in the radar image using the camera position in the world coordinate system and the imaging information of the sensor; A data processing method comprising: 19. The data processing method according to claim 18, A data processing method, wherein the marker is visible to the first camera and is not visible to the radar image. 20. The data processing method according to claim 19, A data processing method, wherein the marker is formed using at least one of paper, wood, cloth, and plastic. 21. To the computer, An object location identification function that identifies a location of an object in an image based on an image of the first camera; an object depth distance extraction function for extracting a depth distance from the first camera to the object; a coordinate transformation function that transforms a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion function that converts the position of the object in the world coordinate system into a label of the object in the image using the position of the first camera in the world coordinate system and imaging information used to generate an image from the measurement results of a sensor; A program that allows you to have 22. In the program according to 21 above, The imaging information includes a start point of a region of interest of an image in a world coordinate system and a length in the world coordinate system of one voxel in the image. 23. In the program according to 21 or 22 above, The object depth distance extraction function is a program that further uses an image generated by a second camera that includes the object to extract the depth distance. 24. In the program according to any one of the above 21 to 23, The object position identification function is a program for identifying the position of the object by identifying the position of a marker attached to the object. 25. In the program according to 24 above, The object depth distance extraction function is a program that calculates the position of the marker using the size of the marker in the image of the first camera, and extracts the depth distance from the first camera to the object based on the position of the marker. 26. In the program according to any one of the above 21 to 25, The sensor performs measurements using radar, Further, a program for causing the computer to have an imaging processing function for generating a radar image based on the radar signal generated by the radar. 27. To the computer, An object location identification function that identifies a location of an object in an image based on an image of the first camera; an object depth distance extraction function for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal; a coordinate transformation function that transforms a position of the object in the image into a position of the object in a world coordinate system based on the depth distance; a label conversion function that converts an object position in a world coordinate system to a label of the object in the radar image using the position of the first camera in a world coordinate system and imaging information of a sensor; A program that allows you to have 28. To the computer, A marker position identification function that identifies a position of a marker attached to an object in an image based on an image of a first camera as a position of the object in the image; an object depth distance extraction function for extracting a depth distance from the first camera to an object using a radar image generated based on a radar signal generated by a sensor; a coordinate conversion function that converts a position of the object in the image into a position of the object in a world coordinate system using a depth distance from the first camera to the object; a label conversion function that converts the position of the object in the world coordinate system into a label of the object in the radar image using the camera position in the world coordinate system and the imaging information of the sensor; A program that allows you to have

Claims

1. an object position specifying means for specifying a position of an object in an image based on an image of the first camera; an object depth distance extraction means for acquiring information indicating a depth corresponding to an imaging region of the first camera from either an image captured by the first camera, an image captured by a second camera different from the first camera, or a measurement result of a radar, and extracting a depth distance from the first camera to the object based on the acquired information indicating the depth corresponding to the imaging region; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion means for converting a position of the object in a world coordinate system into a label of the object in the radar image, using the position of the first camera in a world coordinate system and radar imaging information used in generating a radar image from a radar measurement result; A data processing device comprising:

2. 2. The data processing device according to claim 1, A data processing apparatus, wherein the radar imaging information includes a start point of an area covered by a radar image in a world coordinate system and a length in the world coordinate system of one voxel in the radar image.

3. 3. The data processing device according to claim 1, The object position specifying means is a data processing device that specifies the position of the object by specifying the position of a marker attached to the object.

4. A data processing device according to any one of claims 1 to 3. The object depth distance extraction means is a data processing device that extracts the depth distance using a captured image generated by the second camera capable of measuring depth, the captured image including the object, as information indicating the depth corresponding to the imaging area.

5. 4. The data processing device according to claim 3, The object depth distance extraction means is a data processing device that calculates a position of the marker using a size of the marker in the image captured by the first camera, and extracts a depth distance from the first camera to the object based on the position of the marker.

6. an object position specifying means for specifying a position of an object in an image based on an image of the first camera; an object depth distance extraction means for extracting a depth distance from the first camera to an object by using a radar image generated based on a radar signal obtained by measurement using a radar; a coordinate conversion means for converting a position of the object in the image into a position of the object in a world coordinate system based on the depth distance; a label conversion means for converting an object position in a world coordinate system into a label of the object in the radar image using a position of the first camera in a world coordinate system and radar imaging information used in generating the radar image from the radar signal; A data processing device comprising:

7. The computer an object position identification process for identifying a position of an object in an image based on an image of the first camera; an object depth distance extraction process for acquiring information indicating a depth corresponding to an imaging region of the first camera from either an image captured by the first camera, an image captured by a second camera different from the first camera, or a measurement result of a radar, and extracting a depth distance from the first camera to the object based on the acquired information indicating the depth corresponding to the imaging region; a coordinate transformation process for transforming a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion process for converting the position of the object in the world coordinate system into a label of the object in the radar image by using the position of the first camera in the world coordinate system and radar imaging information used in generating a radar image from the measurement results of the radar; The data processing method used.

8. On the computer, An object location identification function that identifies a location of an object in an image based on an image of the first camera; an object depth distance extraction function that acquires information indicating a depth corresponding to an imaging region of the first camera from either an image captured by the first camera, an image captured by a second camera different from the first camera, or a measurement result of a radar, and extracts a depth distance from the first camera to the object based on the acquired information indicating a depth corresponding to the imaging region; a coordinate transformation function that transforms a position of the object in the image into a position of the object in a world coordinate system using the depth distance; a label conversion function that converts the position of the object in the world coordinate system into a label of the object in the radar image by using the position of the first camera in the world coordinate system and radar imaging information used in generating a radar image from the measurement results of the radar; A program that allows you to have

Images