Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device is provided with: a circuit that receives a bit stream from an autonomous traveling device having an imaging unit, decodes an image captured by the imaging unit on the basis of the bit stream, and transmits the decoded image to the autonomous traveling device; in-image detection information, which is detection information in the image relating to a target included in the image, is decoded on the basis of the bitstream, and actual space detection information, which is detection information in an actual space relating to the target, is decoded on the basis of the bitstream.

Description

Technical Field

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method. Background Art

[0002] In Patent Document 1, an autonomous driving vehicle including an object detection unit and an autonomous module is disclosed. The object detection unit measures a sensing distance using a camera or the like. The autonomous module determines a control range within the sensing distance and reflects the driving tendency of the learned user and the driving tendency defined by external data in the driving control-related data of the vehicle.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: U.S. Patent Application Publication No. 2021 / 0278840 Summary of the Invention

[0006] An object of the present disclosure is to effectively utilize detection information related to a target existing around an autonomous driving device, thereby improving the safety of the autonomous driving device such as collision avoidance performance.

[0007] A decoding device according to an aspect of the present disclosure includes: a circuit; and a memory connected to the circuit, the circuit performing the following processing: receiving a bitstream from an autonomous driving device having a photographing unit, decoding an image photographed by the photographing unit according to the bitstream, decoding detection information in the image related to a target included in the image, that is, in-image detection information, according to the bitstream, and decoding detection information in the actual space related to the target, that is, actual space detection information, according to the bitstream. Brief Description of the Drawings

[0008] Figure 1 FIG. schematically shows the configuration of an information processing system according to the first embodiment of the present disclosure.

[0009] Figure 2 FIG. is a flowchart showing the process executed by the circuit of the encoding device.

[0010] Figure 3 FIG. schematically shows an example of the driving state of the autonomous driving device.

[0011] Figure 4 FIG. schematically shows an example of an image photographed by the photographing unit.

[0012] Figure 5 FIG. schematically shows an example of the analysis result of the image.

[0013] Figure 6 It is a diagram that schematically shows an example of actual space detection information and in-image detection information related to a target.

[0014] Figure 7 It is a diagram that shows an example of the coordinate system of an autonomous driving device.

[0015] Figure 8 It is a diagram that schematically shows the data structure of a bitstream.

[0016] Figure 9 It is a diagram that schematically shows an example of syntax.

[0017] Figure 10 It is a flowchart that shows the flow of processing executed by the circuit of a decoding device.

[0018] Figure 11 It is a diagram that schematically shows an example of a display image.

[0019] Figure 12 It is a diagram that schematically shows an example of a display image.

[0020] Figure 13 It is a diagram that schematically shows an example of a display image.

[0021] Figure 14 It is a diagram showing Figure 13 a part extracted from the display image shown.

[0022] Figure 15 It is a diagram that schematically shows an example of a display image.

[0023] Figure 16 It is a diagram that schematically shows the structure of an information processing system related to a modification example.

[0024] Figure 17 It is a flowchart that shows the flow of processing executed by the circuit of an encoding device.

[0025] Figure 18 It is a diagram that schematically shows an example of syntax.

[0026] Figure 19 It is a flowchart that shows the flow of processing executed by the circuit of a decoding device.

[0027] Figure 20 It is a diagram that schematically shows an example of a display image.

[0028] Figure 21 It is a diagram that schematically shows an example of a display image.

[0029] Figure 22 This is a diagram schematically showing the structure of the information processing system according to the second embodiment of the present disclosure.

[0030] Figure 23 This is a flowchart showing the process executed by the circuit of the decoding device.

[0031] Figure 24 This is a diagram schematically showing an example of an image captured by the imaging unit.

[0032] Figure 25 This is a diagram schematically showing an example of a map. Detailed Embodiments

[0033] (Understanding underlying the present disclosure)

[0034] In a product delivery system using an autonomous driving device such as an autonomous vehicle, the unmanned autonomous driving device travels on the sidewalk at a speed similar to that of pedestrians to deliver products. In the autonomous driving device, a camera, a GPS receiver, an obstacle sensor, etc. are mounted, and they complement each other's functions to achieve safe driving. However, there are technical limitations in reliably achieving safe driving even in a very complex environment, and in the event of a problem, the intervention of a human operator based on remote operation is required. Therefore, the autonomous driving device transmits the image captured by the camera mounted on the device in real time to the monitoring center where the operator is stationed.

[0035] However, in a situation where one operator monitors multiple autonomous driving devices at the same time, etc., even if the operator visually recognizes the image received from the autonomous driving device, it is difficult to instantaneously and accurately grasp the situation around the autonomous driving device.

[0036] To solve this problem, the present inventor obtained the following understanding and thus conceived the present disclosure: By not only transmitting the image captured by the camera, but also transmitting the detection information in the actual space detected by the sensors mounted in the autonomous driving device from the autonomous driving device to the monitoring center and effectively utilizing it, the above problem can be solved.

[0037] Next, each embodiment of the present disclosure will be described.

[0038] The decoding device according to the first aspect of the present disclosure includes: a circuit; and a memory connected to the circuit, and the circuit performs the following processing: receiving a bitstream from an autonomous driving device having a photographing unit, decoding an image photographed by the photographing unit according to the bitstream, decoding in-image detection information, which is detection information within the image related to a target included in the image, according to the bitstream, and decoding detection information in the actual space, which is detection information in the actual space related to the target, according to the bitstream.

[0039] According to the first aspect, it is possible to effectively utilize in-image detection information and detection information in the actual space related to a target existing around the autonomous driving device on the decoding device side, thereby improving the safety of the autonomous driving device such as collision avoidance performance. In addition, it is possible to improve the decision-making of the operation of the autonomous driving device for improving the safety of the autonomous driving device in a complex environment.

[0040] In the decoding device according to the second aspect of the present disclosure, in the first aspect, the in-image detection information related to the target may include position information of the target within the image.

[0041] According to the second aspect, it is possible to effectively utilize the position information of the target within the image on the decoding device side.

[0042] In the decoding device according to the third aspect of the present disclosure, in the first aspect or the second aspect, the detection information in the actual space related to the target may include at least one of position information of the target in the actual space and movement information of the target in the actual space.

[0043] According to the third aspect, it is possible to effectively utilize the position information and movement information of the target in the actual space on the decoding device side.

[0044] In the decoding device according to the fourth aspect of the present disclosure, in the third aspect, the position information of the target may include coordinate values in a circular coordinate system based on the autonomous driving device, coordinate values in a Cartesian coordinate system based on the autonomous driving device, or coordinate values in a geographic coordinate system.

[0045] According to the fourth aspect, it is possible to accurately determine the position of the target by using position information including coordinate values.

[0046] In the decoding device according to the fifth aspect of the present disclosure, in the third aspect or the fourth aspect, the movement information of the target may include at least one of the absolute speed of the target or the relative speed of the target with respect to the autonomous driving device, and the absolute movement direction of the target or the relative movement direction of the target with respect to the autonomous driving device.

[0047] According to the fifth mode, at least one of the speed and the moving direction of the target can be accurately determined by moving information including at least one of the absolute speed or relative speed and the absolute moving direction or relative moving direction.

[0048] In any one of the first to fifth modes of the decoding apparatus according to the sixth mode of the present disclosure, the circuit may further decode actual space detection information related to the autonomous driving device according to the bitstream, and the actual space detection information related to the autonomous driving device includes position information of the autonomous driving device in the actual space and moving information of the autonomous driving device in the actual space.

[0049] According to the sixth mode, the position information and the moving information of the autonomous driving device in the actual space can be effectively utilized on the decoding apparatus side.

[0050] In the sixth mode of the decoding apparatus according to the seventh mode of the present disclosure, the position information of the autonomous driving device may include coordinate values in a geographic coordinate system.

[0051] According to the seventh mode, the absolute position of the autonomous driving device can be accurately determined by the position information including coordinate values in a geographic coordinate system.

[0052] In the sixth mode or the seventh mode of the decoding apparatus according to the eighth mode of the present disclosure, the moving information of the autonomous driving device may include at least one of the absolute speed and the absolute moving direction of the autonomous driving device.

[0053] According to the eighth mode, at least one of the absolute speed and the absolute moving direction of the autonomous driving device can be accurately determined by the moving information.

[0054] In any one of the sixth to eighth modes of the decoding apparatus according to the ninth mode of the present disclosure, the circuit may decode the in-image detection information and the actual space detection information related to the target and the actual space detection information related to the autonomous driving device according to the SEI area of the bitstream.

[0055] According to the ninth mode, by encoding the in-image detection information and the actual space detection information related to the target and the actual space detection information related to the autonomous driving device into the SEI area of the bitstream, processing can be easily performed on the decoding apparatus side.

[0056] In the decoding device according to the tenth aspect of the present disclosure, in any one of the first to ninth aspects, the circuit may further generate a display image for human vision, the display image including: the target; and at least one of the display position information of the target in the actual space and the display movement information of the target in the actual space.

[0057] According to the tenth aspect, by visually recognizing the display image including the display position information and the display movement information of the target in the actual space, the operator can instantaneously and accurately grasp the situation around the autonomous driving device.

[0058] In the decoding device according to the eleventh aspect of the present disclosure, in the tenth aspect, the autonomous driving device may include a plurality of autonomous driving devices, and the circuit generates the display image based on a plurality of images decoded from a plurality of bitstreams received from the plurality of autonomous driving devices.

[0059] According to the eleventh aspect, since the dead angles during shooting can be complemented by the plurality of images captured by the plurality of autonomous driving devices, the convenience of the display image can be improved.

[0060] In the decoding device according to the twelfth aspect of the present disclosure, in the tenth aspect or the eleventh aspect, the circuit may use the position information of the target in the actual space included in the actual space detection information related to the target decoded from the bitstream as the display position information of the target, and use the movement information of the target in the actual space included in the actual space detection information related to the target decoded from the bitstream as the display movement information of the target.

[0061] According to the twelfth aspect, the display position information and the display movement information can be easily generated.

[0062] In the decoding device according to the 13th aspect of the present disclosure, in the 10th or 11th aspect, the circuit may further perform the following processing: According to the bitstream, decode the position information of the autonomous driving device in the actual space and the movement information of the autonomous driving device in the actual space, and use the position information of the autonomous driving device in the actual space decoded from the bitstream to convert the position information of the target in the actual space included in the actual space detection information related to the target decoded from the bitstream, thereby calculating the display position information of the target. Use the movement information of the autonomous driving device in the actual space decoded from the bitstream to convert the movement information of the target in the actual space included in the actual space detection information related to the target decoded from the bitstream, thereby calculating the display movement information of the target.

[0063] According to the 13th aspect, it is possible to convert the relative position of the target into an absolute position for display, and it is possible to convert the relative speed and relative movement direction of the target into an absolute speed and absolute movement direction for display.

[0064] In the decoding device according to the 14th aspect of the present disclosure, in the 10th or 11th aspect, the circuit may calculate the display movement information of the target based on a plurality of position information of the target in the actual space at a plurality of times included in the actual space detection information related to the target decoded from the bitstream.

[0065] According to the 14th aspect, even when the bitstream received from the autonomous driving device does not include the movement information of the target in the actual space, it is possible to appropriately calculate the display movement information of the target based on the plurality of position information of the target in the actual space at a plurality of times.

[0066] In the decoding device according to the 15th aspect of the present disclosure, in any one of the 1st to 14th aspects, the decoding device is mounted on another autonomous driving device different from the autonomous driving device, and the circuit may further control the operation of the other autonomous driving device based on the in-image detection information and the actual space detection information related to the target decoded from the bitstream received from the autonomous driving device, and the in-image detection information and the actual space detection information related to the target included in the image captured by the imaging unit of the other autonomous driving device.

[0067] According to the 15th aspect, by mutually communicating within a plurality of autonomous driving devices information on in-image detection related to a target and actual space detection information, each autonomous driving device can effectively utilize these pieces of information received from another autonomous driving device for the motion control of its own device, thereby enabling improvement in the safety of the autonomous driving device such as collision avoidance performance.

[0068] In the decoding device according to the 16th aspect of the present disclosure, in the 15th aspect, the circuit can convert the actual space detection information related to the target using a coordinate system based on the autonomous driving device, which is decoded from the bitstream received from the autonomous driving device, into the actual space detection information related to the target using a coordinate system based on the other autonomous driving device.

[0069] According to the 16th aspect, through the coordinate system conversion process, each autonomous driving device can use the actual space detection information related to the target detected by another autonomous driving device as the actual space detection information related to the target detected by its own autonomous driving device.

[0070] The encoding device according to the 17th aspect of the present disclosure is mounted on an autonomous driving device having an imaging unit, and the encoding device includes: a circuit; and a memory connected to the circuit, and the circuit performs the following processes: encoding an image captured by the imaging unit into a bitstream, encoding the in-image detection information, which is the detection information within the image related to the target included in the image, into the bitstream, and encoding the detection information in the actual space related to the target, that is, the actual space detection information, into the bitstream.

[0071] According to the 17th aspect, by encoding the in-image detection information and the actual space detection information related to a target existing around the autonomous driving device into a bitstream, these pieces of information can be effectively utilized on the side of the decoding device that receives the bitstream, thereby enabling improvement in the safety of the autonomous driving device such as collision avoidance performance. In addition, it is possible to improve the decision-making of the motion of the autonomous driving device for improving the safety of the autonomous driving device in a complex environment.

[0072] In the encoding device according to the 18th aspect of the present disclosure, in the 17th aspect, the in-image detection information related to the target may include the position information of the target within the image.

[0073] According to the 18th aspect, it is possible to effectively utilize the position information of the target within the image on the side of the decoding device.

[0074] In the 19th aspect of the present disclosure, in the 17th or 18th aspect, the actual space detection information related to the target may include at least one of the position information of the target in the actual space and the movement information of the target in the actual space.

[0075] According to the 19th aspect, it is possible to effectively utilize the position information and movement information of the target in the actual space on the decoding device side.

[0076] In the 20th aspect of the present disclosure, in the 19th aspect, the position information of the target may include coordinate values in a circular coordinate system based on the autonomous driving device, coordinate values in a Cartesian coordinate system based on the autonomous driving device, or coordinate values in a geographic coordinate system.

[0077] According to the 20th aspect, it is possible to accurately determine the position of the target through the position information including coordinate values.

[0078] In the 21st aspect of the present disclosure, in the 19th or 20th aspect, the movement information of the target may include at least one of the absolute speed of the target or the relative speed of the target with respect to the autonomous driving device, and the absolute movement direction of the target or the relative movement direction of the target with respect to the autonomous driving device.

[0079] According to the 21st aspect, it is possible to accurately determine at least one of the speed and the movement direction of the target through the movement information including at least one of the absolute speed or the relative speed, and the absolute movement direction or the relative movement direction.

[0080] In the 22nd aspect of the present disclosure, in any one of the 17th to 21st aspects, the circuit may further encode the actual space detection information related to the autonomous driving device into the bitstream, and the actual space detection information related to the autonomous driving device includes the position information of the autonomous driving device in the actual space and the movement information of the autonomous driving device in the actual space.

[0081] According to the 22nd aspect, it is possible to effectively utilize the position information and movement information of the autonomous driving device in the actual space on the decoding device side.

[0082] In the 23rd aspect of the present disclosure, in the 22nd aspect, the position information of the autonomous driving device may include coordinate values in a geographic coordinate system.

[0083] According to the 23rd aspect, it is possible to accurately determine the absolute position of the autonomous driving device through the position information including coordinate values in a geographic coordinate system.

[0084] In the 24th mode of the present disclosure, in the 22nd mode or the 23rd mode, the movement information of the autonomous driving device may include at least one of the absolute speed and the absolute movement direction of the autonomous driving device.

[0085] According to the 24th mode, at least one of the absolute speed and the absolute movement direction of the autonomous driving device can be accurately determined by the movement information.

[0086] In the 25th mode of the present disclosure, in any of the 17th mode to the 24th mode, the circuit may encode the in-image detection information and the actual space detection information related to the target into the SEI area of the bitstream.

[0087] According to the 25th mode, by encoding the in-image detection information and the actual space detection information related to the target into the SEI area of the bitstream, processing can be easily performed on the decoder side.

[0088] In the 26th mode of the present disclosure, in any of the 17th mode to the 25th mode, the autonomous driving device may further include: a detection unit that detects the target in the actual space, and the circuit derives the actual space detection information related to the target based on the value obtained by converting the detection value of the detection unit from the coordinate system of the detection unit to the coordinate system of the autonomous driving device.

[0089] According to the 26th mode, through the conversion process from the own coordinate system of the detection unit to the common coordinate system of the autonomous driving device, the actual space detection information related to the target can be appropriately derived.

[0090] The decoding method according to the 27th mode of the present disclosure is performed by a decoding device as follows: receiving a bitstream from an autonomous driving device having a photographing unit, decoding an image photographed by the photographing unit according to the bitstream, decoding the in-image detection information, that is, the in-image detection information related to the target included in the image, according to the bitstream, and decoding the detection information in the actual space, that is, the actual space detection information related to the target, according to the bitstream.

[0091] According to the 27th mode, the in-image detection information and the actual space detection information related to the targets existing around the autonomous driving device can be effectively utilized on the decoder side, thereby improving the safety of the autonomous driving device such as the collision avoidance performance. In addition, the decision-making of the operation of the autonomous driving device for improving the safety of the autonomous driving device in a complex environment can be improved.

[0092] The encoding method involved in the 28th mode of the present disclosure is performed by an encoding device installed in an autonomous driving device having a camera unit to perform the following processing: encoding an image captured by the camera unit into a bit stream, encoding detection information within the image related to a target contained in the image, i.e., in-image detection information, into the bit stream, and encoding detection information in an actual space related to the target, i.e., actual space detection information, into the bit stream.

[0093] According to the 28th aspect, by encoding the in-screen detection information and the real-space detection information about the target existing around the autonomous driving device into a bit stream, the information can be effectively used on the decoding device side receiving the bit stream, thereby improving the safety of the autonomous driving device such as collision avoidance performance. In addition, the intention decision of the action of the autonomous driving device for improving the safety of the autonomous driving device in a complex environment can be improved.

[0094] (Embodiments of the present disclosure)

[0095] Hereinafter, the embodiments of the present disclosure will be described in detail using the drawings. In addition, elements given the same reference numerals in different drawings represent the same or corresponding elements.

[0096] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are an example and are not intended to limit the present disclosure. In addition, for the structural elements in the following embodiments that are not recorded in the independent technical solutions representing the highest concept, they are described as arbitrary structural elements. In addition, in all embodiments, each content can also be combined.

[0097] (First embodiment)

[0098] Figure 1 1 is a diagram schematically showing the configuration of an information processing system according to the first embodiment of the present disclosure. The information processing system includes an autonomous driving device 1 , a transmission path NW, a decoding device 2 , and a display device 3 .

[0099] The autonomous driving device 1 is, for example, an autonomous driving vehicle that delivers goods without human intervention in a goods delivery system. The autonomous driving device 1 includes an encoding device 11, a camera 12, and an object detection unit 13. Figure 1 One autonomous driving device 1 is shown in FIG. 1 , but a plurality of autonomous driving devices 1 of the same structure may also be present.

[0100] The encoding device 11 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 is configured to include a processor such as a CPU. The memory 22 is configured to include any recording medium such as a ROM, a RAM, an HDD, an SSD, or a semiconductor memory. The memory 22 stores data such as the processing object or the data during processing based on the circuit 21.

[0101] The imaging unit 12 is configured to include a camera having an optical system and a CMOS image sensor or the like. The imaging unit 12 images the surroundings of the autonomous driving device 1 (particularly in front of the moving direction).

[0102] The target detection unit 27 is configured to include a LiDAR (Light Detection And Ranging), etc. The target detection unit 27 detects targets existing around the autonomous driving device 1 in the actual space. The target detection unit 27 inputs the detection information in the actual space related to the target, that is, the actual space detection information, into the circuit 21. The actual space detection information includes the distance to the target, the direction where the target exists, and the shape of the target, etc.

[0103] The transmission path NW is the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), or any combination thereof. It is desirable that the transmission path NW is a dedicated network that ensures secure communication through access restrictions, etc.

[0104] The decoding device 2 and the display device 3 are arranged, for example, in a monitoring center where an operator who remotely monitors the autonomous driving device 1 is stationed.

[0105] The decoding device 2 includes a circuit 31 and a memory 32 connected to the circuit 31. The circuit 31 is configured to include a processor such as a CPU. The memory 32 is configured to include any recording medium such as a ROM, a RAM, an HDD, an SSD, or a semiconductor memory. The memory 32 stores data such as the processing object or the data during processing based on the circuit 31.

[0106] The display device 3 is a liquid crystal display, an organic EL display, or the like.

[0107] The circuit 21 of the encoding device 11 sends the bitstream BS to the decoding device 2 via the transmission path NW. The circuit 31 of the decoding device 2 receives the bitstream BS and generates a display image for the human vision of the operator. The display device 3 displays the display image.

[0108] Figure 2 It is a flowchart showing the process executed by the circuit 21 of the encoding device 11.

[0109] In step SP11, circuit 21 acquires the image captured by imaging unit 12. The image includes objects existing around autonomous driving device 1.

[0110] Figure 3 FIG. is an example schematically showing the driving state of autonomous driving device 1. On sidewalk 5, there are autonomous driving devices 1A, 1B and pedestrians 6A to 6C. Autonomous driving device 1A travels upward in the drawing. Autonomous driving device 1B travels downward in the drawing in the forward direction of the moving direction of autonomous driving device 1A. Pedestrian 6A walks rightward in the drawing in the forward direction of the moving direction of autonomous driving device 1A. Pedestrian 6B walks downward in the drawing in the forward direction of the moving direction of autonomous driving device 1A. Pedestrian 6C walks upward in the drawing in the backward direction of the moving direction of autonomous driving device 1A.

[0111] Figure 4 FIG. is an example schematically showing image 100 captured by imaging unit 12 of autonomous driving device 1A. Image 100 includes autonomous driving device 1B moving toward the own machine (autonomous driving device 1A), pedestrians 6B, and pedestrian 6A crossing in front of the own machine.

[0112] In step SP12, circuit 21 acquires actual space detection information related to the objects included in image 100 from target detection unit 13. For each of autonomous driving device 1B and pedestrians 6A, 6B as objects, the actual space detection information related to the object includes the distance to the object, the direction where the object exists, the shape of the object, etc. Step SP12 may also be executed simultaneously with step SP11.

[0113] In step SP13, circuit 21 analyzes image 100 acquired in step SP11. The analysis of image 100 includes, for example, object detection processing and object recognition processing using a presumption model completed by machine learning. Circuit 21 generates detection information within image 100, i.e., in-image detection information, related to the objects included in image 100. The in-image detection information includes the position information of the objects within image 100 as a result of the object detection processing. The in-image detection information may also include the attribute information of the objects within image 100, etc. as a result of the object recognition processing. The attribute indicates the category of the object such as an autonomous driving device, a person, or a bicycle.

[0114] Figure 5 FIG. is an example schematically showing the analysis result of image 100 by circuit 21. Circuit 21 sets bounding box BB surrounding the object as the position information of the object within image 100 in image 100. Specifically, circuit 21 sets bounding box BB1 surrounding autonomous driving device 1B, bounding box BB2 surrounding pedestrian 6A, and bounding box BB3 surrounding pedestrian 6B.

[0115] The circuit 21 specifies the position and shape of the bounding box BB based on the coordinate values of a specific vertex (e.g., the upper left vertex) of the bounding box BB, as well as the height and width of the bounding box BB. Alternatively, the circuit 21 can also specify the position and shape of the bounding box BB based on the coordinate values of the center point of the bounding box BB, as well as the height and width of the bounding box BB. Alternatively, the circuit 21 can also specify the position and shape of the bounding box BB based on the coordinate values of the four vertices of the bounding box BB. Alternatively, the circuit 21 can also specify the position and shape of the bounding box BB based on the coordinate values of two vertices located on the diagonal of the bounding box BB.

[0116] In addition, the circuit 21 establishes a correspondence between the object in the actual space detected by the object detection unit 13 and the object in the image 100 detected by image analysis.

[0117] Figure 6 FIG. is a diagram schematically showing an example of the actual space detection information and the in-image detection information related to the object. The actual space detection information includes position information and movement information. The position information includes distance and direction. The movement information includes speed and movement direction. The circuit 21 calculates the speed and movement direction of the object based on the detection results of the positions of the object detection unit 13 at multiple times. However, the calculation of the movement information including the speed and movement direction of the object can also be performed by the circuit 31 of the decoding device 2 instead of the circuit 21 of the encoding device 11. The in-image detection information includes information indicating the position and shape of the corresponding bounding box BB.

[0118] The origin of the coordinate system of the object detection unit 13 is the center point of the object detection unit 13, and the origin of the coordinate system of the autonomous driving device 1A is the center point of the autonomous driving device 1A, and the two are different. Therefore, the circuit 21 derives the actual space detection information related to the object based on the value obtained by converting the detection value of the object detection unit 13 from its own coordinate system to the common coordinate system of the autonomous driving device 1A.

[0119] Figure 7 FIG. is a diagram showing an example of the coordinate system of the autonomous driving device 1A. In Figure 7 the example shown, the coordinate system of the autonomous driving device 1A is a circular coordinate system, and the origin O of the coordinate system is the center point of the autonomous driving device 1A. The angle 0° of the coordinate system corresponds to the moving direction of the autonomous driving device 1A. The position of the object is shown by the distance coordinate value and the angle coordinate value in the circular coordinate system. In addition, the moving direction of the object is shown by the angle coordinate value in the circular coordinate system.

[0120] In Figure 6In the example shown, the distance, direction, speed, and moving direction are the relative distance, relative direction, relative speed, and relative moving direction of the target with respect to the autonomous driving device 1A.

[0121] Regarding the autonomous driving device 1B as the target, the corresponding bounding box BB is the bounding box BB1, the relative distance is 5.0 m, the relative direction is 30.0°, the relative speed is 8.5 km / h, and the relative moving direction is 180.0°. Additionally, regarding the pedestrian 6A as the target, the corresponding bounding box BB is the bounding box BB2, the relative distance is 9.8 m, the relative direction is -28.5°, the relative speed is 7.7 km / h, and the relative moving direction is -90.0°. Additionally, regarding the pedestrian 6B as the target, the corresponding bounding box BB is the bounding box BB3, the relative distance is 13.1 m, the relative direction is 10.0°, the relative speed is 12.5 km / h, and the relative moving direction is 180.0°.

[0122] Furthermore, the coordinate system of the autonomous driving device 1A is not limited to the above-mentioned circular coordinate system and can also be a Cartesian coordinate system. For example, in the Cartesian coordinate system, the front-rear direction of the autonomous driving device 1A can be set as the X-axis, the left-right direction of the autonomous driving device 1A can be set as the Y-axis, and the center point of the autonomous driving device 1A can be set as the origin, and the position of the target can be represented by the values of the X-axis and the Y-axis.

[0123] In step SP14, the circuit 21 encodes the image 100 obtained in step SP11 into the bitstream BS.

[0124] Figure 8 is a diagram schematically showing the data structure of the bitstream BS. The bitstream BS has a header 41 and a payload 42. The header 41 includes a supplemental enhancement information (SEI) region 43 for storing additional information.

[0125] The circuit 21 encodes the image 100 including the target into the payload 42 of the bitstream BS.

[0126] In step SP15, the circuit 21 encodes the in-image detection information related to the target into the bitstream BS. The circuit 21 encodes the in-image detection information generated in step SP13 into a given part of the header 41. The given part is the SEI region 43. The given part can also be a VPS, SPS, PPS, PH, SH, APS, or a tile header, etc. Alternatively, the given part can also be an annotated region SEI (ARSEI) region for storing bounding box information.

[0127] In step SP16, circuit 21 encodes the actual space detection information related to the target into bitstream BS. Circuit 21 encodes the actual space detection information obtained in step SP12 into a given part of header 41. The given part is SEI area 43. The given part can also be VPS, SPS, PPS, PH, SH, APS, or block header, etc. Alternatively, the given part can also be the ARSEI area for storing bounding box information. In addition, steps SP14 - SP16 can also be executed simultaneously.

[0128] Figure 9 It is a diagram that simply shows an example of the syntax. num_objects represents the number of targets included in image 100. first_information contains the in-image detection information related to the target. second_information contains the actual space detection information related to the target.

[0129] In step SP17, circuit 21 sends bitstream BS to decoding device 2.

[0130] Figure 10 It is a flowchart showing the processing flow executed by circuit 31 of decoding device 2.

[0131] In step SP21, circuit 31 receives bitstream BS from autonomous driving device 1A.

[0132] In step SP22, circuit 31 decodes image 100 containing the target according to payload 42 of bitstream BS.

[0133] In step SP23, circuit 31 decodes the in-image detection information related to the target according to header 41 of bitstream BS. When the in-image detection information is encoded into SEI area 43 of header 41, circuit 31 decodes the in-image detection information according to SEI area 43. When the in-image detection information is encoded into the ARSEI area of header 41, circuit 31 decodes the in-image detection information according to the ARSEI area.

[0134] In step SP24, circuit 31 decodes the actual space detection information related to the target according to header 41 of bitstream BS. When the actual space detection information is encoded into SEI area 43 of header 41, circuit 31 decodes the actual space detection information according to SEI area 43. When the actual space detection information is encoded into the ARSEI area of header 41, circuit 31 decodes the actual space detection information according to the ARSEI area. In addition, steps SP22 - SP24 can also be executed simultaneously.

[0135] In step SP25, circuit 31 generates a display image 200 for the human vision of an operator based on the images and information decoded in steps SP22 to SP24.

[0136] Figure 11 FIG. is a diagram schematically showing an example of the display image 200. The display image 200 includes an image 100, position information for displaying a target in the actual space, and movement information D1 to D3 for displaying a target in the actual space.

[0137] The position information for display includes frames F1 to F3 corresponding to the bounding boxes BB1 to BB3, and distance information indicating the relative distance of each target with respect to the autonomous driving device 1A. Circuit 31 uses the position information of the target in the actual space included in the actual space detection information decoded from the bitstream BS as the position information for displaying the target.

[0138] The movement information D1 to D3 for display includes relative speed information indicating the relative speed of each target with respect to the autonomous driving device 1A, and relative movement direction information indicating the relative movement direction of each target with respect to the autonomous driving device 1A. In Figure 11 the example shown, the relative movement direction information is represented by a thick-line arrow graphic. Circuit 31 uses the movement information of the target in the actual space included in the actual space detection information decoded from the bitstream BS as the movement information D1 to D3 for displaying the target.

[0139] In addition, circuit 31 emphasizes and displays frames F1 and F3 by thick lines or coloring or the like. The frames F1 and F3 correspond to targets that may collide with or approach the own vehicle (autonomous driving device 1A) among the multiple targets included in the display image 200. Circuit 31 determines the possibility of collision with or approach to the own vehicle based on the relative position, relative distance, and relative movement direction of the target with respect to the own vehicle. Thus, an operator visually recognizing the display image 200 can quickly identify the targets among the multiple targets included in the display image 200 that should be focused on and monitored.

[0140] According to the encoding device 11 according to the present embodiment, by encoding the in-frame detection information and the actual space detection information related to the targets existing around the autonomous driving device 1 into the bitstream BS, these information can be effectively utilized on the side of the decoding device 2 that receives the bitstream BS, and thus the safety of the autonomous driving device 1 such as the collision avoidance performance can be improved. In addition, the decision-making of the operation of the autonomous driving device 1 for improving the safety of the autonomous driving device 1 in a complex environment can be improved.

[0141] In addition, according to the encoding device 11 according to the present embodiment, the in-image detection information includes the position information of the target in the image 100, whereby the position information of the target in the image 100 can be effectively utilized on the decoding device 2 side.

[0142] In addition, according to the encoding device 11 according to the present embodiment, the actual space detection information includes the position information and movement information of the target in the actual space, whereby the position information and movement information of the target in the actual space can be effectively utilized on the decoding device 2 side. In addition, one of the position information and movement information of the target in the actual space may be omitted.

[0143] In addition, according to the encoding device 11 according to the present embodiment, the position information of the target includes coordinate values in a circular coordinate system based on the autonomous driving device 1, whereby the position of the target can be accurately determined by the position information including the coordinate values.

[0144] In addition, according to the encoding device 11 according to the present embodiment, the movement information of the target includes the relative speed and relative movement direction of the target with respect to the autonomous driving device 1, whereby the speed and movement direction of the target can be accurately determined.

[0145] In addition, according to the encoding device 11 according to the present embodiment, the in-image detection information and the actual space detection information related to the target are encoded into the SEI area 43 of the bitstream, whereby processing can be easily performed on the decoding device 2 side.

[0146] In addition, according to the encoding device 11 according to the present embodiment, the actual space detection information related to the target can be appropriately derived by conversion processing from the own coordinate system of the target detection unit 13 to the common coordinate system of the autonomous driving device 1.

[0147] In addition, according to the decoding device 2 according to the present embodiment, the in-image detection information and the actual space detection information related to the target existing around the autonomous driving device 1 can be effectively utilized, whereby the safety of the autonomous driving device 1 such as the collision avoidance performance can be improved. In addition, the decision-making of the operation of the autonomous driving device 1 for improving the safety of the autonomous driving device 1 in a complex environment can be improved.

[0148] In addition, according to the decoding device 2 according to the present embodiment, the in-image detection information includes the position information of the target in the image 100, whereby the position information of the target in the image 100 can be effectively utilized.

[0149] In addition, according to the decoding device 2 according to the present embodiment, the actual space detection information includes the position information and movement information of the target in the actual space, whereby the position information and movement information of the target in the actual space can be effectively utilized.

[0150] In addition, according to the decoding device 2 according to the present embodiment, the target position information includes coordinate values in a circular coordinate system with the autonomous driving device 1 as a reference, whereby the position of the target can be accurately determined by the position information including the coordinate values.

[0151] In addition, according to the decoding device 2 according to the present embodiment, the target movement information includes the relative speed and relative movement direction of the target with respect to the autonomous driving device 1, whereby the speed and movement direction of the target can be accurately determined.

[0152] In addition, according to the decoding device 2 according to the present embodiment, the in-image detection information and the actual space detection information related to the target are encoded in the SEI area 43 of the bitstream, whereby processing can be easily performed on the decoding device 2 side.

[0153] In addition, according to the decoding device 2 according to the present embodiment, by visually recognizing the display image 200 including the display position information and the display movement information of the target in the actual space, an operator can instantaneously and accurately grasp the situation around the autonomous driving device 1.

[0154] In addition, according to the decoding device 2 according to the present embodiment, the position information of the target included in the actual space detection information is used as the display position information, and the movement information of the target included in the actual space detection information is used as the display movement information, whereby the display position information and the display movement information can be easily generated.

[0155] Hereinafter, each modification according to the first embodiment described above will be described. The modifications described below can be applied in any combination.

[0156] (First Modification)

[0157] Figure 12 FIG. is an example showing a simplified display image 201 as a modification of the display image 200. The display image 201 includes a graphic G0 imitating the autonomous driving device 1A as the host, a graphic G1 imitating the autonomous driving device 1B as the target, a graphic G2 imitating the pedestrian 6A as the target, a graphic G3 imitating the pedestrian 6B as the target, the display position information of the target in the actual space, and the display movement information D1 to D3 of the target in the actual space. The display image 201 has a coordinate plane of a circular coordinate system with the position of the graphic G0 as the origin O.

[0158] The position information for display includes graphics G1 to G3 and distance information indicating the relative distance of each target with respect to the autonomous driving device 1A. For example, the distance coordinate value L1 and the angular coordinate value θ1 of the circular coordinate system related to graphic G1 correspond to the distance coordinate value and the angular coordinate value included in the actual space detection information related to the autonomous driving device 1B.

[0159] The movement information D1 to D3 for display includes relative speed information indicating the relative speed of each target with respect to the autonomous driving device 1A, and relative movement direction information indicating the relative movement direction of each target with respect to the autonomous driving device 1A. The relative movement direction information is represented as a graphic of a thick arrow.

[0160] The circuit 31 can also perform an emphasized display on the graphics G1 and G3 corresponding to the targets among the multiple targets that have the possibility of colliding with or approaching the own device by means of flashing or coloring, etc.

[0161] According to this modification example, the operator of the visual recognition display image 201 can quickly determine the target among the multiple targets that should be focused on and monitored.

[0162] (Second Modification Example)

[0163] Figure 13 FIG. is an example diagram showing a simplified display image 202 as a modification example of the display image 200. The display image 202 includes graphics G0 to G3, position information for display of the targets in the actual space, and movement information D1 to D3 for display of the targets in the actual space. The display image 202 has a coordinate plane of a Cartesian coordinate system with the position of graphic G0 as the origin O. The Cartesian coordinate system has orthogonal X-axis and Y-axis.

[0164] In this modification example, the coordinate system of the autonomous driving device 1A is a Cartesian coordinate system, and the origin O of the coordinate system is the center point of the autonomous driving device 1A. The X-axis of the coordinate system corresponds to the moving direction of the autonomous driving device 1A. The position of the target is shown by the X coordinate value and the Y coordinate value in the Cartesian coordinate system.

[0165] The position information for display includes graphics G1 to G3 and distance information indicating the relative distance of each target with respect to the autonomous driving device 1A. For example, the X coordinate value X1 and the Y coordinate value Y1 of the Cartesian coordinate system related to graphic G1 correspond to the X coordinate value and the Y coordinate value included in the actual space detection information related to the autonomous driving device 1B.

[0166] The display moving information D1 to D3 includes relative speed information indicating the relative speed of each target with respect to the autonomous driving device 1A, and relative movement direction information indicating the relative movement direction of each target with respect to the autonomous driving device 1A. The relative movement direction information is represented as a thick-line arrow graphic.

[0167] The circuit 31 can also perform emphasized display on the graphics G1 and G3 corresponding to the targets among the multiple targets that have the possibility of colliding with or approaching the own device, such as by flashing or coloring.

[0168] According to this modification example, the operator of the visual recognition display image 201 can quickly determine the target among the multiple targets that should be focused on and monitored.

[0169] (Third Modification Example)

[0170] In this modification example, the correspondence in the case where the actual space detection information decoded from the bitstream BS does not include the movement information (speed and movement direction) of the target is described.

[0171] Figure 14 is from Figure 13 The figure shows the extraction of the graphic G1 from the display image 202 shown. The circuit 31 calculates the display moving information D1 of the target based on the multiple position information of the target corresponding to the graphic G1 included in the actual space detection information decoded from the bitstream BS at multiple times. The circuit 31 calculates the movement distance and movement direction of the target in the actual space based on the X coordinate value X1(t) and Y coordinate value Y1(t) of the target at a certain time t, and the X coordinate value X1(t - 1) and Y coordinate value Y1(t - 1) of the target at the previous time t - 1. In addition, the circuit 31 calculates the speed of the target based on the calculated movement distance and the time difference Δt between the time t and the time t - 1.

[0172] Equation (1) shows an example of the calculation formula for the speed V of the target.

[0173] [Mathematical Formula 1]

[0174]

[0175] Equation (2) shows an example of the calculation formula for the movement direction A of the target.

[0176] [Mathematical Formula 2]

[0177]

[0178] The circuit 31 generates the display moving information D1, and the display moving information D1 includes speed information indicating the calculated speed and movement direction information indicating the calculated movement direction. In addition, inFigure 14 In the example shown, the Cartesian coordinate system is described, but the same processing can also be performed with the circular coordinate system.

[0179] According to this modification example, even when the bit stream BS received from the autonomous driving device 1 does not include the movement information of the target in the actual space, the circuit 31 can appropriately calculate the display movement information D1 to D3 of the target based on the multiple position information of the target at multiple times in the actual space.

[0180] (Fourth Modification Example)

[0181] Figure 15 FIG. is a diagram schematically showing an example of the display image 201. With the graphic G0 as the center, a boundary Z is set around it. The boundary Z corresponds to a circle with a radius of, for example, 1 m centered on the position of the autonomous driving device 1A in the actual space.

[0182] When a certain target enters within the boundary Z, the circuit 31 determines that there is a high possibility of collision between the autonomous driving device 1A and the target, and performs an alarm display within the display image 201. For example, when the autonomous driving device 1B corresponding to the graphic G1 enters within the boundary Z, the circuit 31 emphasizes the display of the graphic G0 and / or the graphic G1 by flashing or coloring. In addition, in Figure 15 In the example shown, the circular coordinate system is described, but the same processing can also be performed with the Cartesian coordinate system.

[0183] According to this modification example, an operator who visually recognizes the display image 201 can quickly grasp the occurrence of a situation where there is a high possibility of collision between the autonomous driving device 1A and the target, and can perform collision avoidance control such as making the autonomous driving device 1A stop urgently by remote operation. In addition, the collision avoidance control can also be executed by automatic control based on the circuit 31 instead of the manual operation performed by the operator.

[0184] (Fifth Modification Example)

[0185] Figure 16 FIG. is a diagram schematically showing the structure of the information processing system according to the fifth modification example. The autonomous driving device 1, in addition to Figure 1 the structure shown in, further includes a position detection unit 14, a speed detection unit 15, and an azimuth detection unit 16.

[0186] The position detection unit 14, the speed detection unit 15, and the orientation detection unit 16 input the detection information in the actual space related to the autonomous driving device 1, that is, the actual space detection information, into the circuit 21. The actual space detection information includes position information and movement information. The position information related to the autonomous driving device 1 includes the absolute position. The movement information related to the autonomous driving device 1 includes the absolute speed and the absolute movement direction.

[0187] The position detection unit 14 is configured to include a GPS receiver or the like. The position detection unit 14 detects the absolute position of the autonomous driving device 1 and inputs the detected value into the circuit 21. The absolute position is the coordinate value (latitude and longitude) in a geographical coordinate system such as the world coordinates.

[0188] The speed detection unit 15 is configured to include a tachymeter. The speed detection unit 15 detects the absolute speed of the autonomous driving device 1 and inputs the detected value into the circuit 21.

[0189] The orientation detection unit 16 is configured to include an acceleration sensor or a gyro sensor or the like. The orientation detection unit 16 detects the front direction or the absolute movement direction of the autonomous driving device 1 and inputs the detected value into the circuit 21.

[0190] Based on the values obtained by converting the respective detection values obtained by the position detection unit 14, the speed detection unit 15, and the orientation detection unit 16 from their own coordinate systems to the common coordinate system of the autonomous driving device 1, the circuit 21 derives the actual space detection information related to the autonomous driving device 1.

[0191] Figure 17 It is a flowchart showing the process executed by the circuit 21 of the encoding device 11. Regarding the same process as the flowchart shown in Figure 2 The description is omitted for the same process as the flowchart shown.

[0192] In step SP31, the circuit 21 acquires the actual space detection information related to the autonomous driving device 1A from the position detection unit 14, the speed detection unit 15, and the orientation detection unit 16.

[0193] In step SP32, the circuit 21 encodes the actual space detection information related to the autonomous driving device 1A into a bitstream BS. The circuit 21 encodes the actual space detection information acquired in step SP31 into a given part of the header 41. The given part is the SEI area 43. The given part may also be the VPS, SPS, PPS, PH, SH, APS, or block header, etc. Or, the given part may also be the ARSEI area.

[0194] Figure 18This is a diagram that schematically shows an example of grammar. The third_information includes actual space detection information related to the autonomous driving device 1A.

[0195] Figure 19 This is a flowchart showing the process executed by the circuit 31 of the decoding device 2. Regarding the same process as the Figure 10 flowchart shown, the description is omitted.

[0196] In step SP33, the circuit 31 decodes the actual space detection information related to the autonomous driving device 1A based on the header 41 of the bitstream BS.

[0197] In step SP25, the circuit 31 generates a display image 200 for the human vision of the operator based on the images and information decoded in steps SP22 - SP24, SP33.

[0198] Figure 20 This is a diagram that schematically shows an example of the display image 200. When generating the display image 200, the circuit 31 can also use the absolute position information of the autonomous driving device 1A in the actual space decoded in step SP33 to convert the relative position information of the target in the actual space decoded in step SP24 into the absolute position information of the target in the actual space.

[0199] In addition, the circuit 31 can also use the absolute movement information of the autonomous driving device 1A in the actual space decoded in step SP33 to convert the relative movement information of the target in the actual space decoded in step SP24 into the absolute movement information of the target in the actual space. In the Figure 20 example of the display movement information D1 - D3 shown, the relative speed of each target with respect to the autonomous driving device 1A is converted into the absolute speed.

[0200] In addition, the circuit 31 can also include the absolute movement information and absolute position information of the autonomous driving device 1A in the actual space decoded in step SP33 in the display image 200. In the Figure 20 example shown, the display image 200 includes the local information C0 representing the absolute speed and absolute position of the autonomous driving device 1A. The local information C0 can also include the absolute movement direction of the autonomous driving device 1A. The absolute movement direction can also be represented, for example, by azimuth information indicating the movement direction of the autonomous driving device 1A.

[0201] In addition, not limited to the display image 200, the process according to this modification example can also be applied to the display images 201, 202.

[0202] According to this modified example, the absolute position information and absolute movement information of the autonomous driving device 1A in the actual space can be effectively utilized on the decoding device 2 side.

[0203] (Sixth Modified Example)

[0204] Figure 21 This is a diagram schematically showing an example of a display image 203 as a modified example of the display image 202. In addition to the graphics G0 to G3 shown in Figure 13 the display image 203 also includes a graphic G4 that mimics the target pedestrian 6C.

[0205] The pedestrian 6C is photographed by the photographing unit 12 of the autonomous driving device 1B. The circuit 21 of the encoding device 11 of the autonomous driving device 1B encodes the image photographed by the photographing unit 12, the in-image detection information related to the target included in the image, and the actual space detection information related to the target into a bitstream BS and sends it to the decoding device 2. This actual space detection information includes the relative position information and relative movement information of the pedestrian 6C with respect to the autonomous driving device 1B, and the absolute position information and absolute movement information of the autonomous driving device 1B.

[0206] The circuit 31 of the decoding device 2 receives a plurality of bitstreams BS from a plurality of autonomous driving devices 1A and 1B. The circuit 31 generates a display image 203 based on a plurality of images decoded from the plurality of bitstreams BS. The circuit 31 uses the absolute position information and absolute movement information of the autonomous driving device 1B to convert the relative position information and relative movement information of the pedestrian 6C with respect to the autonomous driving device 1B, thereby calculating the absolute position information and absolute movement information of the pedestrian 6C. Thus, the graphic G4 corresponding to the pedestrian 6C that cannot be photographed by the photographing unit 12 of the autonomous driving device 1A can be included in the display image 203. The display movement information D0 to D4 includes absolute speed information indicating the absolute speed of each target and absolute movement direction information indicating the absolute movement direction of each target. In addition, in the Figure 21 example shown, the Cartesian coordinate system is described, but the same processing can also be performed for the circular coordinate system.

[0207] According to this modified example, since the dead angles during photographing can be mutually compensated by a plurality of images photographed by a plurality of autonomous driving devices 1A and 1B, the convenience of the display image 203 can be improved.

[0208] (Second Embodiment)

[0209] In the second embodiment, the communication between a plurality of autonomous driving devices 1 is described.

[0210] Figure 22FIG. is a diagram schematically showing the configuration of an information processing system according to a second embodiment of the present disclosure. The information processing system includes autonomous driving devices 1A and 1B.

[0211] The autonomous driving device 1A includes: an encoding device 11A, a photographing unit 12A, a target detection unit 13A, a position detection unit 14A, a speed detection unit 15A, an azimuth detection unit 16A, a decoding device 2A, and a driving unit 17A. The encoding device 11A includes a circuit 21A and a memory 22A. The decoding device 2A includes a circuit 31A and a memory 32A. The driving unit 17A is configured to include a driving motor of the autonomous driving device 1A and the like.

[0212] The autonomous driving device 1B includes: an encoding device 11B, a photographing unit 12B, a target detection unit 13B, a position detection unit 14B, a speed detection unit 15B, an azimuth detection unit 16B, a decoding device 2B, and a driving unit 17B. The encoding device 11B includes a circuit 21B and a memory 22B. The decoding device 2B includes a circuit 31B and a memory 32B. The driving unit 17B is configured to include a driving motor of the autonomous driving device 1B and the like.

[0213] The circuit 21B of the encoding device 11B transmits a bit stream BSB, and the circuit 31A of the decoding device 2A receives the bit stream BSB. The encoding device 11A is mounted on the autonomous driving device 1B, and the decoding device 2A is mounted on another autonomous driving device 1A different from the autonomous driving device 1B. Similarly, the circuit 21A of the encoding device 11A transmits a bit stream BSA, and the circuit 31B of the decoding device 2B receives the bit stream BSA.

[0214] Figure 23 FIG. is a flowchart showing the process executed by the circuit 31A of the decoding device 2A.

[0215] In step SP51, the circuit 31A receives the bit stream BSB from the autonomous driving device 1B.

[0216] In step SP52, the circuit 31A decodes the image 100B including the target according to the payload 42 of the bit stream BSB.

[0217] Figure 24 FIG. is a diagram schematically showing an example of an image 100B captured by the photographing unit 12B of the autonomous driving device 1B. The image 100B includes the autonomous driving device 1A and a pedestrian 6C walking behind the autonomous driving device 1A as targets.

[0218] In step SP53, the circuit 31A decodes the in-image detection information related to the target included in the image 100B according to the header 41 of the bit stream BSB.

[0219] In step SP54, circuit 31A decodes the actual space detection information related to the target included in image 100B based on the header 41 of bitstream BSB.

[0220] In step SP55, circuit 31A decodes the actual space detection information related to autonomous driving device 1B based on the header 41 of bitstream BSB.

[0221] In step SP56, circuit 31A obtains image 100 captured by the imaging unit 12A of autonomous driving device 1A, the in-image detection information and actual space detection information related to the target included in image 100, and the actual space detection information related to autonomous driving device 1A from circuit 21A.

[0222] In step SP57, circuit 31A controls the driving of autonomous driving device 1A through the driving unit 17A based on the in-image detection information and actual space detection information related to the target decoded from bitstream BSB, and the in-image detection information and actual space detection information related to the target obtained from circuit 21A.

[0223] Specifically, circuit 31A generates a map 300 representing the position information and movement information of the targets existing around autonomous driving device 1A. At this time, circuit 31A converts the actual space detection information related to the target using the coordinate system based on autonomous driving device 1B decoded from bitstream BSB into the actual space detection information related to the target using the coordinate system based on autonomous driving device 1A. For example, circuit 31A uses the absolute position information and absolute movement information of autonomous driving device 1B to convert the relative position information and relative movement information of pedestrian 6C with respect to autonomous driving device 1B, thereby calculating the absolute position information and absolute movement information of pedestrian 6C. Circuit 31A includes the absolute position information and absolute movement information of the target corresponding to pedestrian 6C in map 300. Further, circuit 31A may also use the absolute position information and absolute movement information of autonomous driving device 1A to convert the absolute position information and absolute movement information of pedestrian 6C, thereby calculating the relative position information and relative speed information of pedestrian 6C with respect to autonomous driving device 1A. In this case, circuit 31A may also include the relative position information and relative movement information of pedestrian 6C with respect to autonomous driving device 1A in map 300.

[0224] Figure 25FIG. is a diagram schematically showing an example of a map 300. The map 300 has a coordinate plane of a Cartesian coordinate system with the absolute position of the autonomous driving device 1A as the origin O and the moving direction of the autonomous driving device 1A as the X-axis. In addition to the absolute position information and absolute movement information of the targets corresponding to the autonomous driving device 1B and the pedestrians 6A and 6B, the map 300 also includes the absolute position information and absolute movement information of the target corresponding to the pedestrian 6C.

[0225] In addition, on the map 300, a boundary Z is set around the autonomous driving device 1A. The boundary Z corresponds to a circle with a radius of, for example, 1 m centered on the position of the autonomous driving device 1A in the actual space. When a certain target enters within the boundary Z, the circuit 31 determines that there is a high possibility of collision between the autonomous driving device 1A and the target, and performs collision avoidance control such as causing the autonomous driving device 1A to stop urgently by driving the driving unit 17A.

[0226] In addition, Figure 25 In the example shown, the Cartesian coordinate system is described, but the same processing can also be performed for the circular coordinate system. In addition, the circuit 31A may generate a table listing the position information and movement information of the targets existing around the autonomous driving device 1A instead of generating the map 300.

[0227] According to this modification example, by mutually communicating the in-image detection information and actual space detection information related to the target between the plurality of autonomous driving devices 1A and 1B, the autonomous driving device 1A can effectively utilize these information received from the other autonomous driving device 1B for the operation control of its own device, thereby improving the safety of the autonomous driving device 1 such as the collision avoidance performance.

[0228] Industrial Applicability

[0229] The present disclosure is particularly useful for applications such as a product delivery system or a traffic system using an autonomous driving device.

Claims

1. A decoding device, comprising: a circuit; and a memory connected to the circuit, wherein the circuit performs the following processing: receiving a bitstream from an autonomous driving device having an imaging unit, decoding an image captured by the imaging unit according to the bitstream, decoding detection information within the image related to a target included in the image, i.e., in-image detection information, according to the bitstream, decoding detection information in the actual space related to the target, i.e., actual space detection information, according to the bitstream.

2. The decoding device according to claim 1, wherein the in-image detection information related to the target includes position information of the target within the image.

3. The decoding device according to claim 1, wherein the actual space detection information related to the target includes at least one of position information of the target in the actual space and movement information of the target in the actual space.

4. The decoding device according to claim 3, wherein the position information of the target includes coordinate values in a circular coordinate system based on the autonomous driving device, coordinate values in a Cartesian coordinate system based on the autonomous driving device, or coordinate values in a geographic coordinate system.

5. The decoding device according to claim 3, wherein the movement information of the target includes at least one of the absolute speed of the target or the relative speed of the target with respect to the autonomous driving device, and the absolute movement direction of the target or the relative movement direction of the target with respect to the autonomous driving device.

6. The decoding device according to claim 1, wherein the circuit further decodes actual space detection information related to the autonomous driving device according to the bitstream, and the actual space detection information related to the autonomous driving device includes position information of the autonomous driving device in the actual space and movement information of the autonomous driving device in the actual space.

7. The decoding device according to claim 6, wherein the position information of the autonomous driving device includes coordinate values in a geographic coordinate system.

8. The decoding device according to claim 6, wherein the movement information of the autonomous driving device includes at least one of the absolute speed of the autonomous driving device and the absolute movement direction.

9. The decoding device according to claim 6, wherein the circuit decodes the in-image detection information and the actual space detection information related to the target, and the actual space detection information related to the autonomous driving device according to a supplementary enhancement information (SEI) area of the bitstream.

10. The decoding device according to claim 1, wherein the circuit further generates a display image for human vision, and the display image includes: the target; and at least one of display position information of the target in the actual space and display movement information of the target in the actual space.

11. The decoding device according to claim 10, wherein the autonomous driving device includes a plurality of autonomous driving devices, The circuit generates the display image based on a plurality of images decoded from a plurality of bitstreams received from the plurality of autonomous driving devices.

12. The decoding device according to claim 10, wherein the circuit uses the position information of the target in the actual space, which is included in the actual space detection information related to the target decoded from the bitstream, as the display position information of the target. the circuit uses the movement information of the target in the actual space, which is included in the actual space detection information related to the target decoded from the bitstream, as the display movement information of the target.

13. The decoding device according to claim 10, wherein the circuit further performs the following processing: decodes the position information of the autonomous driving device in the actual space and the movement information of the autonomous driving device in the actual space according to the bitstream; uses the position information of the autonomous driving device in the actual space decoded from the bitstream to convert the position information of the target in the actual space, which is included in the actual space detection information related to the target decoded from the bitstream, thereby calculating the display position information of the target; uses the movement information of the autonomous driving device in the actual space decoded from the bitstream to convert the movement information of the target in the actual space, which is included in the actual space detection information related to the target decoded from the bitstream, thereby calculating the display movement information of the target.

14. The decoding device according to claim 10, wherein the circuit calculates the display movement information of the target based on a plurality of position information of the target in the actual space at a plurality of times, which is included in the actual space detection information related to the target decoded from the bitstream.

15. The decoding device according to claim 1, wherein the decoding device is mounted on another autonomous driving device different from the autonomous driving device, the circuit further controls the operation of the other autonomous driving device based on the in-image detection information and the actual space detection information related to the target decoded from the bitstream received from the autonomous driving device, and the in-image detection information and the actual space detection information related to the target included in the image captured by the imaging unit of the other autonomous driving device.

16. The decoding device according to claim 15, wherein the circuit converts the actual space detection information related to the target, which is decoded from the bitstream received from the autonomous driving device and uses the coordinate system based on the autonomous driving device, into the actual space detection information related to the target using the coordinate system based on the other autonomous driving device.

17. An encoding device mounted on an autonomous driving device having an imaging unit, comprising: a circuit; and a memory connected to the circuit The circuit performs the following processing: encoding the image captured by the capturing unit into a bitstream, encoding the in-image detection information, i.e., the detection information within the image related to the target included in the image, into the bitstream, encoding the detection information in the actual space related to the target, i.e., the actual space detection information, into the bitstream.

18. The encoding apparatus according to claim 17, wherein the in-image detection information related to the target includes the position information of the target within the image.

19. The encoding apparatus according to claim 17, wherein the actual space detection information related to the target includes at least one of the position information of the target in the actual space and the movement information of the target in the actual space.

20. The encoding apparatus according to claim 19, wherein the position information of the target includes coordinate values in a circular coordinate system based on the autonomous driving device, coordinate values in a Cartesian coordinate system based on the autonomous driving device, or coordinate values in a geographic coordinate system.

21. The encoding apparatus according to claim 19, wherein the movement information of the target includes at least one of the absolute speed of the target or the relative speed of the target with respect to the autonomous driving device, and the absolute movement direction of the target or the relative movement direction of the target with respect to the autonomous driving device.

22. The encoding apparatus according to claim 17, wherein the circuit further encodes the actual space detection information related to the autonomous driving device into the bitstream, the actual space detection information related to the autonomous driving device includes the position information of the autonomous driving device in the actual space and the movement information of the autonomous driving device in the actual space.

23. The encoding apparatus according to claim 22, wherein the position information of the autonomous driving device includes coordinate values in a geographic coordinate system.

24. The encoding apparatus according to claim 22, wherein the movement information of the autonomous driving device includes at least one of the absolute speed of the autonomous driving device and the absolute movement direction.

25. The encoding apparatus according to claim 22, wherein the circuit encodes the in-image detection information and the actual space detection information related to the target, and the actual space detection information related to the autonomous driving device, into the supplementary enhancement information (SEI) area of the bitstream.

26. The encoding apparatus according to claim 17, wherein the autonomous driving device further has a detection unit that detects the target in the actual space, and the circuit derives the actual space detection information related to the target based on the value obtained by converting the detection value of the detection unit from the coordinate system of the detection unit to the coordinate system of the autonomous driving device.

27. A decoding method, wherein the decoding apparatus performs the following processing: receiving a bitstream from an autonomous driving device having a capturing unit, decoding the image captured by the capturing unit according to the bitstream, Decode the in-image detection information in the image related to the target included in the image, i.e., the in-image detection information, according to the bitstream. Decode the detection information in the actual space related to the target, i.e., the actual space detection information, according to the bitstream.

28. An encoding method, which is executed by an encoding device mounted on an autonomous driving device having an imaging unit, performs the following processing: Encode the image captured by the imaging unit into a bitstream. Encode the in-image detection information in the image related to the target included in the image, i.e., the in-image detection information, into the bitstream. Encode the detection information in the actual space related to the target, i.e., the actual space detection information, into the bitstream.

Images