Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device comprises: a circuit; and a memory connected to the circuit. The circuit: receives a bit stream from an autonomous travelling device which includes an imaging unit; decodes, from the bit stream, an image captured by the imaging unit; decodes, from the bit stream, intra-image detection information, which is detection information contained in the image and which relates to an object included in the image; and decodes, from the bit stream, real space detection information, which is detection information contained in real space and which relates to the object.

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method.

[0002] Patent Literature 1 discloses an autonomous vehicle equipped with an object detection unit and an autonomous module. The object detection unit measures the detection distance using a camera or the like. The autonomous module determines a control range limited within the detection distance and reflects the learned driving tendencies of the user and driving tendencies defined by external data in the vehicle's driving control-related data.

[0003] US Patent Application Publication No. 2021 / 0278840

[0004] The present disclosure aims to effectively utilize detection information regarding objects present around an autonomous driving device, thereby improving the safety of the autonomous driving device, such as its collision avoidance performance.

[0005] A decoding device according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit receives a bitstream from an autonomous driving device having an image capturing unit, decodes an image captured by the image capturing unit from the bitstream, decodes intra-image detection information from the bitstream, which is detection information within the image regarding an object included in the image, and decodes real-space detection information from the bitstream, which is detection information in real space regarding the object.

[0006] 14 is a diagram illustrating, in a simplified form, the configuration of an information processing system according to a first embodiment of the present disclosure. FIG. 15 is a flowchart illustrating the flow of processing performed by a circuit of an encoding device. FIG. 16 is a diagram illustrating, in a simplified form, an example of a traveling situation of an autonomous mobile device. FIG. 17 is a diagram illustrating, in a simplified form, an example of an image captured by a capturing unit. FIG. 18 is a diagram illustrating, in a simplified form, an example of an image analysis result. FIG. 19 is a diagram illustrating, in a simplified form, an example of real-space detection information and in-image detection information related to an object. FIG. 19 is a diagram illustrating, in a simplified form, an example of a coordinate system of an autonomous mobile device. FIG. 19 is a diagram illustrating, in a simplified form, an example of a bitstream data structure. FIG. 19 is a diagram illustrating, in a simplified form, an example of a syntax. FIG. 19 is a flowchart illustrating the flow of processing performed by a circuit of a decoding device. FIG. 19 is a diagram illustrating, in a simplified form, an example of a display image. FIG. 19 is a diagram illustrating, in a simplified form, an example of a display image. FIG. 19 is a diagram illustrating, in a simplified form, a part of a figure extracted from the display image shown in FIG. 13. FIG. 19 is a diagram illustrating, in a simplified form, an example of a display image. FIG. 19 is a diagram illustrating, in a simplified form, the configuration of an information processing system according to a modified example. FIG. 19 is a flowchart illustrating the flow of processing performed by a circuit of an encoding device. FIG. 19 is a diagram illustrating, in a simplified form, an example of a syntax. FIG. 1 is a diagram showing, in a simplified form, an example of a display image. FIG. 2 is a diagram showing, in a simplified form, an example of a display image. FIG. 3 is a diagram showing, in a simplified form, a configuration of an information processing system according to a second embodiment of the present disclosure. FIG. 4 is a flowchart showing a flow of processing executed by a circuit of a decoding device. FIG. 5 is a diagram showing, in a simplified form, an example of an image captured by a capturing unit. FIG. 6 is a diagram showing, in a simplified form, an example of a map.

[0007] (Findings underlying the present disclosure) In a product delivery system using autonomous vehicles or other autonomous devices, unmanned autonomous vehicles travel along sidewalks at a speed comparable to that of pedestrians to deliver products. The autonomous vehicles are equipped with cameras, GPS receivers, obstacle sensors, and the like, which complement each other's functions to ensure safe driving. However, there are technical limitations to ensuring safe driving even in highly complex environments, and if a problem occurs, remote intervention by a human operator is required. Therefore, the autonomous vehicles transmit images captured by their onboard cameras in real time to a monitoring center where an operator is always present.

[0008] However, when one operator is monitoring multiple autonomous mobile devices at the same time, it is difficult for the operator to instantly and accurately grasp the situation around the autonomous mobile device even if they visually view the images received from the autonomous mobile device.

[0009] In order to solve such problems, the inventors have come up with the idea of ​​the present disclosure, based on the knowledge that the above problems can be solved by transmitting not only images captured by a camera but also detection information in real space detected by sensors mounted on the autonomous mobile device to a monitoring center and making effective use of the information.

[0010] Next, each aspect of the present disclosure will be described.

[0011] A decoding device according to a first aspect of the present disclosure includes a circuit and a memory connected to the circuit, wherein the circuit receives a bitstream from an autonomous driving device having an image capturing unit, decodes an image captured by the image capturing unit from the bitstream, decodes intra-image detection information from the bitstream, which is detection information within the image relating to an object included in the image, and decodes real-space detection information from the bitstream, which is detection information in real space relating to the object.

[0012] According to the first aspect, the decoding device can effectively utilize on-screen detection information and real-space detection information regarding objects present around the autonomous driving device, thereby improving the safety of the autonomous driving device, such as collision avoidance performance, and also improving decision-making regarding the operation of the autonomous driving device to improve the safety of the autonomous driving device in complex environments.

[0013] In the decoding device according to the second aspect of the present disclosure, in the first aspect, the intra-image detection information regarding the object may include position information of the object within the image.

[0014] According to the second aspect, the position information of the object within the image can be effectively utilized on the decoding device side.

[0015] In the decoding device according to the third aspect of the present disclosure, in the first or second aspect, the real space detection information regarding the object may include at least one of position information of the object in the real space and movement information of the object in the real space.

[0016] According to the third aspect, the position information and movement information of the object in the real space can be effectively utilized on the decoding device side.

[0017] In the decoding device according to the fourth aspect of the present disclosure, in the third aspect, the location information of the object may include coordinate values ​​in a circular coordinate system based on the autonomous mobile device, coordinate values ​​in a Cartesian coordinate system based on the autonomous mobile device, or coordinate values ​​in a geographic coordinate system.

[0018] According to the fourth aspect, the position of the object can be accurately identified by the position information including coordinate values.

[0019] In the decoding device of the fifth aspect of the present disclosure, in the third or fourth aspect, the movement information of the object may include at least one of the absolute speed of the object or the relative speed of the object with respect to the autonomous mobile device, and the absolute movement direction of the object or the relative movement direction of the object with respect to the autonomous mobile device.

[0020] According to the fifth aspect, at least one of the speed and the moving direction of the object can be accurately identified by movement information including at least one of the absolute speed or the relative speed and the absolute moving direction or the relative moving direction.

[0021] In a decoding device according to a sixth aspect of the present disclosure, in any one of the first to fifth aspects, the circuit may further decode real-space detection information related to the autonomous driving device from the bit stream, and the real-space detection information related to the autonomous driving device may include position information of the autonomous driving device in the real space and movement information of the autonomous driving device in the real space.

[0022] According to the sixth aspect, the position information and movement information of the autonomous mobile device in real space can be effectively utilized on the decoding device side.

[0023] A decoding device according to a seventh aspect of the present disclosure is the sixth aspect, wherein the location information of the autonomously moving device includes coordinate values ​​in a geographic coordinate system.

[0024] According to the seventh aspect, the absolute position of the autonomously moving device can be accurately identified using position information that includes coordinate values ​​in a geographic coordinate system.

[0025] A decoding device according to an eighth aspect of the present disclosure is the sixth or seventh aspect, wherein the movement information of the autonomous mobile device may include at least one of an absolute speed and an absolute movement direction of the autonomous mobile device.

[0026] According to the eighth aspect, at least one of the absolute speed and the absolute movement direction of the autonomous mobile device can be accurately identified using the movement information.

[0027] In a decoding device according to a ninth aspect of the present disclosure, in any one of the sixth to eighth aspects, the circuit may decode the in-image detection information and the real-space detection information regarding the object, and the real-space detection information regarding the autonomous driving device from an SEI region of the bitstream.

[0028] According to the ninth aspect, the in-image detection information and real-space detection information relating to the object and the real-space detection information relating to the autonomous driving device are encoded in the SEI region of the bitstream, which allows for easy processing on the decoding device side.

[0029] In a decoding device according to a tenth aspect of the present disclosure, in any one of the first to ninth aspects, the circuit may further generate a display image for human vision, and the display image may include the object and at least one of display position information of the object in the real space and display movement information of the object in the real space.

[0030] According to the tenth aspect, by visually viewing a display image including display position information and display movement information of an object in real space, the operator can instantly and accurately grasp the situation around the autonomous mobile device.

[0031] A decoding device according to an eleventh aspect of the present disclosure is, in the tenth aspect, the autonomous driving device may include a plurality of autonomous driving devices, and the circuit may generate the display image based on a plurality of images decoded from a plurality of bitstreams received from the plurality of autonomous driving devices.

[0032] According to the eleventh aspect, multiple images captured by multiple autonomous mobile devices can complement each other's blind spots during image capture, thereby improving the convenience of the displayed image.

[0033] In the decoding device according to the twelfth aspect of the present disclosure, in the tenth or eleventh aspect, the circuit may use position information of the object in the real space contained in the real space detection information about the object decoded from the bitstream as the display position information of the object, and may use movement information of the object in the real space contained in the real space detection information about the object decoded from the bitstream as the display movement information of the object.

[0034] According to the twelfth aspect, the display position information and the display movement information can be generated easily.

[0035] In a decoding device according to a thirteenth aspect of the present disclosure, in the tenth or eleventh aspect, the circuit may further decode position information of the autonomous mobile device in the real space and movement information of the autonomous mobile device in the real space from the bitstream, calculate the display position information of the object by converting the position information of the object in the real space included in the real space detection information about the object decoded from the bitstream using the position information of the autonomous mobile device in the real space decoded from the bitstream, and calculate the display movement information of the object by converting the movement information of the object in the real space included in the real space detection information about the object decoded from the bitstream using the movement information of the autonomous mobile device in the real space decoded from the bitstream.

[0036] According to the thirteenth aspect, the relative position of the object can be converted into an absolute position for display, and the relative speed and relative movement direction of the object can be converted into an absolute speed and absolute movement direction for display.

[0037] In a decoding device according to a fourteenth aspect of the present disclosure, in the tenth or eleventh aspect, the circuit may calculate the display movement information of the object based on multiple pieces of position information of the object in the real space at multiple times, which are included in the real space detection information regarding the object decoded from the bitstream.

[0038] According to the fourteenth aspect, even if the bit stream received from the autonomous mobile device does not include movement information of the object in real space, the display movement information of the object can be appropriately calculated based on multiple position information of the object in real space at multiple times.

[0039] A decoding device according to a fifteenth aspect of the present disclosure, in any one of the first to fourteenth aspects, may be mounted on an autonomous driving device other than the autonomous driving device, and the circuit may further control the operation of the other autonomous driving device based on the in-image detection information and real-space detection information regarding the object decoded from the bitstream received from the autonomous driving device, and the in-image detection information and real-space detection information regarding an object included in an image captured by a capturing unit possessed by the other autonomous driving device.

[0040] According to the fifteenth aspect, by mutually communicating in-image detection information and real-space detection information regarding objects between multiple autonomous driving devices, each autonomous driving device can effectively utilize this information received from other autonomous driving devices to control its own operation, thereby improving the safety of the autonomous driving devices, such as their collision avoidance performance.

[0041] A decoding device according to a sixteenth aspect of the present disclosure is, in the fifteenth aspect, preferably wherein the circuit converts the real-space detection information about the object, decoded from the bitstream received from the autonomous mobile device and using a coordinate system based on the autonomous mobile device, into real-space detection information about the object using a coordinate system based on the other autonomous mobile device.

[0042] According to the 16th aspect, by performing a coordinate system conversion process, each autonomous driving device can use real-space detection information regarding an object detected by another autonomous driving device as real-space detection information regarding an object detected by its own autonomous driving device.

[0043] A coding device according to a seventeenth aspect of the present disclosure is an coding device mounted on an autonomous mobile device having an imaging unit, and includes a circuit and a memory connected to the circuit, wherein the circuit encodes an image captured by the imaging unit into a bitstream, encodes intra-image detection information, which is detection information within the image regarding an object included in the image, into the bitstream, and encodes real-space detection information, which is detection information in real space regarding the object, into the bitstream.

[0044] According to the seventeenth aspect, by encoding in-screen detection information and real-space detection information regarding objects present around the autonomous driving device into a bitstream, this information can be effectively utilized on the decoding device side that receives the bitstream, thereby improving the safety of the autonomous driving device, such as collision avoidance performance, and also improving decision-making regarding the operation of the autonomous driving device to improve the safety of the autonomous driving device in complex environments.

[0045] An encoding device according to an eighteenth aspect of the present disclosure is the seventeenth aspect, wherein the intra-image detection information regarding the object preferably includes position information of the object within the image.

[0046] According to the eighteenth aspect, the position information of an object within an image can be effectively utilized on the decoding device side.

[0047] In the encoding device according to the 19th aspect of the present disclosure, in the 17th or 18th aspect, the real-space detection information regarding the object may include at least one of position information of the object in the real space and movement information of the object in the real space.

[0048] According to the nineteenth aspect, the position information and movement information of the object in the real space can be effectively utilized on the decoding device side.

[0049] In the encoding device according to the 20th aspect of the present disclosure, in the 19th aspect, the position information of the object may include coordinate values ​​in a circular coordinate system based on the autonomous mobile device, coordinate values ​​in a Cartesian coordinate system based on the autonomous mobile device, or coordinate values ​​in a geographic coordinate system.

[0050] According to the twentieth aspect, the position of the object can be accurately identified by the position information including coordinate values.

[0051] In the encoding device of the 21st aspect of the present disclosure, in the 19th or 20th aspect, the movement information of the object may include at least one of the absolute speed of the object or the relative speed of the object with respect to the autonomous mobile device, and the absolute movement direction of the object or the relative movement direction of the object with respect to the autonomous mobile device.

[0052] According to the twenty-first aspect, at least one of the speed and the moving direction of the object can be accurately identified by movement information including at least one of the absolute speed or the relative speed and the absolute moving direction or the relative moving direction.

[0053] In the encoding device according to the 22nd aspect of the present disclosure, in any one of the 17th to 21st aspects, the circuit may further encode real-space detection information related to the autonomous mobile device into the bit stream, and the real-space detection information related to the autonomous mobile device may include position information of the autonomous mobile device in the real space and movement information of the autonomous mobile device in the real space.

[0054] According to the 22nd aspect, the position information and movement information of the autonomous mobile device in real space can be effectively utilized on the decoding device side.

[0055] In the encoding device according to a 23rd aspect of the present disclosure, in the 22nd aspect, the location information of the autonomously moving device may include coordinate values ​​in a geographic coordinate system.

[0056] According to the twenty-third aspect, the absolute position of the autonomously moving device can be accurately identified using position information including coordinate values ​​in a geographic coordinate system.

[0057] In the encoding device according to a 24th aspect of the present disclosure, in the 22nd or 23rd aspect, the movement information of the autonomous mobile device may include at least one of an absolute speed and an absolute movement direction of the autonomous mobile device.

[0058] According to the twenty-fourth aspect, at least one of the absolute speed and the absolute movement direction of the autonomous mobile device can be accurately identified from the movement information.

[0059] In the encoding device according to the 25th aspect of the present disclosure, in any one of the 17th to 24th aspects, the circuit may encode the intra-image detection information and the real-space detection information relating to the object into an SEI region of the bitstream.

[0060] According to the twenty-fifth aspect, by encoding the intra-image detection information and real-space detection information relating to the object into the SEI region of the bitstream, processing can be easily performed on the decoding device side.

[0061] In the encoding device according to the 26th aspect of the present disclosure, in any one of the 17th to 25th aspects, the autonomous mobile device may further have a detection unit that detects the object in the real space, and the circuit may derive the real-space detection information regarding the object based on a value obtained by converting the detection value by the detection unit from the coordinate system of the detection unit to the coordinate system of the autonomous mobile device.

[0062] According to the twenty-sixth aspect, real-space detection information relating to the object can be appropriately derived by a conversion process from the unique coordinate system of the detection unit to a coordinate system common to the autonomous mobile device.

[0063] A decoding method according to a 27th aspect of the present disclosure includes a decoding device receiving a bitstream from an autonomous driving device having an image capturing unit, decoding an image captured by the image capturing unit from the bitstream, decoding intra-image detection information from the bitstream which is detection information within the image regarding an object included in the image, and decoding real-space detection information from the bitstream which is detection information in real space regarding the object.

[0064] According to the 27th aspect, the decoding device can effectively utilize on-screen detection information and real-space detection information regarding objects present around the autonomous driving device, thereby improving the safety of the autonomous driving device, such as collision avoidance performance, and also improving decision-making regarding the operation of the autonomous driving device to improve the safety of the autonomous driving device in complex environments.

[0065] In an encoding method according to a 28th aspect of the present disclosure, an encoding device mounted on an autonomous mobile device having an imaging unit encodes an image captured by the imaging unit into a bitstream, encodes intra-image detection information, which is detection information within the image regarding an object included in the image, into the bitstream, and encodes real-space detection information, which is detection information in real space regarding the object, into the bitstream.

[0066] According to the 28th aspect, by encoding in-screen detection information and real-space detection information regarding objects present around the autonomous driving device into a bitstream, this information can be effectively utilized on the decoding device side that receives the bitstream, thereby improving the safety of the autonomous driving device, such as collision avoidance performance, and also improving decision-making regarding the operation of the autonomous driving device to improve the safety of the autonomous driving device in complex environments.

[0067] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0068] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0069] 1 is a diagram showing a simplified configuration of an information processing system according to a first embodiment of the present disclosure. The information processing system includes an autonomous mobile device 1, a transmission line NW, a decoding device 2, and a display device 3.

[0070] The autonomous mobile device 1 is, for example, an autonomous mobile vehicle that delivers goods unmanned in a product delivery system. The autonomous mobile device 1 has an encoding device 11, an imaging unit 12, and an object detection unit 13. Although one autonomous mobile device 1 is shown in Fig. 1, multiple autonomous mobile devices 1 with the same configuration may exist.

[0071] The encoding device 11 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 is configured to include a processor such as a CPU. The memory 22 is configured to include any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 22 stores data to be processed by the circuit 21 or data in the middle of processing.

[0072] The image capturing unit 12 is configured with a camera having an optical system and a CMOS image sensor, etc. The image capturing unit 12 captures images of the surroundings of the autonomous mobile device 1 (particularly, the area ahead in the direction of movement).

[0073] The object detection unit 27 is configured with LiDAR (Light Detection And Ranging) and the like. The object detection unit 27 detects objects that exist around the autonomous mobile device 1 in real space. The object detection unit 27 inputs real-space detection information, which is detection information related to the object in real space, to the circuit 21. The real-space detection information includes the distance to the object, the direction in which the object is located, the shape of the object, and the like.

[0074] The transmission path NW is the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. The transmission path NW is preferably a private network or the like in which secure communication is ensured by access restrictions.

[0075] The decoding device 2 and the display device 3 are arranged, for example, in a monitoring center where an operator who remotely monitors the autonomous mobile device 1 resides.

[0076] The decoding device 2 includes a circuit 31 and a memory 32 connected to the circuit 31. The circuit 31 is configured to include a processor such as a CPU. The memory 32 is configured to include any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 32 stores data to be processed by the circuit 31 or data in the middle of processing.

[0077] The display device 3 is a liquid crystal display, an organic EL display, or the like.

[0078] The circuit 21 of the encoding device 11 transmits the bit stream BS to the decoding device 2 via the transmission path NW. The circuit 31 of the decoding device 2 receives the bit stream BS and generates a display image for human vision by an operator. The display device 3 displays the display image.

[0079] FIG. 2 is a flowchart showing the flow of processing executed by the circuit 21 of the encoding device 11.

[0080] In step SP11, the circuit 21 acquires an image captured by the image capturing unit 12. The image includes objects present around the autonomous mobile device 1.

[0081] FIG. 3 is a diagram schematically illustrating an example of a traveling situation of the autonomous mobile device 1. Autonomous mobile devices 1A and 1B and pedestrians 6A to 6C are present on a sidewalk 5. The autonomous mobile device 1A is traveling upward on the page. The autonomous mobile device 1B is traveling downward on the page, ahead of the autonomous mobile device 1A in the direction of movement. The pedestrian 6A is walking to the right on the page, ahead of the autonomous mobile device 1A in the direction of movement. The pedestrian 6B is walking downward on the page, ahead of the autonomous mobile device 1A in the direction of movement. The pedestrian 6C is walking upward on the page, behind the autonomous mobile device 1A in the direction of movement.

[0082] 4 is a diagram schematically illustrating an example of an image 100 captured by the image capturing unit 12 of the autonomous mobile device 1A. The image 100 includes the autonomous mobile device 1B and a pedestrian 6B moving toward the autonomous mobile device (autonomous mobile device 1A), and a pedestrian 6A crossing in front of the autonomous mobile device.

[0083] In step SP12, circuit 21 acquires real-space detection information related to objects included in image 100 from object detection unit 13. The real-space detection information related to the objects includes, for each of the objects, autonomous mobile device 1B and pedestrians 6A and 6B, the distance to the object, the direction in which the object is located, the shape of the object, etc. Step SP12 may be executed simultaneously with step SP11.

[0084] In step SP13, the circuit 21 analyzes the image 100 acquired in step SP11. The analysis of the image 100 includes, for example, object detection processing and object recognition processing using a machine-learned estimation model. The circuit 21 generates in-image detection information, which is detection information within the image 100 regarding objects included in the image 100. The in-image detection information includes position information of the objects within the image 100 as a result of the object detection processing. The in-image detection information may also include attribute information of the objects within the image 100 as a result of the object recognition processing. The attributes represent the type of object, such as an autonomous mobile device, a person, or a bicycle.

[0085] 5 is a diagram schematically illustrating an example of the analysis results of the image 100 by the circuit 21. The circuit 21 sets a bounding box BB surrounding the object within the image 100 as position information of the object within the image 100. Specifically, the circuit 21 sets a bounding box BB1 surrounding the autonomous mobile device 1B, a bounding box BB2 surrounding the pedestrian 6A, and a bounding box BB3 surrounding the pedestrian 6B.

[0086] The circuit 21 specifies the position and shape of the bounding box BB using the coordinate values ​​of a particular vertex of the bounding box BB (for example, the vertex of the upper left corner) and the height and width of the bounding box BB. Alternatively, the circuit 21 may specify the position and shape of the bounding box BB using the coordinate values ​​of the center point of the bounding box BB and the height and width of the bounding box BB. Alternatively, the circuit 21 may specify the position and shape of the bounding box BB using the coordinate values ​​of four vertices of the bounding box BB. Alternatively, the circuit 21 may specify the position and shape of the bounding box BB using the coordinate values ​​of two vertices located diagonally opposite the bounding box BB.

[0087] The circuit 21 also associates the objects in the real space detected by the object detection unit 13 with the objects in the image 100 detected by image analysis.

[0088] 6 is a simplified diagram showing an example of real-space detection information and intra-image detection information related to an object. The real-space detection information includes position information and movement information. The position information includes distance and direction. The movement information includes speed and movement direction. The circuit 21 calculates the speed and movement direction of the object based on the position detection results of the object detection unit 13 at multiple times. However, the calculation of the movement information including the speed and movement direction of the object may be performed by the circuit 31 of the decoding device 2 instead of the circuit 21 of the encoding device 11. The intra-image detection information includes information indicating the position and shape of the corresponding bounding box BB.

[0089] The origin of the coordinate system of object detection unit 13 is the center point of object detection unit 13, and the origin of the coordinate system of autonomous mobile device 1A is the center point of autonomous mobile device 1A, and these two are different. Therefore, circuit 21 derives real-space detection information related to the object based on values ​​obtained by converting the detection values ​​by object detection unit 13 from the unique coordinate system of object detection unit 13 to the common coordinate system of autonomous mobile device 1A.

[0090] 7 is a diagram showing an example of a coordinate system of the autonomous mobile device 1A. In the example shown in FIG. 7, the coordinate system of the autonomous mobile device 1A is a circular coordinate system, and the origin O of the coordinate system is the center point of the autonomous mobile device 1A. An angle of 0° in the coordinate system corresponds to the movement direction of the autonomous mobile device 1A. The position of an object is indicated by distance coordinate values ​​and angle coordinate values ​​in the circular coordinate system. Furthermore, the movement direction of the object is indicated by angle coordinate values ​​in the circular coordinate system.

[0091] In the example shown in FIG. 6, the distance, direction, speed, and movement direction are the relative distance, relative direction, relative speed, and relative movement direction of the object with respect to the autonomous mobile device 1A.

[0092] For the object autonomous mobile device 1B, the corresponding bounding box BB is bounding box BB1, the relative distance is 5.0 m, the relative direction is 30.0°, the relative speed is 8.5 km / h, and the relative movement direction is 180.0°. For the object pedestrian 6A, the corresponding bounding box BB is bounding box BB2, the relative distance is 9.8 m, the relative direction is −28.5°, the relative speed is 7.7 km / h, and the relative movement direction is −90.0°. For the object pedestrian 6B, the corresponding bounding box BB is bounding box BB3, the relative distance is 13.1 m, the relative direction is 10.0°, the relative speed is 12.5 km / h, and the relative movement direction is 180.0°.

[0093] The coordinate system of the autonomous mobile device 1A is not limited to the circular coordinate system described above, and may be a Cartesian coordinate system. The Cartesian coordinate system may, for example, indicate the front-to-rear direction of the autonomous mobile device 1A as the X axis, the left-to-right direction of the autonomous mobile device 1A as the Y axis, and the center point of the autonomous mobile device 1A as the origin, and the position of an object may be indicated by values ​​on the X axis and the Y axis.

[0094] In step SP14, the circuit 21 encodes the image 100 acquired in step SP11 into a bit stream BS.

[0095] 8 is a simplified diagram showing the data structure of a bitstream BS. The bitstream BS has a header 41 and a payload 42. The header 41 includes an SEI (Supplemental Enhancement Information) area 43 for storing additional information.

[0096] The circuit 21 encodes the image 100 containing the object into the payload 42 of the bitstream BS.

[0097] In step SP15, the circuit 21 encodes the intra-image detection information related to the object into a bitstream BS. The circuit 21 encodes the intra-image detection information generated in step SP13 into a predetermined location in the header 41. The predetermined location is the SEI region 43. The predetermined location may be the VPS, SPS, PPS, PH, SH, APS, tile header, or the like. Alternatively, the predetermined location may be an annotated region SEI (ARSEI) region for storing bounding box information.

[0098] In step SP16, the circuit 21 encodes the real space detection information related to the object into a bitstream BS. The circuit 21 encodes the real space detection information acquired in step SP12 into a predetermined location in the header 41. The predetermined location is the SEI area 43. The predetermined location may be the VPS, SPS, PPS, PH, SH, APS, or tile header, or the like. Alternatively, the predetermined location may be the ARSEI area for storing bounding box information. Note that steps SP14 to SP16 may be executed simultaneously.

[0099] 9 is a simplified diagram showing an example of syntax. num_objects indicates the number of objects included in the image 100. first_information includes in-image detection information related to the object. second_information includes real-space detection information related to the object.

[0100] In step SP17, the circuit 21 transmits the bit stream BS to the decoding device 2.

[0101] FIG. 10 is a flowchart showing the flow of processing executed by the circuit 31 of the decoding device 2.

[0102] In step SP21, the circuit 31 receives the bit stream BS from the autonomous mobile device 1A.

[0103] In step SP22, the circuit 31 decodes the image 100 including the object from the payload 42 of the bitstream BS.

[0104] In step SP23, the circuit 31 decodes intra-image detection information relating to the object from the header 41 of the bitstream BS. If the intra-image detection information is encoded in the SEI region 43 of the header 41, the circuit 31 decodes the intra-image detection information from the SEI region 43. If the intra-image detection information is encoded in the ARSEI region of the header 41, the circuit 31 decodes the intra-image detection information from the ARSEI region.

[0105] In step SP24, the circuit 31 decodes real space detection information related to the object from the header 41 of the bitstream BS. If the real space detection information is encoded in the SEI area 43 of the header 41, the circuit 31 decodes the real space detection information from the SEI area 43. If the real space detection information is encoded in the ARSEI area of ​​the header 41, the circuit 31 decodes the real space detection information from the ARSEI area. Note that steps SP22 to SP24 may be executed simultaneously.

[0106] In step SP25, the circuit 31 generates a display image 200 for human vision by the operator based on the images and information decoded in steps SP22 to SP24.

[0107] 11 is a simplified diagram showing an example of a display image 200. The display image 200 includes the image 100, display position information of the object in the real space, and display movement information D1 to D3 of the object in the real space.

[0108] The display position information includes frames F1 to F3 corresponding to bounding boxes BB1 to BB3, and distance information indicating the relative distance of each object with respect to the autonomous mobile device 1 A. The circuit 31 uses the position information of the object in real space, which is included in the real space detection information decoded from the bitstream BS, as the display position information of the object.

[0109] The display movement information D1 to D3 includes relative speed information indicating the relative speed of each object with respect to the autonomous mobile device 1A, and relative movement direction information indicating the relative movement direction of each object with respect to the autonomous mobile device 1A. In the example shown in Figure 11, the relative movement direction information is represented by a bold arrow. The circuit 31 uses the movement information of the object in real space, which is included in the real space detection information decoded from the bitstream BS, as the display movement information D1 to D3 of the object.

[0110] Furthermore, the circuit 31 highlights, by using a thick line or coloring, frames F1, F3 corresponding to objects that may collide with or approach the autonomous mobile device 1A (among the multiple objects included in the display image 200). The circuit 31 determines whether or not there is a possibility of a collision with or approaching the autonomous mobile device 1A based on the relative position, relative distance, and relative movement direction of the object relative to the autonomous mobile device. This allows an operator viewing the display image 200 to quickly identify an object that should be monitored with particular attention from among the multiple objects included in the display image 200.

[0111] According to the encoding device 11 of this embodiment, by encoding the on-screen detection information and real-space detection information relating to objects present around the autonomous mobile device 1 into a bit stream BS, this information can be effectively utilized on the decoding device 2 side that receives the bit stream BS, thereby improving the safety of the autonomous mobile device 1, such as the collision avoidance performance. Furthermore, it is possible to improve the decision-making of the operation of the autonomous mobile device 1 to improve the safety of the autonomous mobile device 1 in complex environments.

[0112] Furthermore, according to the encoding device 11 of this embodiment, the intra-image detection information includes position information of the object within the image 100, so that the position information of the object within the image 100 can be effectively utilized on the decoding device 2 side.

[0113] Furthermore, according to the encoding device 11 of this embodiment, the real space detection information includes position information and movement information of the object in real space, so that the position information and movement information of the object in real space can be effectively utilized on the decoding device 2 side. Note that either the position information or the movement information of the object in real space may be omitted.

[0114] Furthermore, according to the encoding device 11 of this embodiment, the position information of an object includes coordinate values ​​in a circular coordinate system based on the autonomous mobile device 1, and the position of the object can be accurately identified using the position information including these coordinate values.

[0115] Furthermore, according to the encoding device 11 of this embodiment, the object movement information includes the relative speed and relative movement direction of the object with respect to the autonomous mobile device 1, so that the speed and movement direction of the object can be accurately identified.

[0116] Furthermore, according to the encoding device 11 of this embodiment, intra-image detection information and real-space detection information relating to an object are encoded into the SEI region 43 of the bitstream, which allows for easy processing on the decoding device 2 side.

[0117] Furthermore, according to the encoding device 11 of this embodiment, real space detection information regarding the object can be appropriately derived by a conversion process from the unique coordinate system of the object detection unit 13 to the common coordinate system of the autonomous mobile device 1.

[0118] Furthermore, the decoding device 2 according to this embodiment can effectively utilize on-screen detection information and real-space detection information regarding objects present around the autonomous mobile device 1, thereby improving the safety of the autonomous mobile device 1, such as the collision avoidance performance. Furthermore, it can improve decision-making regarding the operation of the autonomous mobile device 1 to improve the safety of the autonomous mobile device 1 in complex environments.

[0119] Furthermore, according to the decoding device 2 of this embodiment, the intra-image detection information includes position information of the object within the image 100, so that the position information of the object within the image 100 can be effectively utilized.

[0120] Furthermore, according to the decoding device 2 of this embodiment, the real space detection information includes position information and movement information of the object in the real space, so that the position information and movement information of the object in the real space can be effectively utilized.

[0121] Furthermore, according to the decoding device 2 of this embodiment, the position information of an object includes coordinate values ​​in a circular coordinate system based on the autonomous mobile device 1, and therefore the position of the object can be accurately identified using the position information including these coordinate values.

[0122] Furthermore, with the decoding device 2 according to this embodiment, the object movement information includes the relative speed and relative movement direction of the object with respect to the autonomous mobile device 1, so that the speed and movement direction of the object can be accurately identified.

[0123] Furthermore, according to the decoding device 2 of this embodiment, intra-image detection information and real-space detection information relating to an object are coded into the SEI region 43 of the bitstream, which allows for easy processing on the decoding device 2 side.

[0124] Furthermore, according to the decoding device 2 of this embodiment, the operator can instantly and accurately grasp the situation around the autonomous mobile device 1 by visually viewing the display image 200, which includes display position information and display movement information of the object in real space.

[0125] Furthermore, according to the decoding device 2 of this embodiment, the position information of the object contained in the real space detection information is used as the display position information, and the movement information of the object contained in the real space detection information is used as the display movement information, thereby making it possible to easily generate the display position information and the display movement information.

[0126] Various modifications of the first embodiment will be described below. The modifications described below can be applied in any combination.

[0127] 12 is a simplified diagram showing an example of display image 201 as a modification of display image 200. Display image 201 includes a figure G0 that resembles autonomous mobile device 1A, which is the autonomous mobile device itself; a figure G1 that resembles autonomous mobile device 1B, which is an object; a figure G2 that resembles pedestrian 6A, which is an object; a figure G3 that resembles pedestrian 6B, which is an object; display position information for the objects in real space; and display movement information D1 to D3 for the objects in real space. Display image 201 has a coordinate plane of a circular coordinate system with the position of figure G0 as the origin O.

[0128] The display position information includes graphics G1 to G3 and distance information indicating the relative distance of each object with respect to the autonomous mobile device 1 A. For example, the distance coordinate value L1 and angle coordinate value θ1 in the circular coordinate system for graphic G1 correspond to the distance coordinate value and angle coordinate value included in the real space detection information for the autonomous mobile device 1B.

[0129] The display movement information D1 to D3 includes relative speed information indicating the relative speed of each object with respect to the autonomous mobile device 1A, and relative movement direction information indicating the relative movement direction of each object with respect to the autonomous mobile device 1A. The relative movement direction information is represented as a bold arrow graphic.

[0130] The circuit 31 may highlight the graphics G1 and G3 corresponding to objects that may collide with or approach the player's ship among the plurality of objects by flashing or coloring them.

[0131] According to this modification, an operator viewing the display image 201 can quickly identify an object that should be monitored with particular attention from among a plurality of objects.

[0132] 13 is a simplified diagram showing an example of a display image 202 as a modification of the display image 200. The display image 202 includes figures G0 to G3, display position information of the objects in real space, and display movement information D1 to D3 of the objects in real space. The display image 202 has a coordinate plane of a Cartesian coordinate system with the position of the figure G0 as the origin O. The Cartesian coordinate system has orthogonal X and Y axes.

[0133] In this modification, the coordinate system of the autonomous mobile device 1A is a Cartesian coordinate system, and the origin O of the coordinate system is the center point of the autonomous mobile device 1A. The X axis of the coordinate system corresponds to the direction of movement of the autonomous mobile device 1A. The position of an object is indicated by the X coordinate value and the Y coordinate value in the Cartesian coordinate system.

[0134] The display position information includes graphics G1 to G3 and distance information indicating the relative distance of each object with respect to the autonomous mobile device 1 A. For example, the X coordinate value X1 and the Y coordinate value Y1 in the Cartesian coordinate system for graphic G1 correspond to the X coordinate value and the Y coordinate value included in the real space detection information for the autonomous mobile device 1B.

[0135] The display movement information D1 to D3 includes relative speed information indicating the relative speed of each object with respect to the autonomous mobile device 1A, and relative movement direction information indicating the relative movement direction of each object with respect to the autonomous mobile device 1A. The relative movement direction information is represented as a bold arrow graphic.

[0136] The circuit 31 may highlight the graphics G1 and G3 corresponding to objects that may collide with or approach the player's ship among the plurality of objects by flashing or coloring them.

[0137] According to this modification, an operator viewing the display image 201 can quickly identify an object that should be monitored with particular attention from among a plurality of objects.

[0138] (Third Modification) In this modification, a description will be given of how to deal with the case where the real space detection information decoded from the bitstream BS does not include object movement information (speed and movement direction).

[0139] FIG. 14 is a diagram showing a figure G1 extracted from the display image 202 shown in FIG. 13. Circuit 31 calculates display movement information D1 for the object based on multiple pieces of position information for the object corresponding to figure G1 at multiple times, which information is included in the real-space detection information decoded from the bitstream BS. Circuit 31 calculates the movement distance and movement direction of the object in real space based on the X coordinate value X1(t) and Y coordinate value Y1(t) of the object at a certain time t, and the X coordinate value X1(t-1) and Y coordinate value Y1(t-1) of the object at an earlier time t-1. Circuit 31 also calculates the velocity of the object based on the calculated movement distance and the time difference Δt between time t and time t-1.

[0140] Equation (1) shows an example of a formula for calculating the velocity V of an object.

[0141]

[0142] Equation (2) shows an example of a formula for calculating the moving direction A of an object.

[0143]

[0144] The circuit 31 generates the display movement information D1 including the speed information indicating the calculated speed and the movement direction information indicating the calculated movement direction. Note that although the Cartesian coordinate system has been described in the example shown in Fig. 14, the same processing can also be performed for the circular coordinate system.

[0145] According to this modified example, even if the bit stream BS received from the autonomous mobile device 1 does not include movement information of the object in real space, the circuit 31 can appropriately calculate the display movement information D1 to D3 of the object based on multiple position information of the object in real space at multiple times.

[0146] 15 is a simplified diagram showing an example of a display image 201. A boundary Z is set around the figure G0 as the center. The boundary Z corresponds to a circle with a radius of, for example, 1 m, centered on the position of the autonomous mobile device 1A in real space.

[0147] When a certain object enters boundary Z, circuit 31 determines that there is a high possibility that autonomous mobile device 1A will collide with the object, and displays an alert in display image 201. For example, when autonomous mobile device 1B corresponding to figure G1 enters boundary Z, circuit 31 highlights figure G0 and / or figure G1 by blinking or coloring, etc. Note that although the example shown in FIG. 15 describes a circular coordinate system, similar processing is also possible for a Cartesian coordinate system.

[0148] According to this modification, an operator viewing display image 201 can quickly grasp the occurrence of a situation in which there is a high possibility that autonomous mobile device 1A will collide with an object, and can perform collision avoidance control such as making an emergency stop of autonomous mobile device 1A by remote control. Note that collision avoidance control may be performed by automatic control by circuit 31 instead of manual operation by the operator.

[0149] 16 is a simplified diagram showing the configuration of an information processing system according to a fifth modification. The autonomous mobile device 1 includes a position detection unit 14, a speed detection unit 15, and an orientation detection unit 16 in addition to the configuration shown in FIG.

[0150] The position detection unit 14, speed detection unit 15, and orientation detection unit 16 input real-space detection information, which is detection information in real space related to the autonomous mobile device 1, to the circuit 21. The real-space detection information includes position information and movement information. The position information related to the autonomous mobile device 1 includes an absolute position. The movement information related to the autonomous mobile device 1 includes an absolute speed and an absolute movement direction.

[0151] The position detection unit 14 is configured to include a GPS receiver, etc. The position detection unit 14 detects the absolute position of the autonomous mobile device 1 and inputs the detected value to the circuit 21. The absolute position is a coordinate value (latitude and longitude) in a geographic coordinate system such as world coordinates.

[0152] The speed detection unit 15 is configured to include a tachometer, etc. The speed detection unit 15 detects the absolute speed of the autonomous mobile device 1 and inputs the detected value to the circuit 21.

[0153] The orientation detection unit 16 is configured to include an acceleration sensor, a gyro sensor, etc. The orientation detection unit 16 detects the forward direction or absolute movement direction of the autonomous mobile device 1, and inputs the detected value to the circuit 21.

[0154] The circuit 21 derives real-space detection information regarding the autonomous mobile device 1 based on the values ​​detected by the position detection unit 14, the speed detection unit 15, and the direction detection unit 16, converted from each detection unit's unique coordinate system to a common coordinate system of the autonomous mobile device 1.

[0155] 17 is a flowchart showing the flow of processing executed by the circuit 21 of the encoding device 11. Description of processing that is the same as that shown in the flowchart of FIG.

[0156] In step SP31, the circuit 21 acquires real space detection information related to the autonomous mobile device 1A from the position detection unit 14, the speed detection unit 15, and the orientation detection unit 16.

[0157] In step SP32, the circuit 21 encodes the real space detection information related to the autonomous mobile device 1A into a bit stream BS. The circuit 21 encodes the real space detection information acquired in step SP31 into a predetermined location in the header 41. The predetermined location is the SEI area 43. The predetermined location may be the VPS, SPS, PPS, PH, SH, APS, or tile header, or the like. Alternatively, the predetermined location may be the ARSEI area.

[0158] 18 is a diagram showing a simplified example of syntax. The third_information includes real space detection information related to the autonomous mobile device 1A.

[0159] 19 is a flowchart showing the flow of processing executed by the circuit 31 of the decoding device 2. Description of processing that is the same as that in the flowchart shown in FIG.

[0160] In step SP33, the circuit 31 decodes the real space detection information related to the autonomous mobile device 1A from the header 41 of the bit stream BS.

[0161] In step SP25, the circuit 31 generates a display image 200 for human vision by the operator based on the images and information decoded in steps SP22 to SP24 and SP33.

[0162] 20 is a simplified diagram showing an example of the display image 200. When generating the display image 200, the circuit 31 may convert the relative position information of the object in real space decoded in step SP24 into absolute position information of the object in real space using the absolute position information of the autonomous mobile device 1A in real space decoded in step SP33.

[0163] Furthermore, circuit 31 may convert the relative movement information of the object in real space decoded in step SP24 into absolute movement information of the object in real space using the absolute movement information of the autonomous mobile device 1A in real space decoded in step SP33. In the example of display movement information D1 to D3 shown in Fig. 20, the relative speed of each object with respect to the autonomous mobile device 1A is converted into an absolute speed.

[0164] Furthermore, the circuit 31 may include absolute movement information and absolute position information of the autonomous mobile device 1A in real space decoded in step SP33 in the display image 200. In the example shown in FIG. 20 , the display image 200 includes host information C0 indicating the absolute speed and absolute position of the autonomous mobile device 1A. The host information C0 may further include the absolute movement direction of the autonomous mobile device 1A. The absolute movement direction may be indicated, for example, by orientation information indicating the movement direction of the autonomous mobile device 1A.

[0165] The processing according to this modification may be applied not only to the display image 200 but also to the display images 201 and 202 .

[0166] According to this modification, the absolute position information and absolute movement information of the autonomous mobile device 1A in real space can be effectively utilized on the decoding device 2 side.

[0167] 21 is a simplified diagram showing an example of a display image 203 as a modification of the display image 202. The display image 203 includes a figure G4 that resembles a pedestrian 6C, which is an object, in addition to the figures G0 to G3 shown in FIG.

[0168] The pedestrian 6C is photographed by the imaging unit 12 of the autonomous mobile device 1B. The circuit 21 of the encoding device 11 of the autonomous mobile device 1B encodes the image photographed by the imaging unit 12, intra-image detection information relating to objects included in the image, and real-space detection information relating to the objects into a bitstream BS and transmits it to the decoding device 2. The real-space detection information includes relative position information and relative movement information of the pedestrian 6C with respect to the autonomous mobile device 1B, and absolute position information and absolute movement information of the autonomous mobile device 1B.

[0169] The circuit 31 of the decoding device 2 receives multiple bitstreams BS from multiple autonomous mobile devices 1A and 1B. The circuit 31 generates a display image 203 based on multiple images decoded from the multiple bitstreams BS. The circuit 31 calculates absolute position information and absolute movement information of the pedestrian 6C by converting the relative position information and relative movement information of the pedestrian 6C relative to the autonomous mobile device 1B using the absolute position information and absolute movement information of the autonomous mobile device 1B. This allows the display image 203 to include a figure G4 corresponding to the pedestrian 6C that cannot be captured by the capture unit 12 of the autonomous mobile device 1A. The display movement information D0 to D4 includes absolute speed information indicating the absolute speed of each object and absolute movement direction information indicating the absolute movement direction of each object. Note that while the example shown in FIG. 21 describes a Cartesian coordinate system, similar processing is also possible for a circular coordinate system.

[0170] According to this modification, multiple images captured by multiple autonomous mobile devices 1A, 1B can complement each other's blind spots during image capture, thereby improving the convenience of the display image 203.

[0171] Second Embodiment In a second embodiment, communication between a plurality of autonomous mobile devices 1 will be described.

[0172] 22 is a diagram showing a simplified configuration of an information processing system according to a second embodiment of the present disclosure. The information processing system includes autonomous mobile devices 1A and 1B.

[0173] The autonomous mobile device 1A has an encoding device 11A, a photographing unit 12A, an object detection unit 13A, a position detection unit 14A, a speed detection unit 15A, an orientation detection unit 16A, a decoding device 2A, and a driving unit 17A. The encoding device 11A includes a circuit 21A and a memory 22A. The decoding device 2A includes a circuit 31A and a memory 32A. The driving unit 17A is configured with the traveling motor of the autonomous mobile device 1A, etc.

[0174] The autonomous mobile device 1B has an encoding device 11B, an imaging unit 12B, an object detection unit 13B, a position detection unit 14B, a speed detection unit 15B, an orientation detection unit 16B, a decoding device 2B, and a driving unit 17B. The encoding device 11B includes a circuit 21B and a memory 22B. The decoding device 2B includes a circuit 31B and a memory 32B. The driving unit 17B is configured to include the traveling motor of the autonomous mobile device 1B, etc.

[0175] Circuit 21B of encoding device 11B transmits bit stream BSB, and circuit 31A of decoding device 2A receives bit stream BSB. Encoding device 11A is mounted on autonomous mobile device 1B, and decoding device 2A is mounted on an autonomous mobile device 1A different from autonomous mobile device 1B. Similarly, circuit 21A of encoding device 11A transmits bit stream BSA, and circuit 31B of decoding device 2B receives bit stream BSA.

[0176] FIG. 23 is a flowchart showing the flow of processing executed by the circuit 31A of the decoding device 2A.

[0177] In step SP51, the circuit 31A receives the bit stream BSB from the autonomous mobile device 1B.

[0178] In step SP52, the circuit 31A decodes the image 100B including the object from the payload 42 of the bitstream BSB.

[0179] 24 is a diagram schematically showing an example of an image 100B captured by the image capturing unit 12B of the autonomous mobile device 1B. Image 100B includes, as objects, the autonomous mobile device 1A and a pedestrian 6C walking behind the autonomous mobile device 1A.

[0180] In step SP53, the circuit 31A decodes the intra-image detection information relating to the object contained in the image 100B from the header 41 of the bitstream BSB.

[0181] In step SP54, the circuit 31A decodes the real space detection information relating to the object contained in the image 100B from the header 41 of the bitstream BSB.

[0182] In step SP55, the circuit 31A decodes the real space detection information related to the autonomous mobile device 1B from the header 41 of the bit stream BSB.

[0183] In step SP56, circuit 31A acquires from circuit 21A the image 100 captured by the photographing unit 12A of the autonomous mobile device 1A, in-image detection information and real-space detection information regarding objects contained in image 100, and real-space detection information regarding the autonomous mobile device 1A.

[0184] In step SP57, circuit 31A controls the driving of autonomous mobile device 1A by drive unit 17A based on the in-image detection information and real-space detection information regarding the object decoded from bit stream BSB and the in-image detection information and real-space detection information regarding the object obtained from circuit 21A.

[0185] Specifically, the circuit 31A generates a map 300 that indicates the position information and movement information of objects present around the autonomous mobile device 1A. In this process, the circuit 31A converts the real-space detection information about the objects, decoded from the bitstream BSB and using a coordinate system based on the autonomous mobile device 1B, into real-space detection information about the objects using a coordinate system based on the autonomous mobile device 1A. For example, the circuit 31A calculates absolute position information and absolute movement information about the pedestrian 6C by converting the relative position information and relative movement information about the pedestrian 6C with respect to the autonomous mobile device 1B using the absolute position information and absolute movement information about the autonomous mobile device 1B. The circuit 31A includes the absolute position information and absolute movement information of the object corresponding to the pedestrian 6C in the map 300. Furthermore, the circuit 31A may calculate the relative position information and relative speed information of the pedestrian 6C with respect to the autonomous mobile device 1A by converting the absolute position information and absolute movement information of the pedestrian 6C using the absolute position information and absolute movement information of the autonomous mobile device 1A. In this case, the circuit 31A may include in the map 300 the relative position information and relative movement information of the pedestrian 6C with respect to the autonomous mobile device 1A.

[0186] 25 is a diagram showing a simplified example of a map 300. The map 300 has a coordinate plane of a Cartesian coordinate system in which the absolute position of the autonomous mobile device 1A is the origin O and the direction of movement of the autonomous mobile device 1A is the X-axis. The map 300 includes absolute position information and absolute movement information of the object corresponding to the autonomous mobile device 1B and pedestrians 6A and 6B, as well as absolute position information and absolute movement information of the object corresponding to the pedestrian 6C.

[0187] Furthermore, on map 300, a boundary Z is set around the autonomous mobile device 1A. Boundary Z corresponds to a circle with a radius of, for example, 1 m, centered on the position of autonomous mobile device 1A in real space. When an object enters boundary Z, circuit 31 determines that there is a high possibility that the autonomous mobile device 1A will collide with the object, and performs collision avoidance control such as bringing the autonomous mobile device 1A to an emergency stop using drive unit 17A.

[0188] 25 has been described using a Cartesian coordinate system, similar processing is possible for a circular coordinate system. Furthermore, instead of generating map 300, circuit 31A may generate a table listing position information and movement information of objects present around autonomous mobile device 1A.

[0189] According to this modified example, by mutually communicating in-image detection information and real-space detection information regarding objects between multiple autonomous mobile devices 1A, 1B, the autonomous mobile device 1A can effectively utilize this information received from another autonomous mobile device 1B to control its own operation, thereby improving the safety of the autonomous mobile device 1, such as its collision avoidance performance.

[0190] The present disclosure is particularly useful when applied to product delivery systems or transportation systems that use autonomous mobile devices.

Claims

1. A circuit, a memory connected to the circuit, comprising: the circuit receives a bitstream from an autonomous driving device having an imaging unit, uses the bitstream to acquire an image captured by the imaging unit, acquire in-image detection information, which is detection information within the image regarding an object included in the image, acquire real-space detection information, which is detection information in real space regarding the object, a decoding device.

2. The in-image detection information regarding the object includes position information of the object within the image, The decoding device according to Claim 1.

3. The real-space detection information regarding the object includes at least one of position information of the object in real space and movement information of the object in real space, The decoding device according to Claim 1.

4. The position information of the object includes coordinate values in a polar coordinate system with respect to the autonomous driving device, coordinate values in a Cartesian coordinate system with respect to the autonomous driving device, or coordinate values in a geographic coordinate system , The decoding device according to Claim 3.

5. The movement information of the object includes at least one of the absolute speed of the object or the relative speed of the object with respect to the autonomous driving device and the absolute movement direction of the object or the relative movement direction of the object with respect to the autonomous driving device, The decoding device according to Claim 3.

6. The circuit further decodes real-space detection information regarding the autonomous driving device from the bitstream, The real-space detection information regarding the autonomous driving device includes position information of the autonomous driving device in real space and movement information of the autonomous driving device in real space, The decoding device according to Claim 1.

7. The position information of the autonomous driving device includes coordinate values in a geographic coordinate system, The decoding device according to Claim 6.

8. The movement information of the autonomous driving device includes at least one of the absolute speed and the absolute movement direction of the autonomous driving device, The decoding device according to Claim 6.

9. The circuit decodes the in-image detection information and the real-space detection information regarding the object and the real-space detection information regarding the autonomous driving device from the SEI (Supplemental Enhancement Information) area of the bitstream, The decoding device according to claim 6.

10. The circuit further generates a display image for human vision, The display image The object, At least one of the position information for displaying the object in the real space and the movement information for displaying the object in the real space, including The decoding device according to claim 1.

11. The autonomous driving device includes a plurality of autonomous driving devices, The circuit generates the display image based on a plurality of images decoded from a plurality of bitstreams received from the plurality of autonomous driving devices. The decoding device according to claim 10.

12. The circuit uses the position information of the object in the real space included in the real space detection information regarding the object, decoded from the bitstream, as the position information for displaying the object, uses the movement information of the object in the real space included in the real space detection information regarding the object, decoded from the bitstream, as the movement information for displaying the object. The decoding device according to claim 10.

13. The circuit further decodes the position information of the autonomous driving device in the real space and the movement information of the autonomous driving device in the real space from the bitstream, calculates the position information for displaying the object by converting the position information of the object in the real space included in the real space detection information regarding the object, decoded from the bitstream, using the position information of the autonomous driving device in the real space, decoded from the bitstream, calculates the movement information for displaying the object by converting the movement information of the object in the real space included in the real space detection information regarding the object, decoded from the bitstream, using the movement information of the autonomous driving device in the real space, decoded from the bitstream. The decoding device according to claim 10.

14. The circuit calculates the movement information for displaying the object based on the position information of the object at a plurality of times in the real space included in the real space detection information regarding the object, decoded from the bitstream. The decoding device according to claim 10.

15. The decoding device is mounted on an autonomous driving device different from the autonomous driving device, and the circuit further controls the operation of the other autonomous driving device based on the in-image detection information and the real-space detection information regarding the object decoded from the bit stream received from the autonomous driving device, and the in-image detection information and the real-space detection information regarding the object included in the image captured by the imaging unit of the other autonomous driving device. The decoding device according to claim 1.

16. The circuit converts the real-space detection information regarding the object, which is decoded from the bit stream received from the autonomous driving device and uses the coordinate system based on the autonomous driving device, into the real-space detection information regarding the object using the coordinate system based on the other autonomous driving device. The decoding device according to claim 15.

17. An encoding device mounted on an autonomous driving device having an imaging unit, comprising a circuit and a memory connected to the circuit, wherein the circuit encodes an image captured by the imaging unit, generates in-image detection information, which is detection information in the image regarding the object included in the image, generates real-space detection information, which is detection information in the real space regarding the object, and generates a bit stream. Encoding device.

18. The in-image detection information regarding the object includes position information of the object in the image. The encoding device according to claim 17.

19. The real-space detection information regarding the object includes at least one of position information of the object in the real space and movement information of the object in the real space. The encoding device according to claim 17.

20. The position information of the object includes coordinate values in a polar coordinate system based on the autonomous driving device, coordinate values in a Cartesian coordinate system based on the autonomous driving device, or coordinate values in a geographic coordinate system. The encoding device according to claim 19.

21. The movement information of the object includes at least one of the absolute speed of the object or the relative speed of the object with respect to the autonomous driving device, and the absolute movement direction of the object or the relative movement direction of the object with respect to the autonomous driving device. The encoding device according to claim 19.

22. ​ ​ The circuit further encodes the real-space detection information regarding the autonomous driving device into the bitstream. The real-space detection information regarding the autonomous driving device includes the position information of the autonomous driving device in the real space and the movement information of the autonomous driving device in the real space. The encoding device according to claim 17.

23. The position information of the autonomous driving device includes coordinate values in a geographical coordinate system. The encoding device according to claim 22.

24. The movement information of the autonomous driving device includes at least one of the absolute speed and the absolute movement direction of the autonomous driving device. The encoding device according to claim 22.

25. The circuit encodes the in-image detection information and the real-space detection information regarding the object and the real-space detection information regarding the autonomous driving device into the SEI (Supplemental Enhancement Information) area of the bitstream. The encoding device according to claim 22.

26. The autonomous driving device further includes a detection unit that detects the object in the real space. Based on the value obtained by converting the detection value by the detection unit from the coordinate system of the detection unit to the coordinate system of the autonomous driving device, the circuit derives the real-space detection information regarding the object. The encoding device according to claim 17.

27. A decoding device receives a bitstream from an autonomous driving device having an imaging unit, uses the bitstream to acquire the image captured by the imaging unit, acquire the in-image detection information, which is the detection information within the image regarding the object included in the image, acquire the real-space detection information, which is the detection information in the real space regarding the object. A decoding method.

28. An encoding device mounted on an autonomous driving device having an imaging unit encodes the image captured by the imaging unit, generates the in-image detection information, which is the detection information within the image regarding the object included in the image, generates the real-space detection information, which is the detection information in the real space regarding the object, and generates a bitstream. An encoding method.