Control device

Abstract

This invention relates to a control device for improving the operability of a mobile body in remote operation. The control device includes: a communication unit for communicating with the mobile body that acquires a captured image of a subject and information about the distance to the subject; and a control unit for reducing the density of a set of dots generated based on the distance information and corresponding to the surface of the subject, and outputting a display image in which each dot is depicted with a different area according to its distance from the mobile body and superimposed on the captured image.

Description

Technical Field

[0001] This disclosure relates to a control device. Background Technology

[0002] Various technologies for remotely operating mobile bodies such as traveling robots have been proposed. For example, Patent Document 1 discloses a technology for displaying a mark in a remote operating device that indicates the target position toward which the mobile body is facing.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2022-173195

[0004] When the difficulty of remote operation changes depending on the surrounding environment and speed of the mobile object's movement path, there is room to improve the operability of remote operation of the mobile object. Summary of the Invention

[0005] The following discloses a control device that can improve the operability of remote operation of a mobile body.

[0006] The control device disclosed herein includes: a communication unit that communicates with a moving body that acquires a captured image of a subject and information about the distance to the subject; and a control unit that reduces the density of a set of points generated based on the distance information and corresponding to the surface of the subject, and outputs a display image in which each point is depicted with a different area according to its distance from the moving body and superimposed on the captured image.

[0007] The control device according to this disclosure can improve the operability of remote operation of a mobile body. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating an example of the structure of a remote operating system.

[0009] Figure 2 It is a timing diagram representing the action steps of a remote operating system.

[0010] Figure 3 This is a diagram representing an example of a display screen.

[0011] Explanation of reference numerals in the attached figures

[0012] 1…Remote operating system; 10…Control device; 12…Mobile body; 101…Communication unit; 102…Storage unit; 103…Control unit; 105…Input unit; 106…Output unit. Detailed Implementation

[0013] The implementation method will be described below.

[0014] Figure 1This diagram illustrates an example configuration of a remote operating system in one embodiment. The remote operating system 1 includes a mobile body 12 and a control device 10 for remotely operating the mobile body 12. The mobile body 12 and the control device 10 are interconnected via a network 11 and mediated by a server device 13 to enable information communication. The mobile body 12 is, for example, a mobile robot used for transporting goods in warehouses, commercial facilities, etc. The control device 10 is an information processing device with communication capabilities and an interface for user operation, such as a PC (personal computer), a smartphone terminal, a tablet terminal, or an operating terminal for the mobile body 12. The network 11 is, for example, the Internet, but can also be an ad hoc network, a LAN (Local Area Network), a MAN (Metropolitan Area Network), or other networks, or any combination thereof. The server device 13 is, for example, a server computer belonging to a cloud computing system or other computing system and equipped with various functions.

[0015] In the remote operating system 1 of this embodiment, the control device 10 includes: a communication unit 101 that communicates with a mobile body 12 that acquires a captured image of a subject and information about the distance to the subject (hereinafter referred to as distance information); and a control unit 103 that reduces the density of a set of dots generated based on the distance information and corresponding to the surface of the subject, and outputs a display image formed by depicting each dot with a different area according to the distance to the mobile body 12 and overlaying it onto the captured image. When depicting various targets, obstacles, etc., in the environment in which the mobile body 12 travels using a set of dots based on the distance information and overlaying them onto the captured image, reducing the density of the dot set reduces the processing load of the control device 10, enabling faster image processing, and displaying each dot with a different area according to the distance allows for a more realistic representation of three-dimensionality. Therefore, the operability of the mobile body 12 in remote operation can be improved.

[0016] like Figure 1 As shown, the control device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106.

[0017] The communication unit 101 is equipped with modules that are compatible with communication standards such as wired LAN, wireless LAN, LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), and is connected to the network 11 via a router or base station. Through the communication unit 101, the control device 10 communicates with the mobile body 12 via the network 11 and with the help of the server device 13.

[0018] Storage unit 102 includes semiconductor memory, magnetic memory, or optical memory. Semiconductor memory includes RAM (Random Access Memory) or ROM (Read Only Memory). RAM includes SRAM (Static RAM) or DRAM (Dynamic RAM), and ROM includes EEPROM (Electrically Erasable Programmable ROM). Storage unit 102 stores the control program, information required for operation, or operation results of control device 10.

[0019] The control unit 103 includes processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), or dedicated circuits such as FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit) to perform information processing of the control device 10.

[0020] The input unit 105 is equipped with a microphone, electrostatic key, touch screen, pointing device, or physical switches such as physical keys and joysticks, and accepts user operations and sends the input information corresponding to the operation to the control unit 103.

[0021] The output unit 106 includes a speaker or a display. The display may be an LCD (Liquid Crystal Display) or an organic EL (Organic Electroluminescence) display, and the output unit 103 may display the processing results.

[0022] The mobile body 12 is a traveling robot equipped with a mechanism for transporting goods and a driving mechanism including wheels. It includes a communication unit 121, a control unit 123, a sensor unit 127, and a drive mechanism 128. The communication unit 121 is compatible with communication standards such as wired LAN, wireless LAN, LTE, 4G, and 5G, and communicates with the control device 10. The control unit 123 includes a processor such as a CPU or a dedicated circuit such as an ASIC, and comprehensively controls the movement of the mobile body 12 while communicating with the control device 10 through the communication unit 121. The sensor unit 127 includes a camera and a ranging sensor. The camera is a visible light camera that captures images based on visible light and is positioned to include at least the environment in front of the moving direction in the field of view when the mobile body 12 moves. The ranging sensor is a sensor used to measure the distance to objects, including ultrasonic sensors, infrared sensors, and LiDAR (Light Detection and Ranging) sensors. The ranging sensor may also include a depth camera. A ranging sensor is positioned at least at the location of the moving body 12, which is capable of measuring the distance to objects within the shooting range of the visible light camera. Additionally, the sensor unit 127 includes an accelerometer, a gyroscope, a distance sensor, and a GPS (Global Positioning System) module. The sensor unit 127 acquires information such as captured images, ranging results, detection results related to the surrounding environment, and detection results of the movement state of the moving body 12, and transmits this information to the control unit 103. The control unit 103 transmits captured images, etc., to the control device 10 via the communication unit 121, receives instructions from the control device 10, and controls the operation of the drive mechanism 128. The drive mechanism 128 includes wheels for travel, motors for driving the wheels, and actuators, and performs forward, backward, and rotational movements of the moving body 12 based on instructions from the control unit 123. Power is supplied to all of the above components from a battery mounted on the moving body 12.

[0023] Figure 2 (A) is a timing diagram used to illustrate the action steps of remote operating system 1. Figure 2 (A) shows an example of the steps involved in the coordinated operation of the mobile body 12 and the control device 10, which are executed at any period, such as tens of microseconds, when the mobile body 12 moves.

[0024] The mobile body 12 takes pictures and measures the distance of objects in its surrounding environment (S20), and sends the captured images and distance information to the control device 10 (S21). In the mobile body 12, under the control of the control unit 123, the visible light camera of the sensor unit 127 captures pictures of the environment around the mobile body 12, such as the environment in front of it in the direction of travel, and the distance of objects within the shooting range is measured by the range sensor. Then, the control unit 123 uses the measurement results of the range sensor to generate distance information. For example, the control unit 123 generates distance information by deriving the spatial coordinates of each reflection point of the object being photographed when the shooting range of the visible light camera is scanned using a sensor such as LiDAR. Alternatively, the control unit 123 generates distance information by deriving the spatial coordinates corresponding to each pixel of the image of the object being photographed using a depth image from a depth camera. Then, the control unit 123 sends the captured images and distance information from the visible light camera to the communication unit 121.

[0025] Next, the control device 10 processes the captured image and distance information obtained from the moving body 12 to generate a set of dots corresponding to the photographed object (S22), and performs voxel filtering on the set of dots (S23) to generate an image for display (S24). First, the control unit 123 generates a set of dots corresponding to each spatial coordinate contained in the distance information. At this time, the set of dots is generated with a density corresponding to the distribution of the spatial coordinates. Through voxel filtering, the set of dots is divided into arbitrary unit spaces, such as cubic spaces of a few centimeters, and representative dots are retained in each unit space while other dots are deleted, thereby reducing the density of the set of dots. Next, the control unit 123 depicts each dot in the reduced-density set of dots with an area corresponding to the distance from the moving body. Each dot can be any shape, such as a circle or an ellipse; the larger the distance, the smaller the dot is depicted, and the smaller the distance, the larger the dot is depicted. Furthermore, the control unit 123 can depict each dot with any different color corresponding to the distance from the moving body. Then, the control unit 123 performs noise removal, deformation correction, and other processing on the captured image, and performs recognition processing on various objects to generate a display image by overlaying the drawn point group onto the corresponding subject in the captured image. The information that establishes the correspondence between the coordinate system in the captured image and the coordinate system in the measurement result of the ranging sensor is pre-stored in the storage unit 122. The control unit 123 can use this information to establish the correspondence between the subject in the captured image and the point group and overlay the point group onto the captured image.

[0026] In generating the image for display, the control unit 123 of the control device 10 can determine the area of ​​each point in the point group based on the camera projection formula of any camera model. The camera model includes a pinhole camera model, a dual-spherical camera model, etc. For example, if the position of point P in the point group in the camera coordinate system is set as (x, y, z), and the pixel position of point P displayed in the captured image is set as point P'(u, v), then the camera projection formulas u and v are expressed as follows.

[0027] u = u(x,y,z), v = v(x,y,z)

[0028] Therefore, the radius r of the circle when point P is depicted as a circle is calculated using the following (Equation 1). Here, voxel_size represents the size of a unit space.

[0029] (Formula 1) r=max(∂u / ∂x+∂u / ∂y+∂u / ∂z,∂v / ∂x+∂v / ∂y+∂v / ∂z)* voxel_size / 2

[0030] In Equation 1 above, the minimum, median, or average value can be used instead of the maximum value obtained by partial differentiation of u and v with respect to x, y, and z, respectively. Alternatively, the axes r_u and r_v of the ellipse when point P is depicted as an ellipse can be derived by Equation 2 below.

[0031] (Equation 2)

[0032] r_u=(∂u / ∂x+∂u / ∂y+∂u / ∂z)* voxel_size / 2

[0033] r_v=(∂v / ∂x+∂v / ∂y+∂v / ∂z)* voxel_size / 2

[0034] By depicting each point with a circle or ellipse whose area is determined as described above, the area of ​​each point corresponding to the distance can be determined without being significantly affected by lens distortion in the captured image.

[0035] Next, the control device 10 displays an image to the user (S24). The control unit 103 displays the image on the display of the output unit 106.

[0036] Figure 3An example of a display image is shown. For example, display image 30 includes an image of the interior environment of an office captured by a moving object 12 moving within the office. In display image 30, a group 32 of dots superimposed on the subjects 31, 33, and 34 is shown. Here, in the group 32-1 corresponding to the subject 31, which is closer to the moving object 12, each dot is depicted as a circle with a relatively large area. In the group 32-2 corresponding to the subject 33, which is farther from the moving object 12, and the group 32-3 corresponding to the subject 34, each dot is depicted as a circle with a relatively small area. Here, the closer group 32-1 can be depicted with a color that is more prominent in brightness or saturation than the farther groups 32-2 and 32-3. Each dot can also be semi-transparent. Alternatively, each dot in the group 32 can also be depicted as an ellipse. The displayed image allows users to visually confirm the environment in front of the moving object 12 with a more realistic sense of depth.

[0037] return Figure 2 (A) The control device 10 accepts the user's operation based on the displayed image (S26) and sends an action instruction corresponding to the operation to the mobile body 12 (S27). The control unit 103 generates an action instruction corresponding to the user's operation on the input unit 105 and sends the action instruction through the communication unit 121. The action instruction is, for example, an instruction to accelerate, decelerate, stop, or turn. Then, the mobile body 12 executes the action corresponding to the action instruction, that is, to accelerate, decelerate, stop, or turn (S28).

[0038] By displaying an image to the user based on the steps described above, the control device 10 enables the user to remotely operate the mobile body 12 while simultaneously checking its position in front. This improves the user's visibility in the direction of travel during operation. Consequently, the operability of the mobile body 12 during remote operation is enhanced.

[0039] In a modified example, the control unit 123 of the moving body 12 may also perform part or all of the processing from point group generation to image generation for display, and the control unit 123 receives information about the processing results from the moving body 12. For example, in Figure 2 The steps shown in (B) are as follows: dot group generation (S22), voxel filtering processing (S23), and display image generation (S24) are performed in the control unit 123 of the mobile body 12; the display image is sent from the mobile body 12 to the control device 10 (S24'); and the received operation display image is displayed by the control device 10 (S25). In this way, the image processing load can be appropriately distributed between the mobile body 12 and the control device 10.

[0040] In the foregoing description, the implementation methods have been described based on the accompanying drawings and embodiments. However, it should be noted that those skilled in the art can easily make various modifications and alterations based on this disclosure. Therefore, these modifications and alterations are included within the scope of this disclosure. For example, the functions included in each mechanism, step, etc., can be reconfigured in a logically consistent manner, and multiple mechanisms, steps, etc., can be combined into one or divided.

Claims

1. A control device, wherein, have: The communication unit communicates with the mobile body that acquires images of the subject and information about the distance to the subject; and The control unit reduces the density of the point group generated based on the distance information and corresponding to the surface of the subject, and outputs a display image in which each point is depicted with a different area according to the distance from the moving object and superimposed on the captured image.

2. The control device according to claim 1, wherein, When a first point is depicted with a first area, a second point that is at a distance greater than the first point from the moving body is depicted with a second area smaller than the first area.

3. The control device according to claim 2, wherein, The first point and the second point are depicted in different colors.

4. The control device according to claim 1, wherein, The density of the point group is reduced by dividing the area of ​​the point group into unit regions and leaving representative points in each unit region.

Images