Remote operation method and remote operation device

Abstract

To provide images from two cameras with partially overlapping coverage areas to a remote operator in the form of a quick and useful composite image.SOLUTION: A method of performing remote operation of a mobile is provided. In this method, data on two types of camera images whose photographic ranges partially overlap and data on the content of the remote operation are acquired. Furthermore, a vertical position at which the two types of camera images are to be combined is set on the basis of the operation content data. Further, the two types of camera images are combined by moving at least one of the two types of camera images in the vertical direction such that the two types of camera images are combined at the set position in the vertical direction. Then, the composite image of the two types of camera images is output from a display of a remote operation device.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a method and apparatus for performing remote operations on a mobile object. [Background technology]

[0002] JP 2019-156299 A discloses a system that generates a composite image viewed from a virtual viewpoint (e.g., above the vehicle) based on multiple camera images acquired from multiple vehicle-mounted cameras. The composite image is generated in an information processing device mounted on the vehicle. The information processing device transmits the composite image to a mobile terminal of a user of the vehicle. The mobile terminal outputs the composite image to a display unit of the mobile terminal. The user of the vehicle inputs information for operating the vehicle to the mobile terminal while viewing the composite image output to the display unit. This input information is transmitted from the mobile terminal to the information processing device. The information processing device operates the vehicle based on the input information.

[0003] In addition to JP 2019-156299 A, examples of documents showing the technical state of the technical field related to the present disclosure include JP 2020-77173 A and JP 2020-161039 A. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-156299 A [Patent Document 2] Patent Publication No. 2020-77173 [Patent Document 3] JP 2020-161039 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of JP 2019-156299 A, a user who gets off a vehicle moves the vehicle from its current location to a parking space while operating a mobile terminal near the parking space. Therefore, this conventional technology that outputs a composite image seen from a virtual viewpoint to a display unit of a mobile terminal is useful from the viewpoint of performing parking safely.

[0006] Consider a case where remote operation is performed on a moving object, such as a vehicle. In this case, multiple camera images are transmitted from the moving object to a remote operation device. These camera images are then output on at least one display of the remote operation device. A remote operator looks at these camera images output on the display and inputs information for the remote operation into the remote operation device.

[0007] However, when focusing on two of a plurality of camera images, there are cases where the same object is reflected in these camera images. For example, when the shooting ranges of the two cameras partially overlap, the same object is reflected in two types of camera images. Information that the same object exists can be said to be unnecessary information for a remote operator. Therefore, it is desirable to process the two types of camera images so as to remove this unnecessary information. For example, if one camera image is combined with the other camera image at the position of the same object, the unnecessary information can be reduced.

[0008] However, the viewpoints of the multiple cameras are different from each other. Therefore, when images obtained from two types of cameras with partially overlapping shooting ranges are combined, distortion or misalignment occurs at the combined part of the composite image. This problem is expected to be solved by processing the composite image. However, excessive focus on processing the camera images during remote work may hinder the prompt provision of the camera images to the remote operator.

[0009] One object of the present invention is to provide a technique that can provide a remote operator with a quick and useful composite image of images obtained from two cameras with partially overlapping coverage. [Means for solving the problem]

[0010] A first aspect of the present invention is a method for performing remote operation on a moving object, and has the following features. The method comprises: Acquiring data of two types of camera images having a partially overlapping shooting range and data of work content in the remote work; setting a vertical position for combining the two types of camera images based on the data of the work content; moving at least one of the two types of camera images in a vertical direction so that the two types of camera images are joined at a set position in the vertical direction, thereby combining the two types of camera images; outputting a composite image of the two types of camera images from a display of a remote operation device; Includes.

[0011] The second invention has the following additional features in addition to the first invention. The method comprises: determining whether or not an input for adjusting a combining position of the two types of camera images has been received from an input device of the remote operation device while the composite image is being output from the display; adjusting the coupling position based on the adjustment input when it is determined that the adjustment input has been received; outputting the composite image having the adjusted combination position from the display; Further includes.

[0012] The third invention has the following further features in the first or second invention. The method comprises: acquiring data on an internal state of the moving object; generating an auxiliary image of the remote operation based on the data of the internal state; adding the auxiliary image to a blank area of ​​the composite image; outputting the composite image to which the auxiliary image has been added from the display; Further includes:

[0013] A fourth aspect of the present invention is an apparatus for performing remote operation on a moving object, and has the following features. The device includes a storage device, a processor, and a display. Various information is stored in the storage device. The processor processes the various information. An image processed by the processor is output from the display. The various types of information include data of images captured by two types of cameras that have a partially overlapping shooting range, and data on the work content of the remote work. The processor, Based on the data of the work content, a vertical position for combining the two types of camera images is set; moving at least one of the two camera images in a vertical direction so that the two camera images are joined at the set position in the vertical direction, thereby combining the two camera images; A composite image of the two types of camera images is output from the display.

[0014] The fifth invention has the following further features in the fourth invention. The apparatus further includes an input device, the input operation being performed by an operator who performs the remote operation. The processor determines whether or not an input for adjusting a combining position of the two types of camera images has been received from the input device while the composite image is being output from the display; If it is determined that the adjustment input has been received, the coupling position is adjusted based on the adjustment input; The composite image with the adjusted combining position is output from the display.

[0015] The sixth invention, in accordance with the fourth or fifth invention, further has the following features. The various information further includes data on the internal state of the moving body. The processor further comprises: generating an auxiliary image of the remote operation based on the data of the internal state; adding the auxiliary image to a blank area of ​​the composite image; The composite image to which the auxiliary image has been added is output from the display. Effect of the Invention

[0016] According to the first or fourth invention, a composite image of two types of camera images combined at a vertical position set based on the data of the work content is output from the display. Therefore, it is possible to generate a composite image that ensures the visibility of the images at the combined position. In addition, it does not take a long time to generate this composite image. Therefore, it is possible to quickly provide the composite image to the operator.

[0017] According to the second or fifth aspect of the present invention, it is possible to provide the operator with a composite image in which two types of camera images are combined at a position on the operator's desired focus.

[0018] According to the third or sixth aspect of the present invention, it is possible to effectively utilize the blank area that is generated by generating a composite image. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a conceptual diagram for explaining a remote operation. [Diagram 2] FIG. 13 is a diagram showing an example of a camera image. [Diagram 3] FIG. 3 is a diagram showing an example of combining the camera images shown in FIG. 2. [Figure 4] FIG. 3 is a diagram showing an example of combining the camera images shown in the upper part of FIG. [Diagram 5] FIG. 3 is a diagram showing an example of combining the camera images shown in the lower part of FIG. [Figure 6] FIG. 13 is a diagram illustrating an example of a working image in which an additional image is inserted into a blank area. [Figure 7] 2 is a block diagram showing a configuration example of a moving body shown in FIG. 1. [Figure 8] FIG. 2 is a diagram illustrating an example of a data configuration of an operation plan. [Figure 9]8 is a block diagram showing an example of a functional configuration of a data processing device of a moving body shown in FIG. 7. [Figure 10] 2 is a block diagram showing an example of the configuration of the remote operation device shown in FIG. 1. [Figure 11] 11 is a block diagram showing an example of the functional configuration of a data processing device of the remote operation apparatus shown in FIG. [Figure 12] 11 is a flowchart showing an example of processing executed by a data processing device of the remote-operated device shown in FIG. [Figure 13] 11 is a flowchart showing another example of the process executed by the data processing device of the remote-operated device shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, a remote operation method and a remote operation device according to an embodiment of the present invention will be described with reference to the drawings. The remote operation method according to the embodiment is realized by computer processing performed in the remote operation device according to the embodiment. In addition, in each drawing, the same or corresponding parts are given the same reference numerals, and the description thereof will be simplified or omitted.

[0021] 1. Overview of the embodiment 1-1.Remote work FIG. 1 is a conceptual diagram explaining remote operation. FIG. 1 illustrates a moving body 1 and a remote operation device 2. The moving body 1 and the remote operation device 2 constitute a remote operation system. The moving body 1 is an example of a target that receives remote operation by the remote operation device 2. Examples of the moving body 1 include vehicles and robots such as logistics robots and work robots.

[0022] An example of remote work is remote driving, which is aimed at operations directly related to the behavior of the moving body 1 (e.g., moving forward, stopping, turning, etc.). Remote work also includes remote command and remote assistance. Remote command is performed for the purpose of operations not directly related to the behavior of the moving body 1 (e.g., turning on the lights of the moving body 1). Remote assistance is performed to assist the recognition and judgment of the moving body 1.

[0023] The mobile object 1 and the remote operation device 2 communicate with each other via a base station (not shown) of a network. For example, the data processing device 12 mounted on the mobile object 1 determines whether or not remote operation is necessary. Then, if it is determined that remote operation is necessary, the data processing device 12 transmits a signal requesting remote operation to the remote operation device 2. When transmitting this signal, or prior to transmitting this signal, the data processing device 12 transmits camera image IMG data to the remote operation device 2. The camera image IMG data is acquired from the camera 11 mounted on the mobile object 1.

[0024] The data processing device 22 of the remote operation device 2 transmits various remote commands INS to the moving body 1 in response to a signal requesting remote operation. Examples of the remote command INS transmitted from the data processing device 22 to the moving body 1 include commands based on the judgment of the operator OP. For example, the operator OP understands the content of the remote operation request by looking at the camera image for remote operation (hereinafter also referred to as "work image") IMG_O output from the display 21 of the remote operation device 2. The operator OP then determines the action that the moving body 1 should take, and inputs instructions corresponding to this action to the remote operation device 2. Examples of the action that the moving body 1 should take include moving forward, stopping, offset avoidance of an obstacle ahead, emergency evacuation, etc. The data processing device 22 generates a remote command INS according to the input data from the operator OP, and transmits it to the moving body 1.

[0025] 1-2. Camera images for remote work FIG. 2 is a diagram showing an example of a camera image IMG. FIG. 2 shows camera images IMG1 to IMG3 captured by three on-board cameras when the moving object 1 is a vehicle. Camera image IMG1 is captured by a camera (front center camera) capturing an image of the space in the center in front of the vehicle. Camera image IMG2 is captured by a camera (left front camera) capturing an image of the space in front of the left of the vehicle, and camera image IMG3 is captured by a camera (right front camera) capturing an image of the space in front of the right of the vehicle. A left portion of the capture range of the front center camera overlaps with that of the left front camera. A right portion of the capture range of the front center camera overlaps with that of the right front camera.

[0026] The upper part of Fig. 2 shows camera images IMG1 to IMG3 taken when entering an intersection. Camera image IMG1 shows a traffic signal, a pedestrian crossing painted on the road surface in front of the vehicle, a guide frame for right turns, etc. Camera image IMG2 shows a pedestrian crossing painted on the road surface in front of the vehicle, as well as a pedestrian crossing painted on the road surface to the left front of the vehicle, and pedestrians on this crosswalk, etc. Camera image IMG3 shows a pedestrian crossing painted on the road surface in front of the vehicle, a guide frame for right turns, as well as a right turn lane and an adjacent lane (oncoming traffic lane) to this lane, etc.

[0027] The lower part of Fig. 2 shows camera images IMG1 to IMG3 taken during offset avoidance. Camera image IMG1 captures the lane on which the vehicle is traveling (hereinafter also referred to as the "driving lane"), trucks traveling on this lane, vehicles stopped on the shoulder of this lane, and the lane adjacent to this lane (the oncoming lane). Camera image IMG2 captures the driving lane and stopped vehicles. These stopped vehicles correspond to the target (obstacle) of offset avoidance. Camera image IMG3 captures the driving lane and oncoming lane.

[0028] The work image IMG_O is generated based on the camera images IMG1-IMG3. For example, if the display 21 includes three displays, three types of work images IMG_O are generated based on the camera images IMG1-IMG3. These work images IMG_O are then output from each of the three displays. If the display 21 includes two displays, two types of work images IMG_O are generated based on the camera images IMG1-IMG3, and these work images IMG_O are output from each of the two displays. If the display 21 includes only one display, one type of work image IMG_O is generated based on the camera images IMG1-IMG3, and this work image IMG_O is output from the single display.

[0029] 1-3. Camera image synthesis In the embodiment, a case is considered in which the total number of displays 21 on which work images IMG_O are output is less than the number of cameras 11 that acquire camera images IMG. In this case, camera images IMG1 to IMGm are acquired simultaneously by m cameras 11 (m≧2). If the displays 21 include n displays (n≦m−1), in order to output a work image IMG_O generated based on camera images IMG1 to IMGm from these displays simultaneously, at least two of the camera images IMG1 to IMGm must be combined (combined). For example, in order to output camera images IMG1 to IMG3 from one display simultaneously, these camera images must be combined to generate one work image IMG_O.

[0030] Fig. 3 is a diagram showing an example of combining camera images IMG1 to IMG3 shown in Fig. 2. The upper part of Fig. 3 corresponds to an example of combining camera images IMG1 to IMG3 shown in the upper part of Fig. 2. In the example shown in the upper part of Fig. 3, the right end of camera image IMG2 is simply combined with the left end of camera image IMG1. Therefore, duplication DP of the same pedestrian is seen at the combined position of camera images IMG1 and IMG2. Furthermore, a position gap PG of the pedestrian crossing is confirmed at this combined position.

[0031] In this example, the left edge of the camera image IMG3 is simply aligned with the right edge of the camera image IMG1. Therefore, an overlapping DP of the same frame is seen at the joining position of the camera images IMG1 and IMG3. Also, a positional misalignment PG of the pedestrian crossing is seen at this joining position.

[0032] The lower part of Fig. 3 corresponds to an example of combining camera images IMG1 to IMG3 shown in the lower part of Fig. 2. The camera image combining method in the lower part of Fig. 3 is the same as that in the upper part of Fig. 3. Therefore, overlapping DPs of the same vehicle are seen at the joining position of camera image IMG1 and camera image IMG2. In addition, overlapping DPs of the same white line (lane marking) and overlapping DPs of the same building are confirmed at the joining position of camera image IMG1 and camera image IMG3.

[0033] As can be seen from the explanation of Fig. 3, if the camera images IMG1 to IMG3 are simply arranged and combined, there is no restriction on the total number of displays, but the information reflected at the joining position may confuse the operator OP. If image processing is performed here, such as changing the joining position of the camera images IMG1 to IMG3, it may be possible to avoid such inconvenience. However, since the viewpoints of the vehicle-mounted cameras are different from each other, distortion or misalignment will occur somewhere in the image after image processing (for example, at the joining position).

[0034] 1-4. Combining camera images using work content With recent image processing technology, it is possible to reduce image distortion and misalignment to the utmost. However, if more image processing than necessary is performed, it takes a long time for the processed image (i.e., the work image IMG_O) to be output to the display 21. Thus, there is a trade-off between outputting a work image IMG_O that is easy for the operator OP to see from the display 21 and generating the work image IMG_O from the camera image IMG in a short time.

[0035] Therefore, in the embodiment, in order to ensure visibility while quickly providing the work image IMG_O, the camera images IMG1 to IMG3 are synthesized based on data on the work content of the remote work. The data on the work content is generated in the data processing device 12, for example, before or simultaneously with the transmission of a request signal for the remote work. The data on the work content is transmitted to the remote work device 2 simultaneously or after the transmission of the request signal.

[0036] For example, consider a case where, when a vehicle as a moving body 1 approaches the entrance to an intersection, a remote operation request signal is transmitted to check the traffic conditions at the intersection. In this case, the data processing device 12, for example, performs vehicle control to stop the vehicle near the entrance to the intersection, and transmits the remote operation request signal. The request signal is transmitted after the vehicle has stopped, or before the vehicle has stopped. When the request signal is transmitted, data on the work content is generated. The data on the work content is expressed, for example, by a binary number indicating the work content, "check the traffic conditions at the intersection." Note that the binary number indicating "check the traffic conditions at the intersection" is set in advance.

[0037] The data processing device 22 sets a connecting position CP in the vertical direction (depth direction) of the camera images IMG1 and IMG2 based on the data of the work content. Alternatively, the data processing device 22 sets a connecting position CP of the camera images IMG1 and IMG3. Alternatively, the data processing device 22 sets the above-mentioned two types of connecting positions CP. The total number of connecting positions CP corresponds to the total number of displays 21.

[0038] Fig. 4 shows an example of combining the camera images IMG1 to IMG3 shown in the upper part of Fig. 2. In this example, the display 21 is only one display, and therefore the camera image IMG1 is combined with the camera image IMG2, and at the same time, the camera image IMG1 is combined with the camera image IMG3. When the work content is "checking the traffic conditions at the intersection," for example, the camera images IMG1 to IMG3 are combined so that the operator OP can have a general view of the entire intersection.

[0039] In the example shown in Fig. 4, two types of combining positions CP are set at the rear side of camera image IMG1. Since the shooting range of the vehicle-mounted camera is known, when combining position CP is set at the rear side of camera image IMG1, camera image IMG1 and camera image IMG2 are combined in the vertical direction at this set position. Camera image IMG1 and camera image IMG3 are also combined in the vertical direction at this set position. In this case, a positional deviation PG of the pedestrian crossing occurs at the front side of camera image IMG1.

[0040] When combining the camera images IMG1, IMG2, and IMG3 in the vertical direction, combining in the horizontal direction (vehicle width direction) is performed in parallel. The combining in the horizontal direction is always performed at the same position regardless of the combining position CP in the vertical direction. In other words, the combining in the horizontal direction is performed at a fixed position regardless of the work content.

[0041] As another example, consider a case where a vehicle as the moving body 1 is scheduled to perform offset avoidance and a remote operation request signal is transmitted to check the vicinity of the vehicle. In this case, the data processing device 12 performs vehicle control to stop the vehicle in front of an obstacle and transmits the request signal. When the request signal is transmitted, data on the work content is generated. The data on the work content is expressed by a binary number indicating the work content, for example, "checking surroundings for offset avoidance" or "offset avoidance by remote driving". Note that such a binary number indicating the work content is set in advance.

[0042] Fig. 5 shows another example of combining the camera images IMG1 to IMG3 shown in the lower part of Fig. 2. As in the example shown in Fig. 4, the display 21 in the example shown in Fig. 5 is only one display. Therefore, the camera image IMG1 is combined with the camera image IMG2, and at the same time, the camera image IMG1 is also combined with the camera image IMG3. When the work content is "checking surroundings for offset avoidance" or "offset avoidance driving", for example, the camera images are combined so that the operator OP can check the approaching state of the vehicle to the obstacle.

[0043] In the example shown in FIG. 5, two types of combining positions CP are set at positions on the front side of the camera image IMG1. As described above, the shooting range of the vehicle-mounted camera is known. Therefore, when the combining position CP is set at a position on the front side of the camera image IMG1, the camera images IMG1 and IMG2 are combined at this set position. The camera images IMG1 and IMG3 are also combined at this set position. In this case, a deformation DF of the building occurs at a position on the back side of the camera image IMG3.

[0044] When combining the camera image IMG1 with the camera images IMG2 and IMG3 in the vertical direction, combining in the horizontal direction is carried out in parallel, just like the example described with reference to FIG.

[0045] Thus, in the embodiment, when two types of camera images IMG whose shooting ranges partially overlap are combined to generate one work image IMG_O, the vertical joining position CP is set based on the data of the work content. Therefore, it is possible to generate a work image IMG_O in which the visibility of the image at the joining position CP is ensured, while allowing for distortion and misalignment of the image at positions other than the joining position CP. Moreover, it does not take a long time to generate this work image IMG_O. Therefore, it is also possible to quickly provide the work image IMG_O to the operator OP.

[0046] 4 and 5, before the camera images IMG1 to IMG3 are combined, general image processing (for example, distortion correction due to the shooting angle, lens characteristics, etc.) is performed on these camera images. When combining the camera images IMG1 to IMG3, in addition to the above-mentioned combining, image processing is also performed to match the sizes of objects captured in these camera images. Therefore, a blank area BR is formed in the working image IMG_O.

[0047] In the example shown in Fig. 4, two blank areas BR are formed in the vertical direction of the camera image IMG1. On the other hand, in the example shown in Fig. 5, in addition to the vertical direction of the camera image IMG1, blank areas BR are also formed to the left of the camera image IMG2 and to the right of the camera image IMG3. An additional image (auxiliary image) may be inserted into such blank areas BR.

[0048] FIG. 6 is a diagram illustrating an example of a work image IMG_O in which an additional image has been inserted into a blank area BR. The work image IMG_O shown in FIG. 6 is based on that shown in FIG. 5, with an additional image IMG4 inserted in a position closer to the viewer than the camera image IMG1. The additional image IMG4 is generated, for example, based on data on the internal state of the moving object 1 (speed data, acceleration data, remaining battery data). If internal state data of the moving object 1 is provided separately from the data of the camera image IMG, the additional image IMG4 can be generated. If such an additional image IMG4 is inserted into the blank area BR, it is possible to make effective use of the blank area BR.

[0049] Hereinafter, a remote operation method and a remote operation device according to an embodiment will be described in detail.

[0050] 2.Remote operation system 2-1. Example of moving object configuration Fig. 7 is a block diagram showing a configuration example of the moving body 1 shown in Fig. 1. As shown in Fig. 7, the moving body 1 includes a camera 11, a data processing device 12, a sensor group 13, a communication device 14, and a traveling device 15. The camera 11, the sensor group 13, the communication device 14, the traveling device 15, and the data processing device 12 are connected to each other, for example, by an in-vehicle network (for example, a CAN (Controller Area Network)).

[0051] The camera 11 captures at least the space in front of the moving body 1 to obtain a camera image IMG. The camera 11 for capturing images of the front includes two cameras whose capturing ranges partially overlap. The camera 11 for capturing images of the front typically includes the front center camera, the left front camera, and the right front camera described in FIG. 2. The camera 11 may include a camera (right camera) that captures images of the right and right rear of the moving body 1, a camera (left camera) that captures images of the left and left rear of the moving body 1, and a camera that captures images of the rear of the moving body 1. In this case, a part of the capturing range of the rear camera may overlap with the capturing ranges of the right and left cameras. The camera 11 transmits data of the camera image IMG to the data processing device 12.

[0052] The data processing device 12 is a computer for processing various data acquired by the moving body 1. The data processing device 12 includes at least one processor 16, at least one memory 17, and an interface 18. The processor 16 includes a CPU (Central Processing Unit). The memory 17 is a volatile memory such as a DDR memory, and expands programs used by the processor 16 and temporarily stores various data. Various data acquired by the moving body 1 is stored in the memory 17. This various data includes the data of the camera image IMG described above. The various data also includes data of the operation plan PLN of the moving body 1 and data of the remote command INS. The interface 18 is an interface with external devices such as the camera 11 and the traveling device 15.

[0053] The sensor group 13 includes a status sensor that detects the internal status of the moving object 1. Examples of the status sensor include a speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. The sensor group 13 also includes a position sensor that detects the position and orientation of the moving object 1. Examples of the position sensor include a Global Navigation Satellite System (GNSS) sensor. The sensor group 20 may further include a recognition sensor other than the camera 11. The recognition sensor recognizes (detects) the surrounding environment of the moving object 1 by using radio waves or light. Examples of the recognition sensor include a millimeter wave radar and a laser imaging detection and ranging (LIDAR). The sensor group 13 transmits data of the internal status to the data processing device 12.

[0054] The communication device 14 performs wireless communication with a base station (not shown) of the network 4. Examples of communication standards for this wireless communication include mobile communication standards such as 4G, LTE, and 5G. The connection destination of the communication device 14 includes the remote working apparatus 2. In communication with the remote working apparatus 2, the communication device 14 transmits various data received from the data processing device 12 to the remote working apparatus 2.

[0055] The traveling device 15 accelerates, decelerates, and steers the moving body 1. When the moving body 1 is a vehicle, the traveling device 15 includes, for example, a motor, a steering device, and a brake device. The motor drives the wheels of the vehicle. The steering device steers the wheels. The brake device applies a braking force to the vehicle. The vehicle is accelerated by controlling the motor. The vehicle is decelerated by controlling the brake device. The vehicle may be braked using regenerative braking controlled by the motor. The vehicle is steered by controlling the steering device.

[0056] 2-2. PLN Operation Plan Here, the driving plan PLN stored in the memory 17 will be described. In the embodiment, it is considered that a vehicle as a moving body 1 performs automatic driving. Automatic driving control is performed by the data processing device 12 based on the driving plan PLN. The driving plan PLN is generated based on a plurality of events that are executed sequentially in a section where automatic driving is planned, for example. The plurality of events include, for example, an acceleration event, a deceleration event, a lane keeping event, a lane change event, and the like. An acceleration event is an event that accelerates the vehicle. A deceleration event is an event that decelerates the vehicle. A lane keeping event is an event that causes the vehicle to travel so as not to deviate from the travel lane. A lane change event is an event that changes the travel lane.

[0057] Fig. 8 is a diagram showing a configuration example of data of the operation plan PLN. In the example shown in Fig. 8, the data of the operation plan PLN includes data of the vehicle travel trajectory TRJ, data of the operation decision DEC, data of the work content REM in the remote operation, and data of the internal state INT of the vehicle.

[0058] The travel trajectory TRJ is generated in response to a plurality of events. The automatic driving of the vehicle is performed by controlling the traveling device 15 so that the vehicle follows the travel trajectory TRJ. The driving decision DEC indicates a decision regarding the execution of vehicle control (i.e., automatic driving control) for making the vehicle follow the travel trajectory TRJ. The travel trajectory TRJ and the driving decision DEC are appropriately changed based on the driving environment during the automatic driving control.

[0059] As a first example, consider a case where an automatic driving control is performed in which a vehicle follows a driving trajectory TRJ generated in response to a lane keeping event. An example of the driving decision DEC in this case is "maintaining the driving lane". When an obstacle that may hinder the execution of the automatic driving control is recognized during a lane keeping event, the future behavior of the obstacle is predicted. If the obstacle is a dynamic object, the trajectory of the obstacle may be predicted based on the future behavior. Once the future behavior is predicted, a new driving decision DEC is generated based on the future behavior. Examples of the new driving decision DEC include offset avoidance, temporary stop in front of the obstacle, and request for remote work. The driving trajectory TRJ is appropriately changed based on the new driving decision DEC.

[0060] As a second example, consider a case where automatic driving control is performed to make the vehicle follow the travel trajectory TRJ generated in response to an acceleration event. In this case, the driving decision DEC is exemplified as "accelerating the vehicle". If the recognition level of an obstacle recognized during an acceleration event is low, a new driving decision DEC is generated. Examples of the new driving decision DEC include stopping acceleration, pausing before an obstacle, and requesting remote work. Stopping acceleration and pausing before an obstacle are driving decisions for increasing the recognition level. Note that cases where the recognition level of an obstacle is low can also occur during events other than an acceleration event. The driving trajectory TRJ is appropriately changed based on the new driving decision DEC.

[0061] As a third example, consider a case where automatic driving control is performed to make a vehicle follow a driving trajectory TRJ generated in response to a lane change (merging) event. In this case, a driving decision DEC is exemplified by "lane change". When geographical conditions such as entering an intersection with poor visibility or merging into a main road with heavy traffic, or time-of-day conditions such as dawn or dusk are satisfied, a new driving decision is generated. In such a case, from the viewpoint of ensuring traffic safety, a new driving decision DEC is generated regardless of whether or not the vehicle recognizes an obstacle. Examples of the new driving decision DEC are setting an upper limit speed and requesting remote work. The driving trajectory TRJ is appropriately changed based on the new driving decision DEC.

[0062] The work content REM is added to the operation plan PLN when a "request for remote operation" is generated as a new operation decision DEC. The work content REM is generated according to the cause of the generation of the new operation decision DEC. In the above-mentioned first example, an obstacle is recognized. Examples of the work content REM in this case include "checking for obstacles," "checking the surroundings of the obstacle," "permitting execution of offset avoidance," and "executing remote operation (offset avoidance by remote operation)." Note that these work content REM are set in advance, and at least one of them is selected when adding the work content REM.

[0063] In the second example described above, the recognition level of the obstacle is low. Therefore, the work content REM in this case is exemplified as "recognizing obstacles," "checking the surroundings of the vehicle," and "performing remote driving (manual offset for obstacles)." In the third example described above, the driving decision DEC is generated because the geographical condition or the time zone condition is satisfied. Therefore, the work content REM in this case is exemplified as "checking the surroundings of the vehicle" and "performing remote driving (avoiding offset by remote driving)." Note that in the second and third examples, as in the first example, at least one work content REM is selected when adding a work content REM.

[0064] The internal state INT is information detected by a vehicle state sensor. Examples of the data of the internal state INT include vehicle speed data, acceleration data, and battery remaining amount data. Note that the data of the internal state INT does not have to be included in the data of the operation plan PLN.

[0065] 2-3. Example of functional configuration of data processing device 12 Fig. 9 is a block diagram showing an example of a functional configuration of the data processing device 12 shown in Fig. 7. As shown in Fig. 9, the data processing device 12 has a sensor processing unit 121, an object recognition unit 122, a behavior prediction unit 123, a driving plan calculation unit 124, a driving route planning unit 125, a mobile object control unit 126, a driving plan transmission unit 127, an image transmission unit 128, and a remote command receiving unit 129. Note that the functions of these blocks are realized by the processor 16 executing various programs stored in the memory 17.

[0066] The sensor processing unit 121 processes data acquired by the camera 11 and the sensor group 13. The sensor processing unit 121 extracts data related to objects around the moving body 1 through this data processing. Examples of objects to be extracted include static objects and dynamic objects. Examples of static objects include traffic facilities such as traffic lights, guardrails, convex mirrors, and road signs. Examples of dynamic objects include pedestrians, bicycles, motorcycles, and vehicles other than the moving body 1. The extracted data is transmitted to the object recognition unit 122.

[0067] The object recognition unit 122 performs a process of integrating data received from the sensor processing unit 121. The object recognition unit 122 identifies a specific type of object around the moving object 1 through this integration process. The object recognition unit 122 also calculates relative data (relative position and relative speed) of the identified object with respect to the moving object 1. For example, the relative position and relative speed of the identified object are calculated by analyzing the camera image IMG. In addition, the relative position and relative speed of the identified object are calculated by analyzing data from a recognition sensor other than the camera 11. The data of the identified object is transmitted to the behavior prediction unit 123 and the driving plan calculation unit 124.

[0068] The behavior prediction unit 123 predicts the future behavior of the identified object based on the data received from the object recognition unit 122. If the identified object is a dynamic object, the trajectory of the dynamic object may be predicted. The trajectory of the dynamic object is predicted based on, for example, time-series relative data (relative position and relative speed) of the identified object.

[0069] The driving plan calculation unit 124 calculates the driving plan PLN. The driving plan calculation unit 124 sets various events such as an acceleration event based on route data (e.g., data on a section where automatic driving is planned) received from the driving route planning unit 125, and calculates the driving plan PLN. The driving plan calculation unit 124 also changes the driving plan PLN calculated based on various events based on data received from the object recognition unit 122 (mainly data on static objects) and data received from the behavior prediction unit 123 (mainly data related to the behavior of dynamic objects). The data of the driving plan PLN is transmitted to the mobile object control unit 126 and the driving plan transmission unit 127.

[0070] Furthermore, when the operation plan calculation unit 124 receives a remote command INS from the remote command receiving unit 129, the operation plan calculation unit 124 changes the operation plan PLN based on the remote command INS. As described above, remote work includes remote driving, remote instruction, and remote assistance. When the remote command INS includes data for remote driving, the operation plan calculation unit 124 changes the operation plan PLN based on this data. When the remote command INS includes data for remote assistance, the operation plan calculation unit 124 resumes the operation plan PLN that was being executed before transmitting a request signal for the remote work based on this data. Or, the operation plan calculation unit 124 changes the operation plan PLN based on data for remote assistance.

[0071] The travel route planning unit 125 plans a travel route for the moving body 1. The travel route is planned based on data on the start and end positions of a section where autonomous driving is planned. There are no particular limitations on the method for planning the travel route, and any known method may be used.

[0072] The moving object control unit 126 controls the traveling device 15 so that the moving object 1 follows the traveling trajectory TRJ included in the operation plan PLN received from the operation plan calculation unit 124. For example, the moving object control unit 126 calculates the deviation between the traveling trajectory TRJ and the moving object 1. Examples of the deviation include a lateral deviation, a yaw angle deviation (azimuth angle deviation), and a speed deviation. Then, the moving object control unit 126 calculates the operation amount of the traveling device 15 so as to reduce this deviation.

[0073] The operation plan transmission unit 127 outputs the data of the operation plan PLN received from the operation plan calculation unit 124 to the communication device 14. As described above, when a "request for remote work" is generated as a new operation decision DEC, a work content REM is generated according to the cause of this operation decision DEC. When this work content REM is generated or after the work content REM is generated, a request signal for remote work is generated. The work content REM and the request signal are generated, for example, in the operation plan calculation unit 124. The data of the operation plan PLN is output to the communication device 14 simultaneously with the output of the request signal to the communication device 14 or after this output.

[0074] The image transmission unit 128 encodes the camera image IMG data and outputs it to the communication device 14. The camera image IMG may be compressed during the encoding process. The camera image IMG data may be output in conjunction with the output of a remote operation request signal to the communication device 14, or may be output independently of the output of the request signal. In other words, the camera image IMG data may be output together with the output of the request signal, or may be output regardless of the output of the request signal. Furthermore, the camera image IMG data may be transmitted to the remote operation device 2 regardless of the generation of the request signal.

[0075] 2-4. Example of remote operation device configuration Fig. 10 is a block diagram showing an example of the configuration of the remote operating device 2 shown in Fig. 1. As shown in Fig. 10, the remote operating device 2 includes a display 21, a data processing device 22, an input device 23, and a communication device 24. The data processing device 22, the display 21, the input device 23, and the communication device 24 are connected via a dedicated network.

[0076] The display 21 is a display device on which the working image IMG_O is output. The display 21 includes n displays (n≦m−1, where m is the total number of cameras 11). The total number of displays 21 is at least 1. In order to separately output the camera images in front and behind the moving body 1, it is desirable that the total number of displays 21 is at least 2.

[0077] The data processing device 22 is a computer for processing various data. The data processing device 22 includes at least one processor 25, at least one memory 26, and an interface 27. The processor 25 includes a CPU. The memory 26 expands programs used by the processor 25 and temporarily stores various data. Input signals from the input device 23 and various data acquired by the remote operation device 2 are stored in the memory 26. The various data include data of the remote command INS. The various data also include data of the camera image IMG and data of the operation plan PLN. The interface 27 is an interface with external devices such as the input device 23.

[0078] The processor 25 performs "image processing" to generate data for a work image IMG_O based on data for the camera image IMG. When data for the operation plan PLN includes data for work content REM, the processor 25 generates data for a work image IMG_O based on the data for the camera image IMG and the data for the work content REM. The processor 25 also performs "display control processing" to output data for the generated work image IMG_O to the display 21 via the interface 27.

[0079] The input device 23 is a device operated by an operator OP. The input device 23 includes, for example, an input unit that accepts input by the operator OP, and a control circuit that generates and outputs data of a remote command INS based on the input. Examples of the input unit include a touch panel, a mouse, a keyboard, a button, and a switch. Examples of input by the operator OP include a movement operation of a cursor output on the display 21, and a selection operation of a button output on the display 21. The input by the operator OP also includes a change operation of the coupling position CP.

[0080] When the moving body 1 is remotely driven, the input device 23 may include an input device for driving. Examples of the input device for driving include a steering wheel, a shift lever, an accelerator pedal, and a brake pedal.

[0081] The communication device 24 performs wireless communication with a base station of the network 4. Examples of communication standards for this wireless communication include mobile communication standards such as 4G, LTE, and 5G. The communication destination of the communication device 24 includes the mobile body 1. In communication with the mobile body 1, the communication device 24 transmits various data received from the data processing device 22 to the mobile body 1.

[0082] 2-5. Example of functional configuration of data processing device 22 Fig. 11 is a block diagram showing an example of a functional configuration of the data processing device 22 shown in Fig. 10. As shown in Fig. 11, the data processing device 22 includes an operation plan receiving unit 221, an image receiving unit 222, an image processing unit 223, a display control unit 224, an OP input receiving unit 225, and a remote command transmitting unit 226. The functions of these blocks are realized by the processor 25 executing various programs stored in the memory 26.

[0083] The operation plan receiving unit 221 receives data of the operation plan PLN acquired by the communication device 24. The operation plan receiving unit 221 transmits the data of the operation plan PLN to the image processing unit 223.

[0084] The image receiving unit 222 decodes the data of the m types of camera images IMG acquired by the communication device 24. If the data of the m types of camera images IMG is compressed, the image receiving unit 222 expands the data during the decoding process. The image receiving unit 222 transmits the data of the m types of camera images IMG after the decoding process to the image processing unit 223.

[0085] The image processing unit 223 performs image processing. In the image processing, data for a work image IMG_O is generated based on data of m types of camera images IMG received from the image receiving unit 222. The total number of work images IMG_O generated corresponds to the total number of displays 21. When data of work content REM is received from the operation plan receiving unit 221, in the image processing, data for a work image IMG_O is generated based on this data and the data of the camera images IMG.

[0086] As described above, the data of the work content REM is expressed by a binary number indicating the work content REM. In the embodiment, the correspondence between the binary number indicating the work content REM and the coupling position CP is set in advance. In addition, three stages of positions (front side, back side, and middle) are prepared as the coupling position CP. The number of stages of the coupling position CP is not limited to this example, and may be two stages (for example, front side and back side), or may be four or more stages.

[0087] Since the shooting range of the camera 11 is known, two types of camera images IMG whose shooting ranges partially overlap can be easily identified. In the image processing, a joining position CP is set based on the correspondence between a binary number indicating the work content REM and the joining position CP. Then, the two types of camera images IMG are joined vertically at this set position.

[0088] When two camera images IMG are combined vertically in the image processing, the camera images IMG are also combined horizontally in parallel. The horizontal combination is always performed at the same position regardless of the vertical combination position CP. In other words, the horizontal combination is performed at a fixed position regardless of the work content REM.

[0089] When data of the internal state INT is received from the operation plan receiving unit 221, in the image processing, data of an additional image (additional image IMG4) may be generated based on this data. In addition, in the image processing, data of a working image IMG_O may be generated based on the data of this additional image and the data of the camera image IMG after the combining process. The data of the additional image is inserted, for example, into the position of the blank area formed by the combining process.

[0090] The display control unit 224 performs a display control process. The display control process is performed based on the data of the work image IMG_O generated by the image processing process. The display control unit 224 also controls the display contents of the display 21 based on an input signal acquired by the OP input receiving unit 225. In controlling the display contents based on the input signal, for example, the display contents are enlarged or reduced based on the input signal, or the coupling position CP is switched based on a change signal for this coupling position CP. In another example, a cursor output on the display 21 is moved based on the input signal, or a button output on the display 21 is selected.

[0091] The OP input receiving unit 225 receives an input signal from the input device 23. If the input signal is related to control of the display content of the display 21, the input signal is transmitted to the display control unit 224. The OP input receiving unit 225 also receives data of a remote command INS generated in the input device 23. The data of the remote command INS is transmitted to the remote command transmitting unit 226.

[0092] The remote command transmitting unit 226 transmits the data of the remote command INS received from the OP input receiving unit 225 to the communication device 24 .

[0093] 2-6. First processing example by data processing device 22 Fig. 12 is a flowchart showing an example of processing executed by the data processing device 22 (processor 25) shown in Fig. 10. The routine shown in Fig. 12 is repeatedly executed at a predetermined control period, for example, when the processor 25 receives a request signal for remote work.

[0094] In the routine shown in Fig. 12, first, data of an operation plan PLN and a camera image IMG is acquired (step S11). The data of the operation plan PLN and the camera image IMG have time stamp data. In the process of step S11, the data of the operation plan PLN and the camera image IMG are associated with each other based on the time stamp data.

[0095] Following the processing of step S11, it is determined whether or not data on work content REM is included in the data on the operation plan PLN (step S12). As described above, when a "remote work request" is generated as a new operation decision DEC, the work content REM is generated according to the cause of this operation decision DEC. Therefore, the determination result of step S12 is usually positive. However, there may be cases where the cause of the operation decision DEC is not identified and the work content REM is not generated. In such exceptional cases, the processing of step S16 is performed.

[0096] If the determination result of step S12 is positive, it is determined whether or not the coupling position CP is on the front side (step S13). As described above, the correspondence between the binary number indicating the work content REM and the coupling position CP is set in advance. Therefore, in the process of step S13, a determination is made based on this correspondence and the binary number indicating the work content REM acquired in step S11. If the determination result of step S13 is positive, the process of step S14 is performed. If not, the process of step S15 is performed.

[0097] In the process of step S14, two types of camera images IMG are combined at the set position on the front side, thereby generating a work image IMG_O. In the process of step S15, two types of camera images 1MG are combined at the set position on the back side, thereby generating a work image IMG_O. In the process of step S16, two types of camera images IMG are combined at the set position in the middle, thereby generating a work image IMG_O.

[0098] Following the processing of step S14, S15, or S16, a display control processing is performed (step S17). The display control processing is performed based on the data of the work image IMG_O generated in step S14, S15, or S16.

[0099] 2-7. Second Processing Example by Data Processing Device 22 Fig. 13 is a flowchart showing another example of the process executed by the data processing device 22 (processor 25) shown in Fig. 10. The routine shown in Fig. 13 is repeatedly executed at a predetermined control period when the processor 25 receives a request signal for a remote operation, similar to the routine shown in Fig. 12.

[0100] In the routine shown in Fig. 13, following the process of step S11, it is determined whether or not there is an input signal (a signal to change the binding position CP) from the operator OP (step S21). If the result of the determination in step S21 is negative, the processes from step S12 onwards are carried out. The contents of the process of step S11 and the contents of the process from step S12 onwards are as described in Fig. 12.

[0101] If the determination result in step S21 is positive, the combining position CP is adjusted according to an input signal from the operator OP (step S22). For example, when a working image IMG_O in which two types of camera images 1MG are combined is output from the display 21 at the rear set position, if an input signal is received requesting switching to the intermediate set position, the combining position CP is changed according to this input signal.

[0102] 3.Effects According to the embodiment described above, when two types of camera images IMG with partially overlapping shooting ranges are combined to generate one working image IMG_O, a vertical joining position CP is set based on the data of the work content REM. Therefore, it is possible to generate a working image IMG_O in which the visibility of the image at the joining position CP is ensured while allowing distortion and misalignment of the image at positions other than the joining position CP. Moreover, it does not take a long time to generate this working image IMG_O. Therefore, it is also possible to quickly provide the working image IMG_O to the operator OP.

[0103] Furthermore, according to the embodiment, when there is an input signal (a signal to change the joining position CP) from the operator OP, a working image IMG_O in which the joining position CP has been changed in accordance with this input signal is output from the display 21. Therefore, it is also possible to provide the operator OP performing remote work with a working image IMG_O in which two types of camera images IMG are joined at a position that the operator OP wishes to focus on. [Explanation of symbols]

[0104] 1. Mobile 2 Remote control equipment 11 Camera 12, 22 Data processing device 21 Display CP bond position OP Operator DEC Operation Decision INS remote command INT Internal state IMG, IMG1~IMG3 Camera images IMG_O Working image PLN Operation Plan TRJ running track

Claims

1. A method for performing remote operation on a moving object, comprising: Acquiring data of two types of camera images having a partially overlapping shooting range and data of work content in the remote work; setting a vertical position for combining the two types of camera images based on the data of the work content; moving at least one of the two types of camera images in a vertical direction so that the two types of camera images are joined at a set position in the vertical direction, thereby combining the two types of camera images; outputting a composite image of the two types of camera images from a display of a remote operation device; A remote operation method comprising:

2. 2. The method of claim 1 , determining whether or not an input for adjusting a combining position of the two types of camera images has been received from an input device of the remote operation device while the composite image is being output from the display; adjusting the coupling position based on the adjustment input when it is determined that the adjustment input has been received; outputting the composite image having the adjusted combination position from the display; The remote operation method further comprises:

3. 3. The method according to claim 1 or 2, acquiring data on an internal state of the moving object; generating an auxiliary image of the remote operation based on the data of the internal state; adding the auxiliary image to a blank area of ​​the composite image; outputting the composite image to which the auxiliary image has been added from the display; The remote operation method further comprises:

4. An apparatus for remotely operating a moving object, comprising: A storage device in which various information is stored; A processor for processing the various information; a display on which an image processed by the processor is output; Equipped with The various information includes data of two types of camera images having a partially overlapping shooting range and data of the work content of the remote work, The processor, Based on the data of the work content, a vertical position for combining the two types of camera images is set; moving at least one of the two camera images in a vertical direction so that the two camera images are joined at a set position in the vertical direction, thereby combining the two camera images; A composite image of the two types of camera images is output from the display. A remote working device characterized by:

5. 5. The apparatus of claim 4, The remote operation system further includes an input device operated by an operator performing the remote operation. The processor further comprises: determining whether or not an input for adjusting a combining position of the two types of camera images has been received from the input device while the composite image is being output from the display; If it is determined that the adjustment input has been received, the coupling position is adjusted based on the adjustment input; The composite image with the adjusted combination position is output from the display. A remote working device characterized by:

6. 6. An apparatus according to claim 4 or 5, The various information further includes data on an internal state of the moving object, The processor further comprises: generating an auxiliary image of the remote operation based on the data of the internal state; adding the auxiliary image to a blank area of ​​the composite image; The composite image to which the auxiliary image has been added is output from the display. A remote working device characterized by:

Images