Remote support system and remote support method

Abstract

A remote support system is a system for performing remote support of a target moving body. A remote support system acquires an image captured by an infrastructure camera and captured by a target moving body. The remote support system performs a differentiation process for differentiating a target moving body from other moving bodies in an image on the basis of individual information specific to the target moving body. The remote support system presents an image obtained as a result of the differentiation processing to a remote support who performs remote support of the target moving body.

Description

Technical Field

[0001] The present disclosure relates to a technology for remotely supporting a mobile object using an infrastructure camera. Background Art

[0002] Patent Document 1 discloses a technology for remotely operating a vehicle. Prior art literature Patent Literature

[0003] Patent Document 1: U.S. Patent Application Publication No. 2021 / 0089018 Summary of the Invention Problems to be solved by the invention

[0004] Consider remote support (remote monitoring, remote assistance, or remote driving) for a mobile object. Typically, a camera mounted on the mobile object captures the surroundings of the mobile object and presents these images to the remote supporter. The remote supporter observes the presented images to identify the surroundings of the mobile object and provide remote support for the mobile object.

[0005] It's also possible to utilize infrastructure cameras for remote support in addition to cameras mounted on mobile objects. In this case, the infrastructure cameras would capture images of the mobile object and its surroundings, and the images also capturing the mobile object would be presented to the remote supporter. However, it's also conceivable that it would be difficult to identify which object in the image is the mobile object being remotely supported. In such cases, the improved accuracy of remote support achieved by using infrastructure cameras would not be fully realized.

[0006] An object of the present disclosure is to provide a technology capable of further improving the accuracy of remote support of a mobile object using an infrastructure camera. Means for solving problems

[0007] A first aspect relates to a remote support system for performing remote support of a target moving object. The remote support system has one or more processors. One or more processors obtain images captured by infrastructure cameras that capture the moving object. The one or more processors perform differentiation processing for differentiating the target moving object from other moving objects in the image based on individual information unique to the target moving object. The one or more processors present the image obtained as a result of the differentiation processing to a remote supporter who is performing remote support of the subject moving body.

[0008] The second aspect relates to a remote support method executed by a computer and used to perform remote support of a target moving object. Remote support methods include: Obtaining an image captured by an infrastructure camera and capturing a moving object; performing differentiation processing for differentiating the target moving object from other moving objects in the image based on individual information unique to the target moving object; and An image obtained as a result of the differentiation process is presented to a remote supporter who performs remote support of the target moving body. Effects of the Invention

[0009] According to the present disclosure, a target mobile object, which is the subject of remote support, is captured in an image captured by an infrastructure camera. Furthermore, differentiation processing is performed to distinguish the target mobile object from other mobile objects in the image. This differentiation processing makes it easier for the remote supporter to identify the target mobile object in the image. This makes it easier for the remote supporter to provide remote support for the target mobile object. As a result, the accuracy of remote support is improved. Furthermore, the convenience for the remote supporter is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram showing a configuration example of a remote support system according to an embodiment. Figure 2 This is a conceptual diagram for explaining processing related to differentiation processing in the remote support system according to the embodiment. Figure 3 This is a flowchart illustrating processing related to differentiation processing in the remote support system according to the embodiment. Figure 4 This is a block diagram illustrating a functional configuration example related to the first differentiation process based on image processing according to the embodiment. Figure 5 This is a conceptual diagram for explaining an example of image processing according to the embodiment. Figure 6 This is a conceptual diagram for explaining another example of image processing according to the embodiment. Figure 7 This is a conceptual diagram for explaining another example of image processing according to the embodiment. Figure 8 This is a conceptual diagram for explaining another example of image processing according to the embodiment. Figure 9 This is a conceptual diagram for explaining an example of the second differentiation process based on vehicle control according to the embodiment. Figure 10 This is a block diagram illustrating an example of a functional configuration related to a second differentiation process based on vehicle control according to the embodiment. Figure 11This is a block diagram illustrating a configuration example of a vehicle according to the embodiment. Figure 12 This is a block diagram showing a configuration example of a remote support terminal according to an embodiment. Figure 13 This is a block diagram showing a configuration example of a management device according to an embodiment. DETAILED DESCRIPTION

[0011] Embodiments of the present disclosure are described with reference to the accompanying drawings.

[0012] 1. Remote support system Consider remote support for mobile bodies. Remote support is a concept that includes remote monitoring, remote assistance, and remote driving. Examples of mobile bodies include vehicles and robots. Vehicles can be self-driving vehicles or vehicles driven by a driver. Examples of robots include logistics robots and the like. As an example, in the following description, consider a case where the mobile body that is the object of remote support is a vehicle. In general terms, "vehicle" in the following description is replaced with "mobile body."

[0013] Figure 1 This is a schematic diagram illustrating an example configuration of a remote support system 1 according to this embodiment. Remote support system 1 includes a vehicle 100, a remote support terminal 200, and a management device 300. Vehicle 100 is the target of remote support by remote supporter X. Remote support terminal 200 is a terminal device operated by remote supporter X when providing remote support to vehicle 100. Remote support terminal 200 can also be referred to as a remote cockpit. Management device 300 manages remote support system 1. Typically, management device 300 is a management server on the cloud. Management device 300 may also be composed of multiple servers performing distributed processing.

[0014] The vehicle 100, the remote support terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote support terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote support terminal 200 can communicate directly without intermediary of the management device 300.

[0015] The driver or the autonomous driving system of the vehicle 100 requests remote support as needed. For example, when the vehicle 100 falls into a situation where autonomous driving is difficult, the autonomous driving system requests remote support. The vehicle 100 sends the remote support request to the management device 300. The remote support request is sometimes a remote assistance request (RFI: Request for Information) and sometimes a remote driving request (RFO: Request for Operation). The management device 300 responds to the remote support request and assigns a remote supporter X to the vehicle 100 that is the target of remote support from multiple candidates. The management device 300 manages the allocation relationship between the vehicle 100 and the remote supporter X, and also provides information on the allocation relationship to the vehicle 100 and the remote support terminal 200. The vehicle 100 and the remote support terminal 200 establish communication based on the information on the allocation relationship. After the communication is established, the vehicle 100 and the remote support terminal 200 can also communicate directly without going through the management device 300.

[0016] The vehicle 100 is equipped with various sensors, including an onboard camera C. The onboard camera C captures images of the surroundings of the vehicle 100 and acquires images showing the surrounding conditions of the vehicle 100. Vehicle information VCL is information obtained by the various sensors, including images obtained by the onboard camera C. The vehicle 100 transmits the vehicle information VCL to the remote support terminal 200.

[0017] The remote support terminal 200 receives vehicle information VCL transmitted from the vehicle 100. The remote support terminal 200 presents the vehicle information VCL to the remote supporter X. Specifically, the remote support terminal 200 includes a display device that displays images and the like. The remote supporter X views the displayed information, identifies the surrounding conditions of the vehicle 100, and performs remote support for the vehicle 100. The remote support information SUP is information related to the remote support performed by the remote supporter X. For example, the remote support information SUP includes instructions or operation steps input by the remote supporter X. The remote support terminal 200 transmits the remote support information SUP to the vehicle 100 as needed.

[0018] The vehicle 100 receives the remote support information SUP transmitted from the remote support terminal 200. The vehicle 100 performs vehicle travel control according to the received remote support information SUP.

[0019] In this embodiment, the remote support system 1 further includes one or more infrastructure cameras 400. The infrastructure cameras 400 are installed in areas where the vehicle 100 travels. In particular, the infrastructure cameras 400 are located in positions capable of capturing images of the vehicle 100. The infrastructure cameras 400 capture images of the vehicle 100 and its surroundings, acquiring images IMG representing the conditions of the vehicle 100 and its surroundings. In other words, the infrastructure cameras 400 also capture images IMG of the vehicle 100, which is the target of remote support.

[0020] The management device 300 communicates with the infrastructure camera 400 to collect and manage the image IMG captured by the infrastructure camera 400. In addition, the management device 300 provides the image IMG captured by the infrastructure camera 400 to the remote support terminal 200.

[0021] In addition to acquiring images captured by the vehicle-mounted camera C, the remote support terminal 200 also acquires images IMG captured by the infrastructure camera 400 and presents these images to the remote supporter X. The images IMG captured by the infrastructure camera 400 also include the vehicle 100 itself, the target of remote support. By also presenting these images IMG to the remote supporter X, it is expected that the accuracy of remote support and the convenience for the remote supporter X will be further improved.

[0022] As an application example, consider a scenario where an autonomous vehicle automatically drives on a factory site. For example, an autonomous vehicle assembled in an assembly plant automatically drives from the assembly plant to the yard. One or more infrastructure cameras 400 are installed on the road from the assembly plant to the yard. By using this infrastructure camera 400, the autonomous vehicle driving automatically can be remotely monitored. In addition, if the autonomous vehicle falls into a situation where it is difficult to drive automatically, the autonomous vehicle can also be remotely driven by using the infrastructure camera 400. In addition, if the autonomous vehicle falls into a situation where it is difficult to drive automatically, it can also be considered to dispatch employees to the site to take over by manual driving, but this is time-consuming and labor-intensive. Remote driving using the infrastructure camera 400 is more convenient and can also save time and labor.

[0023] 2. Overview of Differentiated Processing For convenience, the vehicle 100 that is the target of remote support will be referred to as a “target vehicle 100T” hereinafter. The assignment relationship between the remote supporter X and the target vehicle 100T is shared by the target vehicle 100T, the remote support terminal 200 , and the management device 300 .

[0024] As described above, the target vehicle 100T is also captured in the image IMG captured by the infrastructure camera 400. In this case, it is conceivable that it would be difficult to identify which object in the image IMG is the target vehicle 100T. For example, if multiple vehicles 100, including the target vehicle 100T, are captured in the same image IMG, it might be difficult for the remote supporter X to immediately identify which one is the target vehicle 100T assigned to them. In such a situation, the improved accuracy and convenience achieved by the infrastructure camera 400 cannot be fully realized.

[0025] Therefore, the present disclosure proposes a technology that can further improve the accuracy of remote support using infrastructure cameras 400. To this end, the remote support system 1 according to this embodiment is configured to differentiate the target vehicle 100T from other vehicles in the image IMG captured by the infrastructure camera 400. Hereinafter, the process of distinguishing the target vehicle 100T from other vehicles in the image IMG will be referred to as "differentiation processing."

[0026] Figure 2 1 is a conceptual diagram for explaining processing related to differentiation processing in the remote support system 1. The remote support system 1 includes a differentiation processing unit 10 and an image presentation unit 20.

[0027] The differentiation processing unit 10 obtains an image IMG captured by the infrastructure camera 400. In this image IMG, at least the target vehicle 100T, which is the target of remote support, is captured. In addition, the differentiation processing unit 10 obtains individual information SPC unique to the target vehicle 100T. For example, the individual information SPC includes position information indicating the actual position of the target vehicle 100T. As another example, the individual information SPC may also include feature quantity information indicating the feature quantity of the target vehicle 100T. Examples of feature quantities include color, shape, and lighting patterns of lighting devices. Typically, such individual information SPC is included in the vehicle information VCL transmitted from the target vehicle 100T. However, the individual information SPC may also be generated separately. Based on the individual information SPC, the differentiation processing unit 10 performs differentiation processing to differentiate the target vehicle 100T in the image IMG from other vehicles.

[0028] For example, the differentiation processing unit 10 processes (processes) the image IMG to differentiate the target vehicle 100T from other vehicles in the image IMG. Image IMG-S is the image IMG obtained through image processing. This differentiation processing (first differentiation processing) based on image processing will be described in detail in Section 3 below.

[0029] As another example, the differentiation processing unit 10 may differentiate the target vehicle 100T in the image IMG from other vehicles by controlling the actual target vehicle 100T. Such differentiation processing (second differentiation processing) based on vehicle control will be described in detail in Section 4 below.

[0030] The image presentation unit 20 receives the image IMG or the processed image IMG-S from the differential processing unit 10. The image presentation unit 20 is included in the remote support terminal 200 and presents the image IMG or the processed image IMG-S to the remote supporter X. More specifically, the image presentation unit 20 displays the image IMG or the processed image IMG-S on a display device.

[0031] Furthermore, as long as the image IMG and individual information SPC can be obtained, the location of the differential processing unit 10 is not limited. As described above, the vehicle 100, the remote support terminal 200, and the management device 300 can communicate with each other via a communication network. In other words, information such as allocation relationship information, image IMG, individual information SPC, and vehicle information VCL can be shared by the vehicle 100, the remote support terminal 200, and the management device 300. Therefore, the differential processing unit 10 can be included in any one of the vehicle 100, the remote support terminal 200, and the management device 300. The differential processing unit 10 can also be dispersed in two or more of the vehicle 100, the remote support terminal 200, and the management device 300.

[0032] In general, the differentiation processing unit 10 and the image presentation unit 20 are implemented by one or more processors and one or more storage devices. The one or more processors perform various information processing. The one or more storage devices store various information required for the processing performed by the one or more processors.

[0033] Figure 3 This is a flowchart illustrating the processing associated with differentiation processing in the remote support system 1. In step S1, one or more processors obtain an image IMG captured by an infrastructure camera 400. The target vehicle 100T, the subject of remote support, is captured in image IMG. In step S2, the one or more processors obtain individual information SPC unique to the target vehicle 100T. In step S3, the one or more processors perform differentiation processing based on the individual information SPC to differentiate the target vehicle 100T from other vehicles in image IMG. In step S4, the one or more processors present the image IMG or image IMG-S obtained as a result of the differentiation processing to the remote supporter X.

[0034] As described above, according to this embodiment, the target vehicle 100T, which is the subject of remote support, is captured in the image IMG captured by the infrastructure camera 400. Furthermore, differentiation processing is performed to distinguish the target vehicle 100T in this image IMG from other vehicles. This differentiation processing makes it easier for remote supporter X to identify the target vehicle 100T in the image IMG. This makes it easier for remote supporter X to provide remote support for the target vehicle 100T. As a result, the accuracy of remote support is improved. Furthermore, the convenience for remote supporter X is enhanced.

[0035] Various examples of differentiation processing according to this embodiment will be described in detail below.

[0036] 3. First differentiation processing based on image processing Figure 4 1 is a block diagram showing an example of a functional configuration related to the first differentiation processing based on image processing. The differentiation processing unit 10 includes a first differentiation processing unit 30. The first differentiation processing unit 30 includes an object identification unit 31 and an image processing unit 32.

[0037] 3-1. Target determination processing The object identification unit 31 acquires individual information SPC unique to the target vehicle 100T. The object identification unit 31 executes "object identification processing" for identifying the target vehicle 100T in the image IMG based on the individual information SPC. Various methods are conceivable as the object identification processing.

[0038] 3-1-1. First Example of Object Determination Processing In the first example, the individual information SPC includes position information indicating the actual position of the target vehicle 100T. Typically, the actual position of the target vehicle 100T is represented by latitude and longitude in an absolute coordinate system. The position information of the target vehicle 100T is included in the vehicle information VCL transmitted from the target vehicle 100T.

[0039] The target identification unit 31 also obtains the camera installation information CAM, which is the installation information of the infrastructure camera 400. More specifically, the camera installation information CAM indicates the installation location, installation orientation, and viewing angle of the infrastructure camera 400. Typically, the installation location of the infrastructure camera 400 is represented by latitude and longitude in an absolute coordinate system. The camera installation information CAM is provided by the infrastructure camera 400 or the management device 300.

[0040] The object identification unit 31 identifies the target vehicle 100T in the image IMG based on the position information of the target vehicle 100T and the camera placement information CAM. More specifically, the camera placement information CAM indicates the area captured by the infrastructure camera 400, i.e., the area captured in the image IMG. Based on this area and the position information of the target vehicle 100T, the displayed position in the image IMG corresponding to the actual position of the target vehicle 100T can be calculated. In the simplest case, the target vehicle 100T is considered to be captured within a certain area surrounding the calculated displayed position.

[0041] Alternatively, the object determination unit 31 may also have an object recognition model based on machine learning. The object recognition model is learned in a manner that can recognize various objects in an image. Typically, the objects recognized by the object recognition model are moving objects. Examples of moving objects include people (pedestrians), vehicles, two-wheeled vehicles, bicycles, etc. The object recognition model may be based on a CNN (Convolutional Neural Network). As another example, the object recognition model may also be based on a Transformer, which is a type of deep learning model.

[0042] The object identification unit 31 detects various objects captured in the image IMG using an object recognition model. The object identification unit 31 can also determine the type of object. As described above, based on the position information of the target vehicle 100T and the camera setup information CAM, the displayed position in the image IMG corresponding to the actual position of the target vehicle 100T is calculated. Among the various objects detected in the image IMG, the object identification unit 31 can identify the object at the displayed position as the target vehicle 100T. Alternatively, the object identification unit 31 can narrow down the image area surrounding the displayed position for object detection.

[0043] The object identification unit 31 may also include a tracker. A tracker is software that automatically tracks the same object in a series of temporally consecutive images IMG based on a tracking algorithm. By using the tracker, the object identification unit 31 can track the target vehicle 100T in the series of temporally consecutive images IMG.

[0044] 3-1-2. Second Example of Object Identification Processing In the second example, the individual information SPC includes feature quantity information representing feature quantities of the target vehicle 100T. Examples of feature quantities include the color of the target vehicle 100T, the shape of the target vehicle 100T, and the lighting pattern of the lighting fixtures mounted on the target vehicle 100T. The lighting pattern of the lighting fixtures may be a pattern unique to the target vehicle 100T. The feature quantity information may be provided by the target vehicle 100T or by the management device 300 that manages each vehicle 100.

[0045] The object identification unit 31 identifies the target vehicle 100T in the image IMG based on the feature quantity information of the target vehicle 100T. For example, the object identification unit 31 includes an object recognition model based on machine learning. The object recognition model is the same as in the first example described above. The object recognition model detects various objects in the image IMG and extracts feature quantities of each detected object. The object identification unit 31 compares the extracted feature quantities of each detected object with the feature quantities of the target vehicle 100T. The object identification unit 31 then identifies an object having extracted feature quantities that are identical or similar to those of the target vehicle 100T as the target vehicle 100T. For example, the object identification unit 31 calculates the similarity (the inverse of the distance) between the extracted feature quantities of each detected object and the feature quantities of the target vehicle 100T in the feature quantity space. The object identification unit 31 then identifies an object whose similarity exceeds a threshold value as the target vehicle 100T.

[0046] Image Processing The image processing unit 32 receives information indicating the result of the object identification process from the object identification unit 31. The information indicating the result of the object identification process includes information regarding the partial image region of the target vehicle 100T identified in the image IMG. When utilizing the aforementioned object recognition model, a bounding box surrounding the identified target vehicle 100T is assigned to the image IMG. The partial image region containing the target vehicle 100T corresponds to the region enclosed by the bounding box. The information indicating the result of the object identification process may also include information regarding the position of the bounding box within the image IMG.

[0047] Based on information about the partial image area capturing the target vehicle 100T, the image processing unit 32 processes the image IMG to distinguish the target vehicle 100T from other vehicles. In other words, the image processing unit 32 distinguishes the target vehicle 100T identified in the image IMG from other vehicles by applying image processing to the image IMG. Image IMG-S is the image processed by the image processing unit 32 relative to the original image IMG. The processed image IMG-S is then presented to the remote supporter X.

[0048] Various examples of image processing can be considered for differentiating the target vehicle 100T. Various examples of image processing will be described below.

[0049] 3-2-1. Highlight (emphasize) display Figure 5 and Figure 6 This is a conceptual diagram for explaining an example of image processing. Figure 5In the example shown, the image processing unit 32 highlights (emphasizes) the target vehicle 100T in the image IMG. For example, the image processing unit 32 creates a bounding box surrounding the target vehicle 100T. The color of the bounding box can be specified by the remote supporter X. As another example, the image processing unit 32 can also create a mark (marker) on the target vehicle 100T. The shape and color of the mark can be specified by the remote supporter X.

[0050] exist Figure 6 In the example shown, the image processing unit 32 increases the brightness or saturation of the partial image region including the target vehicle 100T compared to other image regions. The partial image region including the target vehicle 100T may be the region enclosed by the aforementioned bounding box. For example, the image processing unit 32 processes the partial image region including the target vehicle 100T to brighten it. Alternatively, the image processing unit 32 may process other image regions that do not include the target vehicle 100T to darken them.

[0051] By highlighting (emphasizing) the target vehicle 100T in this manner, the remote supporter X can easily recognize the target vehicle 100T in the image IMG-S.

[0052] 3-2-2. Zoom in (Zoomin) Figure 7 This is a conceptual diagram for explaining another example of image processing. The image processing unit 32 zooms in on the target vehicle 100T in the image IMG, thereby magnifying the target vehicle 100T in the image IMG. This emphasizes the target vehicle 100T, achieving the same effect as when it is highlighted.

[0053] As an added benefit of the zoomed-in lens, the area surrounding the target vehicle 100T is also magnified, allowing remote supporter X to clearly observe the area near the target vehicle 100T. For example, when the target vehicle 100T is entering a garage, it is preferable for remote supporter X to clearly observe the area near the target vehicle 100T. As another example, when the target vehicle 100T is entering a narrow alley, it is preferable for remote supporter X to clearly observe the area near the target vehicle 100T.

[0054] When entering a parking garage or entering a narrow alley, the target vehicle 100T is traveling at a relatively low speed. Furthermore, when the target vehicle 100T is traveling at a relatively low speed, there is no need to observe a position far from the target vehicle 100T. Therefore, a zoom-in operation may be selected when the target vehicle 100T is traveling at a relatively low speed. Specifically, when the target vehicle 100T's speed is below a predetermined threshold, the image processing unit 32 may zoom in on the target vehicle 100T. Furthermore, the target vehicle 100T's speed is obtained based on the vehicle information VCL transmitted from the target vehicle 100T.

[0055] 3-2-3. Assignment of different information Figure 8 This is a conceptual diagram illustrating another example of image processing. Image processing unit 32 assigns different information to target vehicle 100T and other vehicles in image IMG. Specifically, image processing unit 32 assigns first information to target vehicle 100T in image IMG and second information to other vehicles in image IMG. The first and second information are different.

[0056] For example, the image processing unit 32 assigns a first mark (first mark) to the target vehicle 100T as the first information and assigns a second mark (second mark) to the other vehicles as the second information. The first mark and the second mark differ in at least one of color, shape, and size. The color, shape, size, etc. of the first mark can be specified by the remote supporter X.

[0057] This method can also be applied to a situation where the same image IMG captured by the same infrastructure camera 400 is shared by multiple remote supporters X. A first remote supporter X-1 provides remote support for a first target vehicle 100T-1, and a second remote supporter X-2 provides remote support for a second target vehicle 100T-2. The first target vehicle 100T-1 and the second target vehicle 100T-2 are captured in the same image IMG. The image processing unit 32 assigns a first mark (first mark) to the first target vehicle 100T-1 and a second mark (second mark) to the second target vehicle 100T-2. The first mark and the second mark can be specified by the first remote supporter X-1 and the second remote supporter X-2, respectively.

[0058] The first information assigned to the first target vehicle 100T-1 may be identification information of the first remote supporter X-1. Similarly, the second information assigned to the second target vehicle 100T-2 may be identification information of the second remote supporter X-2. Examples of identification information include the ID number, facial image, avatar, and icon of the remote supporter X. The identification information may be specified by the remote supporter X.

[0059] By assigning different information to the target vehicle 100T and other vehicles in this manner, the remote supporter X can easily recognize the target vehicle 100T in the image IMG-S.

[0060] 3-2-4. Combination and Selection Two or more of the above-exemplified image processing methods may be combined.

[0061] 3-3. Effect As described above, the first differentiation processing applies image processing to image IMG, thereby differentiating the target vehicle 100T from other vehicles. This first differentiation processing allows remote supporter X to more easily identify the target vehicle 100T in image IMG-S. This makes it easier for remote supporter X to remotely support the target vehicle 100T. As a result, the accuracy of remote support is improved. Furthermore, the convenience for remote supporter X is enhanced.

[0062] 4. Second differentiated processing based on vehicle control Figure 9 This is a conceptual diagram illustrating an example of a second differentiation process based on vehicle control. For example, by lighting (flashing) the hazard lights of the actual target vehicle 100T, the target vehicle 100T in the image IMG can be differentiated from other vehicles. This allows the remote supporter X to easily identify the target vehicle 100T in the image IMG.

[0063] However, arbitrarily lighting hazard lights on ordinary roads may cause misunderstanding and confusion to other drivers. Therefore, lighting of hazard lights is permitted when the target vehicle 100T is within a specified area. Examples of the specified area include private land, factories, parking lots, and road shoulders.

[0064] Figure 10 1 is a block diagram showing an example of a functional configuration related to the second differentiation processing based on vehicle control. The differentiation processing unit 10 includes a second differentiation processing unit 40. The second differentiation processing unit 40 includes a condition determination unit 41 and a vehicle control unit 42.

[0065] The condition determination unit 41 obtains individual information SPC specific to the target vehicle 100T. The individual information SPC includes position information indicating the actual position of the target vehicle 100T. Typically, the actual position of the target vehicle 100T is represented by latitude and longitude in an absolute coordinate system. The position information of the target vehicle 100T is included in the vehicle information VCL transmitted from the target vehicle 100T.

[0066] Furthermore, the condition determination unit 41 acquires map information MAP. The map information MAP has the positions of predetermined areas registered in advance. Examples of the predetermined areas include private land, factories, parking lots, and roadside.

[0067] The condition determination unit 41 determines whether the target vehicle 100T is present in a predetermined area based on the position information of the target vehicle 100T and the map information MAP. The condition determination unit 41 notifies the vehicle control unit 42 of the determination result.

[0068] The vehicle control unit 42 communicates with the target vehicle 100T and controls the target vehicle 100T. In particular, the vehicle control unit 42 controls the lighting device 140 mounted on the target vehicle 100T. Examples of the lighting device 140 include hazard lights, taillights, and headlights already installed on the target vehicle 100T. Alternatively, the lighting device 140 may be a special light retrofitted to the target vehicle 100T.

[0069] If the target vehicle 100T is within the specified area, the vehicle control unit 42 activates the lighting device 140. This distinguishes the target vehicle 100T from other vehicles in the image IMG. As a result, the remote supporter X can easily identify the target vehicle 100T in the image IMG. On the other hand, if the target vehicle 100T is outside the specified area, the vehicle control unit 42 disables the lighting device 140.

[0070] As a modified example, the permission condition for permitting the operation of the lighting device 140 may also include, in addition to "the target vehicle 100T is outside the specified area," "preparation for remote operation of the target vehicle 100T." When a remote operation request (RFO) is issued from the target vehicle 100T, the management device 300 assigns a remote support person X to the target vehicle 100T. The management device 300 provides information about the assignment relationship to the target vehicle 100T and the remote support terminal 200. The target vehicle 100T and the remote support terminal 200 establish communication based on the assignment relationship information. After communication is established, the remote support person X (remote operator) begins remote operation of the target vehicle 100T. The preparation period for remote operation of the target vehicle 100T refers to the period from the issuance of the remote operation request to the start of remote operation of the target vehicle 100T by the remote support person X. Therefore, the period during which communication is established between the target vehicle 100T and the remote support terminal 200 is included in the preparation period for remote operation. In addition to the location information of the target vehicle 100T, the individual information SPC may also include information indicating the status of the remote driving of the target vehicle 100T. Based on this individual information SPC, the condition determination unit 41 determines whether the permission condition for permitting the operation of the lighting 140 has been met. If the permission condition has been met, the vehicle control unit 42 activates the lighting 140. On the other hand, if the permission condition has not been met, the vehicle control unit 42 prohibits the operation of the lighting 140. Therefore, after communication is established, if the remote supporter X (remote operator) begins remote driving of the target vehicle 100T, the vehicle control unit 42 stops the operation of the lighting 140.

[0071] As described above, the second differentiation processing activates the lighting device 140 mounted on the target vehicle 100T, thereby distinguishing the target vehicle 100T from other vehicles in the image IMG. This second differentiation processing makes it easier for remote supporter X to identify the target vehicle 100T in the image IMG. This makes it easier for remote supporter X to provide remote support for the target vehicle 100T. As a result, the accuracy of remote support is improved. Furthermore, the convenience for remote supporter X is enhanced.

[0072] Furthermore, the second differentiation processing eliminates the need for complex image processing, thereby reducing the processing load on the differentiation processing unit 10 .

[0073] 5. Combination It is also possible to combine the first differentiation processing described in Section 3 with the second differentiation processing described in Section 4. That is, the differentiation processing unit 10 may include both the first differentiation processing unit 30 and the second differentiation processing unit 40.

[0074] 6. Vehicle Example 6-1. Example of configuration Figure 11 1 is a block diagram showing a configuration example of a vehicle 100 . The vehicle 100 includes a communication device 110 , a sensor group 120 , a travel device 130 , a lighting device 140 , and a control device 150 .

[0075] The communication device 110 communicates with the outside of the vehicle 100 . For example, the communication device 110 communicates with the remote support terminal 200 and the management device 300 .

[0076] The sensor group 120 includes an identification sensor, a vehicle state sensor, a position sensor, and the like. The identification sensor identifies (detects) the surrounding conditions of the vehicle 100. Examples of the identification sensor include an onboard camera C, LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, and the like. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System).

[0077] The driving device 130 includes a steering device, a drive device, and a brake device. The steering device steers the wheels. For example, the steering device includes an Electric Power Steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The brake device generates braking force.

[0078] The lighting device 140 includes hazard lights, tail lights, headlights, etc. The lighting device 140 may also include a special light installed on the vehicle 100 .

[0079] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter referred to as the processor 160) and one or more storage devices 170 (hereinafter referred to as the storage device 170). The processor 160 performs various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various information required for the processes performed by the processor 160. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The control device 150 may also include one or more ECUs (Electronic Control Units).

[0080] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 are realized by the processor 160 executing the vehicle control program PROG1. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded in a computer-readable recording medium.

[0081] 6-2. Driving environment information The control device 150 uses the sensor group 120 to obtain driving environment information ENV indicating the driving environment of the vehicle 100 . The driving environment information ENV is stored in the storage device 170 .

[0082] Driving environment information ENV includes surrounding condition information representing the recognition results of the recognition sensor. For example, surrounding condition information includes images captured by the vehicle-mounted camera C. Surrounding condition information may also include object information related to objects surrounding vehicle 100. Examples of objects surrounding vehicle 100 include pedestrians, other vehicles (such as those traveling ahead or parked), white lines, signals, signs, and roadside structures. Object information indicates the relative position and speed of the object relative to vehicle 100.

[0083] In addition, the driving environment information ENV includes vehicle state information indicating the vehicle state detected by the vehicle state sensor.

[0084] Furthermore, the driving environment information ENV includes position information indicating the position and orientation of the vehicle 100. The position information is obtained by a position sensor. High-precision position information can also be obtained by localization processing using map information and surrounding situation information (object information).

[0085] 6-3. Vehicle driving control The control device 150 performs vehicle travel control for controlling the travel of the vehicle 100. The vehicle travel control includes steering control, drive control, and brake control. The control device 150 performs vehicle travel control by controlling the travel device 130 (steering device, drive device, and brake device).

[0086] The control device 150 can perform autonomous driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory for the vehicle 100 to follow the driving plan based on the driving environment information ENV. The target trajectory includes a target position and a target speed. Furthermore, the control device 150 controls vehicle driving so that the vehicle 100 follows the target trajectory.

[0087] 6-4. Processing related to remote support Hereinafter, a case where remote support is performed for the vehicle 100 will be described. The control device 150 communicates with the remote support terminal 200 via the communication device 110 .

[0088] The control device 150 transmits vehicle information VCL to the remote support terminal 200. Vehicle information VCL is information required for remote support by the remote supporter X and includes at least a portion of the aforementioned driving environment information ENV. For example, vehicle information VCL includes information about the surrounding conditions (particularly images). Vehicle information VCL may also include vehicle status information (such as speed).

[0089] The vehicle information VCL may also include individual information SPC specific to the vehicle 100. For example, the individual information SPC may include position information obtained by the aforementioned position sensor. The individual information SPC may also include feature information representing characteristic quantities of the vehicle 100. Examples of such feature quantities include color, shape, and the lighting pattern of the lighting fixture 140. The individual information SPC may also include information indicating the remote driving status of the vehicle 100. The control device 150 transmits the vehicle information VCL including the individual information SPC to the differentiation processing unit 10.

[0090] Furthermore, the control device 150 receives remote support information SUP from the remote support terminal 200. The remote support information SUP is information related to remote support by the remote support X. For example, the remote support information SUP includes the amount of operation by the remote support X. The control device 150 controls vehicle travel according to the received remote support information SUP.

[0091] 7. Example of remote support terminal Figure 122 is a block diagram showing a configuration example of the remote support terminal 200 . The remote support terminal 200 includes a communication device 210 , an output device 220 , an input device 230 , and a control device 250 .

[0092] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0093] The output device 220 outputs various information. For example, the output device 220 includes a display device. The display device displays various information to present the information to the remote supporter X. As another example, the output device 220 may also include a speaker.

[0094] Input device 230 receives input from remote support person X. Examples of input device 230 include a touch panel, buttons, and remote operating components. Remote operating components are components that remote support person X (remote operator) operates when remotely driving vehicle 100. Examples include a steering wheel, an accelerator pedal, a brake pedal, and direction indicators. The remote driving component may also be a touch panel.

[0095] The control device 250 controls the remote support terminal 200. The control device 250 includes one or more processors 260 (hereinafter referred to as processors 260) and one or more storage devices 270 (hereinafter referred to as storage devices 270). The processors 260 execute various processes. For example, the processor 260 includes a CPU. The storage device 270 stores various information required for the processes performed by the processor 260. Examples of the storage device 270 include volatile memory, nonvolatile memory, HDD, SSD, and the like.

[0096] The remote support program PROG2 is a computer program executed by processor 260. The functions of control device 250 are realized by processor 260 executing the remote support program PROG2. The remote support program PROG2 is stored in storage device 270. Alternatively, the remote support program PROG2 may be recorded on a computer-readable recording medium. The remote support program PROG2 may also be provided via a network.

[0097] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives the vehicle information VCL transmitted from the vehicle 100. The control device 250 displays the vehicle information VCL, including an image, on a display device, thereby presenting the vehicle information VCL to the remote supporter X. The remote supporter X can recognize the status of the vehicle 100 and the surrounding conditions based on the vehicle information VCL displayed on the display device.

[0098] As needed, remote support X operates input device 230 to provide remote support for vehicle 100. For example, remote support X issues various instructions (e.g., a start instruction) via input device 230. When remotely operating vehicle 100, remote support X operates the remote driving component. The amount of operation of the remote driving component is detected by sensors installed in the remote driving component. Control device 250 generates remote support information SUP including the instructions or operation amount input by remote support X. Control device 250 then transmits remote support information SUP to vehicle 100 via communication device 210.

[0099] The control device 250 may also have the function of the differential processing unit 10. In this case, the control device 250 obtains the vehicle information VCL including the individual information SPC from the vehicle 100. In addition, the control device 250 obtains the camera setting information CAM related to the infrastructure camera 400 via the management device 300. Moreover, the control device 250 obtains the image IMG captured by the infrastructure camera 400 via the management device 300. The obtained information and image IMG are stored in the storage device 270. The control device 250 performs differential processing based on the obtained information. For example, the control device 250 performs the first differential processing to generate the image IMG-S (see the above-mentioned part 3). As another example, the control device 250 performs the second differential processing by sending a control instruction to the vehicle 100 via the communication device 210 (see the above-mentioned part 4).

[0100] The control device 250 has the function of the image presenting unit 20. That is, the control device 250 presents the image IMG or the processed image IMG-S to the remote supporter X via the output device 220. More specifically, the control device 250 displays the image IMG or the processed image IMG-S on the display device.

[0101] 8. Example of management device Figure 13 3 is a block diagram showing a configuration example of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0102] The communication device 310 communicates with the vehicle 100 , the remote support terminal 200 , and the infrastructure camera 400 .

[0103] The control device 350 controls the management device 300. The control device 350 includes one or more processors 360 (hereinafter referred to as processors 360) and one or more storage devices 370 (hereinafter referred to as storage devices 370). The processors 360 execute various processes. For example, the processor 360 includes a CPU. The storage device 370 stores various information required for the processes executed by the processor 360. Examples of the storage device 370 include volatile memory, nonvolatile memory, HDD, SSD, and the like.

[0104] The management program PROG3 is a computer program executed by the processor 360. The functions of the control device 350 are realized by the execution of the management program PROG3 by the processor 360. The management program PROG3 is stored in the storage device 370. Alternatively, the management program PROG3 may be recorded on a computer-readable recording medium. The management program PROG3 may also be provided via a network.

[0105] The control device 350 communicates with the vehicle 100 and the remote support terminal 200 via the communication device 310. The control device 350 receives vehicle information VCL transmitted from the vehicle 100. Furthermore, the control device 350 transmits the received vehicle information VCL to the remote support terminal 200. Furthermore, the control device 350 receives remote support information SUP transmitted from the remote support terminal 200. Furthermore, the control device 350 transmits the received remote support information SUP to the vehicle 100.

[0106] Furthermore, the control device 350 communicates with the infrastructure camera 400 via the communication device 310 and obtains the image IMG captured by the infrastructure camera 400. The control device 350 provides the image IMG to the remote support terminal 200. Furthermore, the control device 350 may also obtain camera setup information CAM related to the infrastructure camera 400 and provide the camera setup information CAM to the remote support terminal 200.

[0107] The control device 350 may also have the function of the differential processing unit 10. In this case, the control device 350 obtains the vehicle information VCL including the individual information SPC from the vehicle 100. In addition, the control device 350 obtains the image IMG and the camera setting information CAM captured by the infrastructure camera 400. The obtained information and image IMG are stored in the storage device 370. The control device 350 performs differential processing based on the obtained information. For example, the control device 350 performs the first differential processing to generate the image IMG-S (see the above-mentioned part 3). And, the control device 350 provides the image IMG-S to the remote support terminal 200. As another example, the control device 350 performs the second differential processing by sending a control instruction to the vehicle 100 via the communication device 310 (see the above-mentioned part 4). Description of Reference Numerals

[0108] 1…remote support system, 10…differential processing unit, 20…image presentation unit, 30…first differential processing unit, 31…target determination unit, 32…image processing unit, 40…second differential processing unit, 41…condition determination unit, 42…vehicle control unit, 100…vehicle, 100T…target vehicle, 140…illuminator, 200…remote support terminal, 300…management device, 400…infrastructure camera, IMG…image, IMG-S…processed image, SPC…individual information, VCL…vehicle information

Claims

1. A remote support system for remotely supporting a mobile object, wherein: The remote support system has one or more processors. The one or more processors are configured to, Obtain an image of the moving object captured by an infrastructure camera, performing differentiation processing for differentiating the target moving object from other moving objects in the image based on individual information unique to the target moving object, The image obtained as a result of the differentiation process is presented to a remote supporter who performs the remote support of the target moving body.

2. The remote support system according to claim 1, wherein: The differentiated processing includes: determining the target moving object in the image based on the individual information specific to the target moving object; and The target moving object identified in the image is differentiated from the other moving objects by applying image processing to the image.

3. The remote support system according to claim 2, wherein: Processing the image includes highlighting the moving object in the image.

4. The remote support system according to claim 3, wherein: Prominently highlighting the moving object in the image includes at least one of providing a bounding box surrounding the moving object, providing a mark to the moving object, and increasing the brightness or saturation of an image region including the moving object compared to other image regions.

5. The remote support system according to claim 2, wherein: Processing the image includes zooming in on the moving object in the image to magnify the moving object. The remote support system according to claim 2 , wherein: Processing the image includes zooming in on the moving object in the image to magnify the moving object when a speed of the moving object is lower than a threshold.

7. The remote support system according to claim 2, wherein: Processing the image includes assigning first information to the target moving object in the image and assigning second information different from the first information to the other moving objects in the image.

8. The remote support system according to claim 7, wherein: The first information is a first mark, The second information is a second symbol different from the first symbol.

9. The remote support system according to claim 7, wherein: The first information is identification information of the remote supporter who performs the remote support of the target moving body, The second information is identification information of another remote supporter who performs remote support for the other mobile object.

10. The remote support system according to any one of claims 2 to 9, wherein: The individual information specific to the target mobile object includes at least one of position information indicating a position of the target mobile object and feature amount information indicating a feature amount of the target mobile object.

11. The remote support system according to claim 10, wherein: The individual information specific to the target moving object includes the position information indicating the position of the target moving object, Determining the target moving object in the image includes determining the target moving object in the image based on the position information of the target moving object and setting information of the infrastructure camera.

12. The remote support system according to claim 10, wherein: The individual information specific to the target moving object includes the feature amount information indicating the feature amount of the target moving object, Determining the target moving object in the image includes determining the target moving object in the image based on the feature amount of the target moving object.

13. The remote support system according to claim 12, wherein: The feature amount of the target moving object includes at least one of a color of the target moving object, a shape of the target moving object, and a lighting pattern of an illuminator mounted on the target moving object.

14. The remote support system according to claim 1, wherein: The individual information specific to the target moving object includes position information indicating the position of the target moving object. The differentiated processing includes: determining whether the target moving object exists in a prescribed area based on the position information of the target moving object; and When the target moving object exists in the predetermined area, the target moving object is distinguished from the other moving objects in the image by operating an illumination device mounted on the target moving object.

15. The remote support system according to claim 1, wherein: The remote support includes remote driving of the target mobile body by the remote supporter, The individual information specific to the target mobile object includes position information indicating the position of the target mobile object and information indicating the state of the remote driving of the target mobile object. The differentiated processing includes: determining whether the target moving object exists in a prescribed area based on the position information of the target moving object; and When the target moving object is present in the predetermined area and the remote driving of the target moving object is in preparation, the target moving object is distinguished from the other moving objects in the image by operating a lighting device mounted on the target moving object.

16. The remote support system according to claim 15, wherein: The differentiation process further includes stopping the operation of the lighting device after the remote supporter starts the remote driving of the target moving object.

17. The remote support system according to any one of claims 14 to 16, wherein: The prescribed area is any one of private land, a factory, a parking lot, and a road shoulder.

18. A remote support method, executed by a computer, for performing remote support of a moving object, wherein: This remote support method includes: Obtain an image of the moving object captured by an infrastructure camera, performing differentiation processing for differentiating the target moving object from other moving objects in the image based on individual information specific to the target moving object; and The image obtained as a result of the differentiation process is presented to a remote supporter who performs the remote support of the target moving body.

Images