Remote support system and remote support method

Abstract

To further improve the accuracy of remote support for mobile objects using infrastructure cameras. [Solution] The remote support system is a system for providing remote support to a target mobile object. The remote support system acquires an image of the target mobile object. The remote support system performs a differentiation process to differentiate the target mobile object in the image from other mobile objects. The remote support system then presents the image obtained as a result of the differentiation process to a remote supporter who provides remote support to the target mobile object.

Description

Technical Field

[0001] The present disclosure relates to a technology for remotely supporting a moving object using an infrastructure camera.

Background Art

[0002] Patent Document 1 discloses a technology for remotely operating a vehicle.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Consider the remote support (remote monitoring, remote assistance, remote driving) of a moving object by a remote supporter. Generally, the situation around the moving object is photographed by a camera mounted on the moving object, and the image is presented to the remote supporter. The remote supporter recognizes the situation around the moving object by looking at the presented image and performs remote support for the moving object.

[0005] In addition to the camera mounted on the moving object, it is also conceivable to use an infrastructure camera for remote support. In this case, the moving object and its surroundings are photographed by the infrastructure camera, and an image in which the moving object is also shown is presented to the remote supporter. However, a situation where it is difficult to identify which of the images is the moving object that is the target of remote support is also conceivable. In such a situation, the effect of improving the accuracy of remote support by using an infrastructure camera cannot be sufficiently obtained.

[0006] One object of the present disclosure is to provide a technology that can further improve the accuracy of remote support for a moving object using an infrastructure camera.

Means for Solving the Problems

[0007] The first aspect relates to remote support systems for providing remote support to target mobile objects. The remote support system comprises one or more processors. One or more processors acquire an image containing the target moving object. One or more processors perform differentiation processing to distinguish the target moving object in the image from other moving objects. One or more processors present the image obtained as a result of the differentiation process to a remote supporter who provides remote support for the target moving object.

[0008] The second aspect relates to remote support methods performed by computers for providing remote support to a target mobile object. Remote support methods are: To obtain an image showing the target moving object, This involves performing a differentiation process to distinguish the target moving object in the image from other moving objects, The image obtained as a result of the differentiation process will be presented to the remote supporter who provides remote support for the target moving object. Includes. [Effects of the Invention]

[0009] According to this disclosure, the image contains a target moving object that is the subject of remote support. A differentiation process is then performed to distinguish the target moving object in the image from other moving objects. This differentiation process makes it easier for the remote supporter to identify the target moving object in the image. In other words, it becomes easier for the remote supporter to provide remote support to the target moving object. As a result, the accuracy of remote support is improved. Furthermore, convenience for the remote supporter is improved. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing an example configuration of a remote support system according to an embodiment. [Figure 2]It is a conceptual diagram for explaining a process related to differentiation processing in a remote support system according to an embodiment. [Figure 3] It is a flowchart showing a process related to differentiation processing in a remote support system according to an embodiment. [Figure 4] It is a block diagram showing a functional configuration example related to a first differentiation process based on image processing according to an embodiment. [Figure 5] It is a conceptual diagram for explaining an example of image processing according to an embodiment. [Figure 6] It is a conceptual diagram for explaining another example of image processing according to an embodiment. [Figure 7] It is a conceptual diagram for explaining still another example of image processing according to an embodiment. [Figure 8] It is a conceptual diagram for explaining still another example of image processing according to an embodiment. [Figure 9] It is a conceptual diagram for explaining an example of a second differentiation process based on vehicle control according to an embodiment. [Figure 10] It is a block diagram showing a functional configuration example related to a second differentiation process based on vehicle control according to an embodiment. [Figure 11] It is a block diagram showing a configuration example of a vehicle according to an embodiment. [Figure 12] It is a block diagram showing a configuration example of a remote support terminal according to an embodiment. [Figure 13] It is a block diagram showing a configuration example of a management device according to an embodiment.

Embodiments for Carrying Out the Invention

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Remote Support System Consider remote support for a moving object. Remote support is a concept that includes remote monitoring, remote assistance, and remote driving. Examples of moving objects include vehicles, robots, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots. As an example, in the following description, consider the case where the moving object that is the target of remote support is a vehicle. In the case of generalization, the "vehicle" in the following description shall be read as "moving object".

[0013] FIG. 1 is a schematic diagram showing a configuration example of a remote support system 1 according to the present embodiment. The remote support system 1 includes a vehicle 100, a remote support terminal 200, and a management device 300. The vehicle 100 is the target of remote support by a remote supporter X. The remote support terminal 200 is a terminal device that is operated when the remote supporter X performs remote support for the vehicle 100. The remote support terminal 200 can also be referred to as a remote cockpit. The management device 300 manages the remote support system 1. Typically, the management device 300 is a management server on the cloud. The management device 300 may be composed of a plurality of servers that perform distributed processing.

[0014] The vehicle 100, the remote support terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote support terminal 200 can communicate with each other via the management device 300. Also, the vehicle 100 and the remote support terminal 200 may perform direct communication without going through the management device 300.

[0015] The driver or autonomous driving system of vehicle 100 requests remote support as needed. For example, if vehicle 100 encounters a situation where autonomous driving is difficult, the autonomous driving system requests remote support. Vehicle 100 sends a remote support request to the management device 300. The remote support request may be a Request for Information (RFI) or a Request for Operation (RFO). In response to the remote support request, the management device 300 assigns a remote supporter X from among several candidates to vehicle 100, which is the target of remote support. The management device 300 manages the assignment relationship between vehicle 100 and remote supporter X, and provides information about this assignment relationship to vehicle 100 and the remote support terminal 200. Based on the assignment relationship information, vehicle 100 and the remote support terminal 200 establish communication. After communication is established, vehicle 100 and the remote support terminal 200 may communicate directly without going through the management device 300.

[0016] Vehicle 100 is equipped with various sensors, including an on-board camera C. The on-board camera C photographs the area around vehicle 100 and acquires images showing the surrounding conditions. Vehicle information VCL is information obtained from the various sensors and includes images obtained from the on-board camera C. Vehicle 100 transmits vehicle information VCL to the remote support terminal 200.

[0017] The remote support terminal 200 receives vehicle information VCL transmitted from vehicle 100. The remote support terminal 200 presents the vehicle information VCL to the remote supporter X. Specifically, the remote support terminal 200 is equipped with a display device and displays images, etc., on the display device. The remote supporter X looks at the displayed information, recognizes the situation around vehicle 100, and provides remote support for vehicle 100. The remote support information SUP is information related to the remote support provided by the remote supporter X. For example, the remote support information SUP includes instructions or operation amounts entered by the remote supporter X. The remote support terminal 200 transmits the remote support information SUP to vehicle 100 as needed.

[0018] Vehicle 100 receives remote support information SUP transmitted from remote support terminal 200. Vehicle 100 performs vehicle driving control according to the received remote support information SUP.

[0019] In this embodiment, the remote support system 1 further includes one or more infrastructure cameras 400. The infrastructure cameras 400 are installed in the area where the vehicle 100 is moving. In particular, the infrastructure cameras 400 are installed in a position where they can photograph the vehicle 100. The infrastructure cameras 400 photograph the vehicle 100 and its surroundings and acquire an image IMG showing the situation of the vehicle 100 and its surroundings. In other words, the infrastructure cameras 400 acquire an image IMG that also shows the vehicle 100, which is the target of remote support.

[0020] The management device 300 communicates with the infrastructure camera 400 and collects and manages the image data (IMG) captured by the infrastructure camera 400. The management device 300 also provides the image data (IMG) captured by the infrastructure camera 400 to the remote support terminal 200.

[0021] The remote support terminal 200 acquires images (IMG) taken by the infrastructure camera 400 in addition to the images taken by the in-vehicle camera C, and presents these images to the remote supporter X. The images (IMG) taken by the infrastructure camera 400 also show the vehicle 100 itself, which is the target of the remote support. By presenting such images (IMG) to the remote supporter X, it is expected that the accuracy of the remote support and the convenience for the remote supporter X will be further improved.

[0022] As one application example, consider a scenario where an autonomous vehicle is driving autonomously within a factory premises. For example, an autonomous vehicle assembled in an assembly plant automatically drives from the assembly plant to the yard. One or more infrastructure cameras 400 are installed along the road from the assembly plant to the yard. By using these infrastructure cameras 400, the autonomous vehicle can be remotely monitored as it drives autonomously. Furthermore, if the autonomous vehicle encounters a situation where it is difficult to drive autonomously, it can be remotely driven using the infrastructure cameras 400. It is also conceivable to send staff to the site to take over the autonomous vehicle by manual driving when it encounters a situation where it is difficult to drive autonomously, but this is time-consuming and labor-intensive. Remote driving using the infrastructure cameras 400 is more convenient and also saves time and effort.

[0023] 2. Overview of Differentiation Process For convenience, the vehicle 100 that is the target of remote support will be referred to as "Target Vehicle 100T" below. The assignment relationship between remote supporter X and Target Vehicle 100T is shared by Target Vehicle 100T, the remote support terminal 200, and the management device 300.

[0024] As mentioned above, the image IMG captured by the infrastructure camera 400 also includes the target vehicle 100T. In this situation, it may be difficult to identify which of the images in the IMG is the target vehicle 100T. For example, if the same image IMG contains multiple vehicles 100, including the target vehicle 100T, it may be difficult for the remote supporter X to immediately identify which one is the target vehicle 100T assigned to them. In such a situation, the benefits of improved accuracy and convenience from using the infrastructure camera 400 may not be fully realized.

[0025] Therefore, this disclosure proposes a technology that can further improve the accuracy of remote support using the infrastructure camera 400. To this end, the remote support system 1 according to this embodiment is configured to differentiate the target vehicle 100T in the image IMG captured by the infrastructure camera 400 from other vehicles. The process for differentiating the target vehicle 100T in the image IMG from other vehicles will be referred to below as the "differentiation process".

[0026] Figure 2 is a conceptual diagram illustrating the processes related to differentiation processing in the remote support system 1. The remote support system 1 includes a differentiation processing unit 10 and an image presentation unit 20.

[0027] The differentiation processing unit 10 acquires an image IMG captured by the infrastructure camera 400. This image IMG contains at least the target vehicle 100T, which is the subject of remote support. The differentiation processing unit 10 also acquires individual information SPC specific to the target vehicle 100T. For example, the individual information SPC includes location information indicating the actual location of the target vehicle 100T. As another example, the individual information SPC may include feature information indicating the feature quantities of the target vehicle 100T. Examples of feature quantities include color, shape, lighting patterns of the lights, etc. Such individual information SPC is typically included in the vehicle information VCL transmitted from the target vehicle 100T. However, the individual information SPC may be created separately. Based on the individual information SPC, the differentiation processing unit 10 performs differentiation processing to differentiate the target vehicle 100T in the image IMG from other vehicles.

[0028] For example, the differentiation processing unit 10 differentiates the target vehicle 100T in the image IMG from other vehicles by processing (modifying) the image IMG. Image IMG-S is the image IMG after image processing. This type of differentiation processing based on image processing (first differentiation processing) will be explained in detail in Section 3 below.

[0029] As another example, the differentiation processing unit 10 may differentiate the target vehicle 100T in the image IMG from other vehicles by controlling the actual target vehicle 100T. Such vehicle control-based differentiation processing (second differentiation processing) will be described in detail in Section 4 below.

[0030] The image display unit 20 receives the image IMG or the processed image IMG-S from the differentiation processing unit 10. The image display unit 20 is included in the remote support terminal 200 and presents the image IMG or the processed image IMG-S to the remote supporter X. More specifically, the image display unit 20 displays the image IMG or the processed image IMG-S on a display device.

[0031] Furthermore, the location of the differentiation processing unit 10 is not limited as long as it is possible to obtain the image IMG and individual information SPC. As described above, the vehicle 100, the remote support terminal 200, and the management device 300 can communicate with each other via a communication network. In other words, information such as assignment relationship information, image IMG, individual information SPC, and vehicle information VCL can be shared by the vehicle 100, the remote support terminal 200, and the management device 300. Therefore, the differentiation processing unit 10 may be included in any of the vehicle 100, the remote support terminal 200, and the management device 300. The differentiation processing unit 10 may be distributed among two or more of the vehicle 100, the remote support terminal 200, and the management device 300.

[0032] In general terms, the differentiation processing unit 10 and the image presentation unit 20 are implemented by one or more processors and one or more storage devices. The one or more processors perform various information processing. The one or more storage devices store various information necessary for processing by the one or more processors.

[0033] Figure 3 is a flowchart showing the processes related to differentiation processing in the remote support system 1. In step S1, one or more processors acquire an image IMG captured by the infrastructure camera 400. The image IMG shows the target vehicle 100T, which is the subject of remote support. In step S2, one or more processors acquire individual information SPC specific to the target vehicle 100T. In step S3, one or more processors perform differentiation processing to differentiate the target vehicle 100T in the image IMG from other vehicles based on the individual information SPC. In step S4, one or more processors present the image IMG or image IMG-S obtained as a result of the differentiation processing to the remote supporter X.

[0034] As described above, according to this embodiment, the target vehicle 100T, which is the target of remote support, is captured in the image IMG by the infrastructure camera 400. A differentiation process is then performed to differentiate the target vehicle 100T in the image IMG from other vehicles. This differentiation process makes it easier for the remote supporter X to identify the target vehicle 100T in the image IMG. In other words, it becomes easier for the remote supporter X to provide remote support to the target vehicle 100T. As a result, the accuracy of remote support is improved. Furthermore, convenience for the remote supporter X is improved.

[0035] The following describes in detail various examples of the differentiation process according to this embodiment.

[0036] 3. First differentiation process based on image processing Figure 4 is a block diagram showing an example of a functional configuration related to the first differentiation process based on image processing. The differentiation processing unit 10 includes the first differentiation processing unit 30. The first differentiation processing unit 30 includes a target identification unit 31 and an image processing unit 32.

[0037] 3-1. Target Identification Process The target identification unit 31 acquires individual information SPC specific to the target vehicle 100T. Based on the individual information SPC, the target identification unit 31 performs a "target identification process" to identify the target vehicle 100T in the image IMG. Various methods can be considered for the target identification process.

[0038] 3-1-1. First example of target identification process In the first example, the individual information SPC includes location information indicating the actual location of the target vehicle 100T. Typically, the actual location of the target vehicle 100T is expressed in latitude and longitude in an absolute coordinate system. The location information of the target vehicle 100T is included in the vehicle information VCL transmitted from the target vehicle 100T.

[0039] Furthermore, the target identification unit 31 acquires camera installation information CAM, which is installation information for the infrastructure camera 400. More specifically, the camera installation information CAM indicates the installation position, orientation, field of view, etc., of the infrastructure camera 400. Typically, the installation position of the infrastructure camera 400 is expressed in latitude and longitude in an absolute coordinate system. The camera installation information CAM is provided by the infrastructure camera 400 or the management device 300.

[0040] The target identification unit 31 identifies the target vehicle 100T in the image IMG based on the location information of the target vehicle 100T and the camera installation information CAM. More specifically, the area captured by the infrastructure camera 400, i.e., the area shown in the image IMG, can be determined from the camera installation information CAM. Based on that area and the location information of the target vehicle 100T, the display position in the image IMG corresponding to the actual location of the target vehicle 100T can be calculated. In its simplest form, the target vehicle 100T is considered to be visible in a certain area around the calculated display position.

[0041] Alternatively, the object identification unit 31 may have an object recognition model based on machine learning. The object recognition model is trained to recognize various objects in an image. Typically, the objects recognized by the object recognition model are moving objects. Examples of moving objects include people (pedestrians), vehicles, motorcycles, bicycles, etc. The object recognition model may be based on a CNN (Convolutional Neural Network). As another example, the object recognition model may be based on a Transformer, which is a type of deep learning model.

[0042] The object identification unit 31 detects various objects in the image IMG by utilizing an object recognition model. The object identification unit 31 may also identify the type of object. As described above, based on the position information of the target vehicle 100T and the camera installation information CAM, the display position in the image IMG corresponding to the actual position of the target vehicle 100T is calculated. The object identification unit 31 can identify the object at that display position from among the various objects detected in the image IMG as the target vehicle 100T. Alternatively, the object identification unit 31 may narrow its object detection to the image area surrounding that display position.

[0043] The target identification unit 31 may include a tracker. The tracker is software that automatically tracks the same object in a series of temporally consecutive images (IMG) based on a tracking algorithm. By using the tracker, the target identification unit 31 can track the target vehicle 100T in a series of temporally consecutive images (IMG).

[0044] 3-1-2. Second example of target identification processing In the second example, the individual information SPC includes feature information that indicates the characteristics of the target vehicle 100T. Examples of features include the color of the target vehicle 100T, the shape of the target vehicle 100T, and the lighting pattern of the lights mounted on the target vehicle 100T. The lighting pattern of the lights may be unique to the target vehicle 100T. The feature information may be provided by the target vehicle 100T or by a management device 300 that manages each vehicle 100.

[0045] The target identification unit 31 identifies the target vehicle 100T in the image IMG based on the feature information of the target vehicle 100T. For example, the target identification unit 31 has an object recognition model based on machine learning. The object recognition model is the same as in the first example described above. The object recognition model detects various objects in the image IMG and extracts the feature quantities of each detected object. The target identification unit 31 compares the extracted feature quantities of each detected object with the feature quantities of the target vehicle 100T. The target identification unit 31 then identifies objects that have the same or similar extracted feature quantities as the target vehicle 100T as the target vehicle 100T. For example, the target identification unit 31 calculates the similarity (reciprocal of distance) between the extracted feature quantities of each detected object and the feature quantities of the target vehicle 100T in the feature space. The target identification unit 31 then identifies objects whose similarity is above a threshold as the target vehicle 100T.

[0046] 3-2. Image Processing The image processing unit 32 receives information from the object identification unit 31 indicating the result of the object identification process. The information indicating the result of the object identification process includes information about a partial image region containing the identified target vehicle 100T within the image IMG. When the object identification model described above is used, a bounding box surrounding the identified target vehicle 100T is added to the image IMG. The partial image region containing the target vehicle 100T corresponds to the region enclosed by that bounding box. The information indicating the result of the object identification process may also include position information of the bounding box within the image IMG.

[0047] The image processing unit 32 processes the image IMG to differentiate the target vehicle 100T from other vehicles based on the information of the partial image region in which the target vehicle 100T is visible. In other words, the image processing unit 32 differentiates the target vehicle 100T identified within the image IMG from other vehicles by applying image processing to the image IMG. Image IMG-S is the image that has undergone image processing by the image processing unit 32 compared to the original image IMG. The processed image IMG-S is then presented to the remote supporter X.

[0048] Various image processing techniques can be used to differentiate the target vehicle, the 100T. The following describes various examples of image processing.

[0049] 3-2-1. Highlighting Figures 5 and 6 are conceptual diagrams illustrating examples of image processing. In the example shown in Figure 5, the image processing unit 32 highlights (emphasizes) the target vehicle 100T in the image IMG. For example, the image processing unit 32 adds a bounding box surrounding the target vehicle 100T. The color of the bounding box may be specified by the remote supporter X. As another example, the image processing unit 32 may add a marker to the target vehicle 100T. The shape and color of the marker may be specified by the remote supporter X.

[0050] In the example shown in Figure 6, the image processing unit 32 increases the brightness or saturation of the partial image region containing the target vehicle 100T compared to other image regions. The partial image region containing the target vehicle 100T may be the region enclosed by the bounding box described above. For example, the image processing unit 32 may process the partial image region containing the target vehicle 100T to make it brighter. Alternatively, the image processing unit 32 may process other image regions that do not contain the target vehicle 100T to make them darker.

[0051] By highlighting (emphasizing) the target vehicle 100T in this way, the remote supporter X can easily identify the target vehicle 100T in the image IMG-S.

[0052] 3-2-2. Zoom In Figure 7 is a conceptual diagram illustrating another example of image processing. The image processing unit 32 zooms in on the target vehicle 100T in the image IMG, thereby enlarging the target vehicle 100T in the image IMG. This also emphasizes the target vehicle 100T, achieving a similar effect to that of highlighting.

[0053] As an additional effect of zooming in, the area around the target vehicle 100T is also magnified, allowing the remote supporter X to observe the area near the target vehicle 100T more closely. For example, when the target vehicle 100T is being driven into a garage, it is desirable for the remote supporter X to be able to observe the area near the target vehicle 100T more closely. Another example is when the target vehicle 100T is being driven into a narrow alley, where it is desirable for the remote supporter X to be able to observe the area near the target vehicle 100T more closely.

[0054] When parking in a garage or entering a narrow alley, the speed of the target vehicle 100T is low. Also, when the speed of the target vehicle 100T is low, there is no need to observe a position far away from the target vehicle 100T. Therefore, zooming in may be selected when the speed of the target vehicle 100T is low. That is, when the speed of the target vehicle 100T is below a predetermined threshold, the image processing unit 32 may zoom in on the target vehicle 100T to enlarge it. The speed of the target vehicle 100T is obtained from the vehicle information VCL transmitted from the target vehicle 100T.

[0055] 3-2-3. Assigning different information Figure 8 is a conceptual diagram illustrating yet another example of image processing. The image processing unit 32 assigns different information to the target vehicle 100T and other vehicles in the image IMG. That is, the image processing unit 32 assigns first information to the target vehicle 100T in the image IMG and second information to the other vehicles in the image IMG. The first and second pieces of information are different from each other.

[0056] For example, the image processing unit 32 assigns a first marker to the target vehicle 100T as first information, and assigns a second marker to other vehicles as second information. The first and second markers differ in at least one of the following: color, shape, and size. The color, shape, size, etc., of the first marker may be specified by the remote supporter X.

[0057] This method may also be applied to cases where the same image IMG captured by the same infrastructure camera 400 is shared by multiple remote supporters X. The first remote supporter X-1 provides remote support for the first target vehicle 100T-1, and the second remote supporter X-2 provides remote support for the second target vehicle 100T-2. Both the first target vehicle 100T-1 and the second target vehicle 100T-2 are visible in the same image IMG. The image processing unit 32 assigns a first marker to the first target vehicle 100T-1 and a second marker to the second target vehicle 100T-2. The first and second markers may be specified by the first remote supporter X-1 and the second remote supporter X-2, respectively.

[0058] The first information assigned to the first target vehicle 100T-1 may be the identification information of the first remote supporter X-1. Similarly, the second information assigned to the second target vehicle 100T-2 may be the identification information of the second remote supporter X-2. Examples of identification information include the ID number, facial image, avatar, icon, etc., of the remote supporter X. The identification information may also be specified by the remote supporter X.

[0059] By assigning different information to the target vehicle 100T and other vehicles in this way, the remote supporter X can easily identify the target vehicle 100T in the image IMG-S.

[0060] 3-2-4. Combinations and Selections Two or more of the image processing techniques exemplified above may be combined.

[0061] 3-3. Effects As explained above, the first differentiation process applies image processing to the image IMG, thereby differentiating the target vehicle 100T from other vehicles. This first differentiation process makes it easier for the remote supporter X to identify the target vehicle 100T in the image IMG-S. In other words, it becomes easier for the remote supporter X to provide remote support to the target vehicle 100T. As a result, the accuracy of remote support improves. Furthermore, convenience for the remote supporter X is improved.

[0062] 4. Second differentiation process based on vehicle control Figure 9 is a conceptual diagram illustrating an example of a second differentiation process based on vehicle control. For example, by illuminating (flashing) the hazard lights of the actual target vehicle 100T, the target vehicle 100T in the image IMG can be differentiated from other vehicles. This allows the remote supporter X to easily identify the target vehicle 100T in the image IMG.

[0063] However, unnecessarily flashing hazard lights on public roads may cause misunderstanding or confusion among other drivers. Therefore, flashing hazard lights is permitted only when the target vehicle 100T is within a designated area. Examples of designated areas include private property, factories, parking lots, and road shoulders.

[0064] Figure 10 is a block diagram showing an example of a functional configuration related to a second differentiation process based on vehicle control. The differentiation processing unit 10 includes a second differentiation processing unit 40. The second differentiation processing unit 40 includes a condition determination unit 41 and a vehicle control unit 42.

[0065] The condition determination unit 41 acquires individual information SPC specific to the target vehicle 100T. Individual information SPC includes location information indicating the actual location of the target vehicle 100T. Typically, the actual location of the target vehicle 100T is expressed in latitude and longitude in an absolute coordinate system. The location information of the target vehicle 100T is included in the vehicle information VCL transmitted from the target vehicle 100T.

[0066] Furthermore, the condition determination unit 41 acquires map information MAP. The map information MAP has the locations of predetermined areas registered in advance. Examples of predetermined areas include private land, factories, parking lots, road shoulders, etc.

[0067] The condition determination unit 41 determines whether the target vehicle 100T is located within a predetermined area based on the target vehicle 100T's location information and map information MAP. The condition determination unit 41 notifies the vehicle control unit 42 of the determination result.

[0068] The vehicle control unit 42 communicates with the target vehicle 100T and controls the target vehicle 100T. In particular, the vehicle control unit 42 controls the lighting devices 140 installed on the target vehicle 100T. Examples of lighting devices 140 include hazard lights, taillights, headlights, etc., that are originally installed on the target vehicle 100T. Alternatively, the lighting devices 140 may be special lamps that have been retrofitted to the target vehicle 100T.

[0069] If the target vehicle 100T is within a predetermined area, the vehicle control unit 42 activates the lights 140. This differentiates the target vehicle 100T from other vehicles in the image IMG. As a result, the remote supporter X can easily identify the target vehicle 100T in the image IMG. On the other hand, if the target vehicle 100T is outside the predetermined area, the vehicle control unit 42 disables the activation of the lights 140.

[0070] As a variation, the permission conditions for allowing the operation of the lighting device 140 may further include, in addition to "the target vehicle 100T is within a designated area," "the remote operation of the target vehicle 100T is being prepared." When a Request for Operation (RFO) is issued from the target vehicle 100T, the management device 300 assigns a remote supporter X to the target vehicle 100T. The management device 300 provides the target vehicle 100T and the remote support terminal 200 with information regarding the assignment relationship. Based on the information regarding the assignment relationship, the target vehicle 100T and the remote support terminal 200 establish communication. After communication is established, the remote supporter X (remote operator) starts the remote operation of the target vehicle 100T. The preparation period for the remote operation of the target vehicle 100T is the period from the issuance of the request for remote operation to the start of the remote operation of the target vehicle 100T by the remote supporter X. Therefore, the period during which communication is established between the target vehicle 100T and the remote support terminal 200 is included in the preparation period for remote operation. The individual information SPC may include not only the location information of the target vehicle 100T but also information indicating the remote operation status of the target vehicle 100T. Based on such individual information SPC, the condition determination unit 41 determines whether or not the permission conditions for permitting the operation of the light fixture 140 are met. If the permission conditions are met, the vehicle control unit 42 activates the light fixture 140. On the other hand, if the permission conditions are not met, the vehicle control unit 42 prohibits the operation of the light fixture 140. Therefore, after communication is established, when the remote supporter X (remote operator) starts remote operation of the target vehicle 100T, the vehicle control unit 42 stops the operation of the light fixture 140.

[0071] As explained above, the second differentiation process activates the lighting device 140 mounted on the target vehicle 100T, thereby differentiating the target vehicle 100T from other vehicles in the image IMG. This second differentiation process makes it easier for the remote supporter X to identify the target vehicle 100T in the image IMG. In other words, it becomes easier for the remote supporter X to provide remote support to the target vehicle 100T. As a result, the accuracy of remote support improves. Furthermore, convenience for the remote supporter X is improved.

[0072] Furthermore, the second differentiation process eliminates the need for complex image processing. Therefore, the processing load on the differentiation processing unit 10 is reduced.

[0073] 5. Combinations A combination of the first differentiation process described in Section 3 above and the second differentiation process described in Section 4 above is also possible. That is, the differentiation processing unit 10 may include both the first differentiation processing unit 30 and the second differentiation processing unit 40.

[0074] 6. Examples of vehicles 6-1. Example Configuration Figure 11 is a block diagram showing an example configuration of vehicle 100. Vehicle 100 is equipped with a communication device 110, a sensor group 120, a running gear 130, lighting equipment 140, and a control device 150.

[0075] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote support terminal 200 and the management device 300.

[0076] The sensor group 120 includes recognition sensors, vehicle status sensors, position sensors, etc. The recognition sensors recognize (detect) the surrounding conditions of the vehicle 100. Examples of recognition sensors include an on-board camera C, LIDAR (Laser Imaging Detection and Ranging), radar, etc. The vehicle status sensors detect the state of the vehicle 100. The vehicle status sensors include a speed sensor, acceleration sensor, yaw rate sensor, steering angle sensor, etc. The position sensors detect the position and orientation of the vehicle 100. For example, the position sensors include a GNSS (Global Navigation Satellite System).

[0077] The running gear 130 includes a steering gear, a drive gear, and a braking gear. The steering gear steers the wheels. For example, the steering gear includes an electric power steering (EPS) system. The drive gear is a power source that generates driving force. Examples of drive gears include an engine, an electric motor, an in-wheel motor, etc. The braking gear generates braking force.

[0078] Lighting devices 140 include hazard lights, taillights, headlights, etc. Lighting devices 140 may also include special lamps that are retrofitted to the vehicle 100.

[0079] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processor 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processor 160 performs various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage devices 170 store various information necessary for processing by the processor 160. Examples of storage devices 170 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The control device 150 may also include one or more ECUs (Electronic Control Units).

[0080] The vehicle control program PROG1 is a computer program executed by the processor 160. The processor 160 executes the vehicle control program PROG1, thereby realizing the functions of the control device 150. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0081] 6-2. Driving Environment Information The control device 150 uses the sensor group 120 to acquire driving environment information ENV, which indicates the driving environment of the vehicle 100. The driving environment information ENV is stored in the storage device 170.

[0082] The driving environment information (ENV) includes surrounding situation information that shows the recognition results from the recognition sensors. For example, the surrounding situation information includes images captured by the on-board camera C. The surrounding situation information may also include object information regarding objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, traffic lights, signs, roadside structures, etc. The object information indicates the relative position and relative speed of the object with respect to the vehicle 100.

[0083] Furthermore, the driving environment information (ENV) includes vehicle status information that indicates the vehicle status detected by the vehicle status sensor.

[0084] Furthermore, the driving environment information (ENV) includes position information indicating the position and orientation of the vehicle 100. The position information is obtained by a position sensor. High-precision position information may also be obtained by localization processing using map information and surrounding environment information (object information).

[0085] 6-3. Vehicle Driving Control The control device 150 performs vehicle driving control to control the movement of the vehicle 100. Vehicle driving control includes steering control, drive control, and braking control. The control device 150 performs vehicle driving control by controlling the driving device 130 (steering device, drive device, and braking device).

[0086] The control device 150 may perform automatic driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory necessary for the vehicle 100 to drive according to the driving plan, based on the driving environment information ENV. The target trajectory includes a target position and a target speed. The control device 150 then performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0087] 6-4. Processing related to remote support The following describes the case where remote support is provided to vehicle 100. The control device 150 communicates with the remote support terminal 200 via the communication device 110.

[0088] The control device 150 transmits vehicle information VCL to the remote support terminal 200. The vehicle information VCL is information necessary for remote support by the remote supporter X and includes at least a part of the driving environment information ENV described above. For example, the vehicle information VCL includes surrounding situation information (especially images). The vehicle information VCL may also include vehicle status information (speed, etc.).

[0089] The vehicle information VCL may further include individual information SPC specific to the vehicle 100. For example, the individual information SPC may include location information obtained by the position sensor described above. The individual information SPC may also include feature information indicating the characteristic quantities of the vehicle 100. Examples of feature quantities include color, shape, lighting pattern of the lights 140, etc. The individual information SPC may also include information indicating the remote operation status of the vehicle 100. The control device 150 transmits the vehicle information VCL, including the individual information SPC, to the differentiation processing unit 10.

[0090] Furthermore, the control device 150 receives remote support information SUP from the remote support terminal 200. Remote support information SUP is information related to remote support provided by remote supporter X. For example, remote support information SUP includes the amount of operation performed by remote supporter X. The control device 150 performs vehicle driving control according to the received remote support information SUP.

[0091] 7. Examples of remote support terminals Figure 12 is a block diagram showing an example configuration of a remote support terminal 200. The remote support terminal 200 includes a communication device 210, an output device 220, an input device 230, and a control device 250.

[0092] The communication device 210 communicates with the vehicle 100 and the management device 300.

[0093] The output device 220 outputs various types of information. For example, the output device 220 includes a display device. The display device presents various types of information to the remote supporter X by displaying the information. As another example, the output device 220 may include a speaker.

[0094] The input device 230 receives input from the remote supporter X. Examples of the input device 230 include a touch panel, buttons, remote control components, etc. The remote control components are components operated by the remote supporter X (remote operator) when remotely driving the vehicle 100. For example, the remote control components include a steering wheel, accelerator pedal, brake pedal, turn signals, etc. The remote control components may also be a touch panel.

[0095] The control device 250 controls the remote support terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processors 260 perform various processes. For example, the processor 260 includes a CPU. The storage devices 270 store various information necessary for processing by the processors 260. Examples of storage devices 270 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0096] The remote support program PROG2 is a computer program executed by the processor 260. The processor 260 executes the remote support program PROG2, thereby realizing the functions of the control unit 250. The remote support program PROG2 is stored in the storage device 270. Alternatively, the remote support program PROG2 may be recorded on a computer-readable recording medium. The remote support program PROG2 may also be provided via a network.

[0097] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL transmitted from the vehicle 100. The control device 250 presents the vehicle information VCL, including an image, to the remote supporter X by displaying it on a display device. Based on the vehicle information VCL displayed on the display device, the remote supporter X can recognize the status of the vehicle 100 and the surrounding conditions.

[0098] The remote supporter X operates the input device 230 as needed to provide remote support to the vehicle 100. For example, the remote supporter X issues various instructions (e.g., a start instruction) via the input device 230. When remotely driving the vehicle 100, the remote supporter X operates the remote driving components. The amount of operation of the remote driving components is detected by a sensor installed on the remote driving components. The control device 250 generates remote support information SUP, which includes the instructions or operation amounts input by the remote supporter X. The control device 250 then transmits the remote support information SUP to the vehicle 100 via the communication device 210.

[0099] The control device 250 may also have the functions of the differentiation processing unit 10. In this case, the control device 250 acquires vehicle information VCL, including individual information SPC, from the vehicle 100. The control device 250 also acquires camera installation information CAM regarding the infrastructure camera 400 via the management device 300. Furthermore, the control device 250 acquires image IMG captured by the infrastructure camera 400 via the management device 300. The acquired information and image IMG are stored in the storage device 270. The control device 250 performs differentiation processing based on the acquired information. For example, the control device 250 performs first differentiation processing and generates image IMG-S (see Section 3 above). As another example, the control device 250 performs second differentiation processing by sending control instructions to the vehicle 100 via the communication device 210 (see Section 4 above).

[0100] The control device 250 has the functions of the image presentation unit 20. That is, the control device 250 presents the image IMG or the processed image IMG-S to the remote supporter X via the output device 220. More specifically, the control device 250 displays the image IMG or the processed image IMG-S on the display device.

[0101] 8. Examples of control devices Figure 13 is a block diagram showing an example configuration of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0102] The communication device 310 communicates with the vehicle 100, the remote support terminal 200, and the infrastructure camera 400.

[0103] The control device 350 controls the management device 300. The control device 350 includes one or more processors 360 (hereinafter simply referred to as processor 360) and one or more storage devices 370 (hereinafter simply referred to as storage devices 370). The processors 360 perform various processes. For example, the processor 360 includes a CPU. The storage devices 370 store various information necessary for processing by the processors 360. Examples of storage devices 370 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0104] The management program PROG3 is a computer program executed by the processor 360. The processor 360 executes the management program PROG3, thereby realizing the functions of the control unit 350. The management program PROG3 is stored in the storage device 370. Alternatively, the management program PROG3 may be recorded on a computer-readable recording medium. The management program PROG3 may also be provided via a network.

[0105] The control device 350 communicates with the vehicle 100 and the remote support terminal 200 via the communication device 310. The control device 350 receives vehicle information VCL transmitted from the vehicle 100. The control device 350 then transmits the received vehicle information VCL to the remote support terminal 200. The control device 350 also receives remote support information SUP transmitted from the remote support terminal 200. The control device 350 then transmits the received remote support information SUP to the vehicle 100.

[0106] Furthermore, the control device 350 communicates with the infrastructure camera 400 via the communication device 310 and acquires the image IMG captured by the infrastructure camera 400. The control device 350 provides the image IMG to the remote support terminal 200. The control device 350 may also acquire camera installation information CAM related to the infrastructure camera 400 and provide the camera installation information CAM to the remote support terminal 200.

[0107] The control device 350 may also have the functions of the differentiation processing unit 10. In this case, the control device 350 acquires vehicle information VCL, including individual information SPC, from the vehicle 100. The control device 350 also acquires image IMG captured by the infrastructure camera 400 and camera installation information CAM. The acquired information and image IMG are stored in the storage device 370. The control device 350 performs differentiation processing based on the acquired information. For example, the control device 350 performs first differentiation processing and generates image IMG-S (see Section 3 above). The control device 350 then provides image IMG-S to the remote support terminal 200. As another example, the control device 350 performs second differentiation processing by sending control instructions to the vehicle 100 via the communication device 310 (see Section 4 above). [Explanation of Symbols]

[0108] 1…Remote support system, 10…Differentiation processing unit, 20…Image presentation unit, 30…First differentiation processing unit, 31…Target identification unit, 32…Image processing unit, 40…Second differentiation processing unit, 41…Condition determination unit, 42…Vehicle control unit, 100…Vehicle, 100T…Target vehicle, 140…Lighting device, 200…Remote support terminal, 300…Management device, 400…Infrastructure camera, IMG…Image, IMG-S…Processed image, SPC…Individual information, VCL…Vehicle information

Claims

1. A remote support system for providing remote support to mobile objects, Equipped with one or more processors, The target mobile object is a mobile object assigned to a remote supporter and is subject to the remote support provided by the remote supporter. The one or more processors described above are: An image showing the aforementioned moving object is obtained, A differentiation process is performed to differentiate the target moving object in the aforementioned image from other moving objects that are not assigned to the remote supporter. The image obtained as a result of the differentiation process is presented to the remote supporter. It is configured in such a way Remote support system.

2. A remote support system according to claim 1, The aforementioned image was captured by an infrastructure camera. Remote support system.

3. A remote support system according to claim 1, The one or more processors are configured to perform the differentiation process based on individual information specific to the target moving object. Remote support system.

4. A remote support method, performed by a computer, for providing remote support to a mobile object, The remote supporter who provides the aforementioned remote support will acquire an image showing the target moving object, The process involves performing a differentiation operation to distinguish the target moving object in the image from other moving objects that are not assigned to the remote supporter, The image obtained as a result of the differentiation process is presented to the remote supporter. including Remote support methods.

Images