Information processing system and method thereof

Abstract

To provide an alternative or improved method for using an image of an environment photographed in a sub-area in a movable area of a mobile body in a movement support of the mobile body.SOLUTION: An information processing system performs: acquiring communication quality information in each one of a plurality of sub-areas in a movable area of a mobile body; and acquiring image data of an environment photographed in each of the sub-areas. In the information processing system, information obtained from the image data of each sub-area is considered when evaluating or using communication quality in the sub-area for a movement support of the mobile body.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an information processing system and a method thereof, and particularly to an information processing system and a method thereof for assisting the movement of a moving object.

Background Art

[0002] Patent Document 1 discloses remote monitoring of a moving object (for example, an autonomous vehicle). Patent Document 1 describes that a device evaluates the communication stability of a plurality of candidate routes of a moving object and determines the moving route of the moving object from among the plurality of candidate routes based on the evaluation result. The communication stability in Patent Document 1 means an index indicating the possibility that the communication quality of a wireless communication network satisfies the communication quality requirement or the possibility that the communication quality of the wireless communication network does not satisfy the communication quality requirement. To evaluate the communication stability of each candidate route, for example, the device uses the communication quality measured for each location and for each communication direction (that is, upstream communication or downstream communication). The device may hold communication quality information for each region of several tens of meters square divided in a mesh shape. When a plurality of communication quality measurement values or analysis values are included in the region, the device may calculate a statistical value such as the median or average value of these and use the calculated statistical value as the representative value of the region. Since the congestion level of the base station can change depending on the time and day of the week, the device may hold quality information for each time zone.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 discloses that communication quality information is retained for each of a plurality of sub-areas within a movable area of a mobile body, the communication stability of a plurality of candidate routes is evaluated, and based on this evaluation result, a movement route of the mobile body is determined from among the plurality of candidate routes. However, Patent Document 1 does not teach considering or using information obtained from image data of the environment captured within each sub-area when evaluating or using the communication quality of each sub-area for mobile body movement support, such as determining a candidate route of the mobile body or determining the operation of the mobile body. An alternative or improved method may be needed for using an image of the environment captured within a sub-area within the movable area of the mobile body in mobile body movement support.

[0005] One of the objects to be achieved by the embodiments disclosed in this specification is to provide an apparatus, a method, and a program that contribute to solving at least one of a plurality of problems including the problems described above. It should be noted that this object is only one of the plurality of objects to be achieved by the plurality of embodiments disclosed in this specification. Other objects or problems and novel features will be clarified from the description of this specification or the accompanying drawings.

Means for Solving the Problems

[0006] In a first aspect, an information processing system is configured to acquire communication quality information for each of a plurality of sub-areas within a movable area of a mobile body, and acquire image data of the environment captured in each of the plurality of sub-areas. The information processing system is configured to consider information obtained from the image data of each sub-area when evaluating or using the communication quality in that sub-area for mobile body movement support.

[0007] In a second aspect, the method performed by the information processing system includes: (a) obtaining communication quality information in each of a plurality of sub-areas within the movable area of the mobile object; (b) obtaining image data of the environment captured in each of the plurality of sub-areas; and (c) considering the information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for assisting the movement of the mobile object.

[0008] In a third aspect, the program includes a set of instructions (software code) for causing a computer to perform the method described in the second aspect when loaded into the computer.

Advantages of the Invention

[0009] According to the above aspect, it is possible to provide an apparatus, a method, and a program that contribute to solving at least one of a plurality of problems including the above-described problems.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0011] Hereinafter, specific embodiments will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant descriptions are omitted as necessary for clarity of explanation.

[0012] Each of the plurality of embodiments described below can be used alone, or two or more embodiments may be appropriately combined. These plurality of embodiments may have different novel features from each other. Therefore, these plurality of embodiments can contribute to achieving different purposes or solving different problems, and can contribute to achieving different effects from each other.

[0013] Each drawing is merely an example for explaining one or more embodiments. Each drawing is not necessarily associated with only one specific embodiment, but may be associated with one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments that are not explicitly illustrated or described. Not all of the features or steps shown in any one drawing for explaining exemplary embodiments are necessarily essential, and some features or steps may be omitted. The order of the steps described in any drawing may be changed as appropriate.

[0014] The following multiple embodiments will be described mainly with respect to an autonomous vehicle traveling on land and its remote monitoring system. However, these embodiments may be applied to other moving bodies and their remote monitoring systems. The other moving bodies may be, for example, ships, aircraft, drones, Uncrewed Aerial Vehicles (UAVs), Automated Guided Vehicle (AGV), Automated Guided Forklift (AGF), Autonomous Mobile Robot (AMR), or Unmanned Ground Vehicle (UGV).

[0015] As used herein, depending on the context, "(if)~then" may be construed to mean "when", "while", "at or around the time", "after", "upon", "in response to determining", "in accordance with a determination", or "in response to detecting". These expressions may be construed to have the same meaning depending on the context.

[0016] First, the configuration and operation of the device common to multiple embodiments will be described. FIG. 1 is a schematic diagram showing various components for remote monitoring of a moving body related to multiple embodiments. The term "remote monitoring" in this specification may mean remote monitoring or remote control or both.

[0017] The mobile body 1 is configured to automatically move without depending on the driver's operation at least under predetermined conditions. The mobile body 1 may be called an autonomous mobile body. The mobile body 1 may be an autonomous vehicle. More specifically, the mobile body 1 may support level 2 or higher among the six levels of autonomous driving levels 0 to 5 defined by the Society of Automotive Engineers (SAE (registered trademark)) International.

[0018] The mobile body 1 is equipped with various sensors. These sensors include sensors for measuring or observing the internal state of the mobile body 1. Furthermore, these sensors include sensors used for measuring or observing the surrounding environment of the mobile body 1. The information on the internal state and the surrounding environment of the mobile body 1 obtained by the sensors can be utilized to control the movement of the mobile body 1. The sensors for measuring the surrounding environment of the mobile body 1 may include one or more cameras, one or more microphones, one or more millimeter-wave radars, one or more Light Detection And Ranging (LiDAR) sensors, one or more Navigation Satellite System (NSS) receivers, or any combination thereof. The NSS receiver receives the positioning signals of one or more NSSs. These NSSs may be one or more Global NSSs (GNSSs), one or more Regional NSSs (RNSSs), or a combination thereof.

[0019] The mobile body 1 includes a device for controlling the driving, braking, and steering systems (e.g., engine, motor, brake, steering wheel) of the mobile body 1 in order to achieve the automatic movement of the mobile body 1. The device controls the driving, braking, and steering systems of the mobile body 1 based at least partially on the information obtained from the sensors mounted on the mobile body 1 and the information received from the remote monitoring system 2.

[0020] The remote monitoring system 2 communicates with the moving body 1 via the network 5. The network 5 typically includes a wireless network. The wireless network may include a cellular network. The cellular network may be a Long Term Evolution (LTE) network or a Fifth Generation (5G) network or a combination thereof. The cellular network may include a Beyond 5G network or a 6G network. The cellular network may be provided by a Mobile Network Operator (MNO) or may be a Non-Public Network (NPN) provided by someone other than the MNO. When the cellular network is an NPN, this may be an independent network called a Stand-alone Non-Public Network (SNPN) or may be an NPN linked to an MNO network called a Public network integrated NPN (PNI-NPN).

[0021] The remote monitoring system 2 monitors the state of the moving body 1 and supports the movement of the moving body 1. The remote monitoring system 2 may determine the planned movement route of the moving body 1 and provide this to the moving body 1, and the moving body 1 may perform automatic movement according to the provided planned movement route. The remote monitoring system 2 may execute remote driving of the moving body 1 as necessary. Remote driving includes performing the driving, braking, and steering of the moving body 1 from the remote monitoring system 2. The remote monitoring system 2 may determine the operation of the moving body 1 (for example, deceleration, stop at a specific location) as necessary and instruct this to the moving body 1. The remote monitoring system 2 may request the moving body 1 to prompt the driver or pilot of the moving body 1 to execute operations for coping with abnormal situations.

[0022] The remote monitoring system 2 may include a plurality of servers or computers, which may be geographically separated from each other. In addition to one or more central servers, the remote monitoring system 2 may include one or more edge computing servers disposed near a cellular network (e.g., Radio Access Network (RAN)) within the network 5.

[0023] The information processing device 3 is used to assist the movement of the mobile body 1. The information processing device 3 may be referred to as an information processing system. In one example, the information processing device 3 evaluates or uses the communication quality in each of a plurality of sub-areas within the movable area of the mobile body 1 for assisting the movement of the mobile body 1. The movable area may be a two-dimensional or three-dimensional area that defines the ground surface or space where the mobile body 1 can travel, operate, or fly. The movable area may be referred to as a navigable area. For example, the movable area may be represented by a two-dimensional map, and each sub-area may be one of discrete geographical elements (i.e., mesh areas) obtained by dividing the two-dimensional map.

[0024] The communication quality of each sub-area is related to the communication quality of the network 5, and more specifically, may be related to the communication quality of a wireless network (e.g., a cellular network) within the network 5. The communication quality of each sub-area may be measured or acquired, for example, by the mobile body 1, but is not limited thereto. Additionally or alternatively, the communication quality of each sub-area may be measured or acquired by a drive test. Additionally or alternatively, the communication quality of each sub-area may be measured or acquired using the Logged Minimization of Drive Test (MDT) technology defined in the 3rd Generation Partnership Project (3GPP (registered trademark)) specifications.

[0025] The communication quality of each sub - area may include the reference signal reception power, reference signal reception quality, or Signal to Noise power Ratio (SNR) of a wireless network (e.g., a cellular network), or any combination thereof. Additionally or alternatively, the communication quality of each sub - area may include the delay, throughput, or packet loss rate of the wireless network or of the entire Network 5, or any combination thereof.

[0026] The communication quality of each sub - area may include statistical values calculated based on multiple communication quality measurement values obtained in each sub - area, such as average values and standard deviations. In other words, the communication quality of each sub - area may indicate the variation or fluctuation of communication quality in each sub - area. From this perspective, the communication quality of each sub - area may be referred to as the communication stability of each sub - area. In other words, the communication quality of each sub - area may include one or more parameters representing the communication stability of each sub - area. Communication stability may refer to the property that the measurement value of communication quality remains at a certain value for a predetermined period of time. A low communication stability may indicate a state where the measurement value fluctuates greatly during a predetermined period of time. A high communication stability may indicate a state where the measurement value remains at a certain value or the variation from the measurement is small during a predetermined period of time.

[0027] The communication quality of each sub - area may be used by the information processing device 3 to determine whether to include the sub - area in the planned movement route of the mobile body 1. Additionally or alternatively, the communication quality of each sub - area may be used by the information processing device 3 to determine the operation of the mobile body 1 within the sub - area or within the surrounding sub - areas. In other words, the information processing device 3 may determine the planned movement route of the mobile body 1 based at least in part on the communication quality of each sub - area. Additionally or alternatively, the information processing device 3 may determine whether to perform a specific operation for assisting the movement of the mobile body 1 based at least in part on the communication quality of each sub - area.

[0028] The information processing device 3 is disposed in one or both of the mobile body 1 and the remote monitoring system 2. One or more functions provided by the information processing device 3 may be distributedly disposed in the mobile body 1 and the remote monitoring system 2. In other words, one or more functions provided by the information processing device 3 may be distributedly disposed in the mobile body 1 and one or more computers in the remote monitoring system 2.

[0029] The information processing device 3 may be implemented by dedicated hardware (e.g., one or more Application Specific Integrated Circuits (ASICs)) or as a programmable logic device (e.g., one or more Field Programmable Gate Arrays (FPGAs)). Alternatively, the information processing device 3 may be implemented by a computer including one or more general-purpose processors and one or more programs. Alternatively, the information processing device 3 may be implemented by any combination of dedicated hardware, one or more programmable logic devices, and one or more general-purpose computers. Thus, part or all of the method or algorithm performed by the information processing device 3 may be implemented by computer program instructions.

[0030] Figure 2 shows an example of an implemented information processing apparatus 3 using a general-purpose computer system. In the example of Figure 2, the computer system includes one or more processors 210, a memory 220, and a mass storage 230, which communicate with each other via a bus 270. The one or more processors 210 may include, for example, one or more Central Processing Units (CPUs), one or more Graphics Processing Units (GPUs), or both. The computer system may also include other devices such as one or more output devices 240, one or more input devices 250, and one or more peripherals 260. The one or more peripherals 260 may include a modem, a network adapter, or any combination thereof.

[0031] One or both of the memory 220 and the mass storage 230 include a computer-readable medium storing one or more sets of computer program instructions. These instructions may be partially or fully arranged in a memory within the one or more processors 210. When these instructions are executed in the one or more processors 210, they cause the one or more processors 210 to provide the functions of the information processing apparatus 3.

[0032] In other words, one or more processors 210 can execute one or more computer programs. These programs, when loaded into a computer, include a set of instructions (or software code) for causing the computer to perform the functions of the information processing apparatus 3. The programs may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray (registered trademark) disk, or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The programs may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0033] <First Embodiment> Figure 3 shows an example of the operation of the information processing apparatus 3. In step 301, the information processing apparatus 3 acquires communication quality information regarding each sub-area within the movable area of the moving body 1. As described above, the communication quality information may indicate, for example, but is not limited to, the communication quality measured by the moving body 1. The information processing apparatus 3 may acquire communication quality information based on the measured communication quality by receiving this from another system, or may acquire this by reading it from a storage device. When at least a part of the functions of the information processing apparatus 3 are arranged in the remote monitoring system 2, the information processing apparatus 3 may receive, from the moving body 1 via the network 5, the communication quality information measured and stored by the moving body 1. When at least a part of the functions of the information processing apparatus 3 are arranged in the moving body 1, the information processing apparatus 3 may use the communication quality information measured and stored by the moving body 1, or may receive the communication quality information from the remote monitoring system 2 via the network 5.

[0034] In step 302, the information processing apparatus 3 acquires image data of the environment captured within each sub-area within the movable area of the moving body 1, or information obtained from the image data. The image data includes one or both of still image data and moving image data. The information obtained from the image data may be information about the image of an object detected from the image data using image recognition technology. The process of obtaining information from the image data, for example, the process of detecting a photographed object from the image data, may be performed by the information processing apparatus 3, the moving body 1, the remote monitoring system 2, or another system.

[0035] The information processing device 3 may acquire the image data (or information obtained from the image data) of each sub-area by receiving this from another system, or may acquire this by reading it from a storage device. When at least a part of the functions of the information processing device 3 are arranged in the remote monitoring system 2, the information processing device 3 may receive the image data (or information obtained from the image data) photographed and stored by the mobile body 1 from the mobile body 1 via the network 5. When at least a part of the functions of the information processing device 3 are arranged in the mobile body 1, the information processing device 3 may use the image data (or information obtained from the image data) photographed and stored by the mobile body 1, or may receive this from the remote monitoring system 2 via the network 5.

[0036] In step 303, when the information processing device 3 evaluates or uses the communication quality in each sub-area for the movement support of the mobile body 1, the information obtained from the image data of the environment photographed within each sub-area is considered. Various non-limiting examples of evaluating or using the communication quality while considering the image data are provided below. At least a part of these examples may be executed in combination.

[0037] In the first example, the information processing device 3 considers whether an image of a specific object is detected from the image data of each sub-area in order to evaluate the communication quality in the sub-area. The information processing device 3 may adjust the communication quality related score of each sub-area used for the movement support of the mobile body 1 based on the information obtained from the image data. In other words, the information processing device 3 may increase or decrease the communication quality score of each sub-area determined based on the measurement result of the communication quality based on the type, number, or state of the object photographed in the image data of the sub-area. The communication quality related score may be a score representing communication stability, in other words, a score representing the variation or fluctuation of the communication quality. Such a communication quality score (that is, a communication stability score) may be determined according to the average value and standard deviation of the measured communication quality.

[0038] Specifically, if an image of a specific object is detected from the image data of each sub-area, the information processing apparatus 3 may adjust the communication quality related score of the sub-area so that the communication quality of the sub-area is considered to be lower than when the image of the specific object is not detected. Specifically, if an image of a specific object is not detected from the image data of each sub-area, the information processing apparatus 3 may calculate a first value of the communication quality related score corresponding to the communication quality measured in the sub-area, and determine this as the communication quality related score of the sub-area. On the other hand, if an image of a specific object is detected from the image data of each sub-area, a second value lower than the first value (calculated according to the communication quality measured in the sub-area) may be determined as the communication quality related score of the sub-area. In this case, the specific object may include at least one of street trees and power lines. If a street tree is detected from the image data, the information processing apparatus 3 may predict the possibility that the communication quality will deteriorate in the future in consideration of the height of the detected street tree, the growth condition of the leaves and branches of the detected street tree, the season in which the street tree is detected, etc., and adjust the communication quality related score based on the prediction result. If a power line is detected from the image data of a certain sub-area, the information processing apparatus 3 may lower the communication quality related score of the sub-area compared to the case where it is not detected.

[0039] Further or alternatively, if an image of a specific object is detected from the image data of each sub-area, the information processing apparatus 3 may adjust the communication quality related score of the sub-area so that the communication quality of the sub-area is considered to be higher than when the image of the specific object is not detected. Specifically, if no image of a specific object is detected from the image data of each sub-area, the information processing apparatus 3 may calculate a first value of the communication quality related score corresponding to the communication quality measured in the sub-area, and determine this as the communication quality related score of the sub-area. On the other hand, if an image of a specific object is detected from the image data of each sub-area, a second value higher than the first value (calculated according to the communication quality measured in the sub-area) may be determined as the communication quality related score of the sub-area. In this case, the specific object may include a running train. If a running train is captured in the image data acquired in relation to (e.g., simultaneously with) the measurement of the communication quality in a certain sub-area, it is possible to determine that the deterioration of the communication quality is temporary when the train passes. Therefore, if a running train is detected from the image data of a certain sub-area, the information processing apparatus 3 may increase the communication quality related score of the sub-area compared to the case where it is not detected. In other words, if a specific object that can be presumed to be the cause of the temporary decrease in communication quality during measurement is detected from the image data of the sub-area, the information processing apparatus 3 may determine that the decrease in communication quality measured in the image area is temporary.

[0040] In the second example, the information processing device 3 may determine an operation for assisting the movement of the moving body 1 based on the communication quality of each sub - area and whether or not an image of a specific object has been detected from the image data of the sub - area. In other words, the information processing device 3 may use the information obtained from the image data of a certain sub - area to determine whether to perform a specific operation for assisting the movement of the moving body 1 based on the communication quality of the sub - area. For example, if no specific object is detected from the image data of a certain sub - area, the information processing device 3 may determine that the communication quality (e.g., communication stability) of that sub - area may decrease. Then, the information processing device 3 may control the driving, braking, and steering of the moving body 1 according to the determination. Further or alternatively, the information processing device 3 may control the moving body 1 to output one or both of a display and a warning sound to notify the driver or pilot of the moving body 1 of the possible decrease in communication quality or stability. If at least a part of the functions of the information processing device 3 is arranged in the remote monitoring system 2, the information processing device 3 may request the moving body 1 via the network 5 to output such a display or warning sound or both. The specific object may include at least one of roadside trees, power transmission lines, and a running train.

[0041] In the third example, the information processing device 3 infers the cause of the deterioration of the communication quality of each sub - area based on the image of the object detected from the image data of the sub - area. The information processing device 3 may also predict the possibility that the communication quality of each sub - area will deteriorate in the future based on the image of the object detected from the image data of the sub - area. The specific object may include at least one of roadside trees, power transmission lines, and a running train. The details of the inference of the cause of the deterioration of the communication quality and the prediction of the possibility of future communication quality deterioration may be the same as those described in the first or second example above.

[0042] When evaluating or using the communication quality in each sub - area for the movement support of the mobile body 1, the information processing device 3 may further consider or use the sound data acquired within each sub - area in association with the image data. For example, the information processing device 3 may utilize the sound data as auxiliary information when detecting an object from the image data.

[0043] The information processing device 3 may perform an operation for improving the communication quality (e.g., communication stability) of each sub - area according to the communication quality of the sub - area. Specifically, if it is determined or predicted that the communication quality (e.g., communication stability) of a certain sub - area is lower than the level or threshold required for the movement of the mobile body 1, the information processing device 3 may perform an operation for improving the communication quality of the sub - area. The information processing device 3 may instruct the mobile body 1 to perform an operation for improving the communication quality of the sub - area.

[0044] The operation for improving the communication quality may include changing the cellular network used by the communication device of the mobile body 1, for example, changing from the cellular network of one network operator to the cellular network of another network operator. Further or alternatively, the operation for improving the communication quality may include changing the wireless communication method used by the communication device of the mobile body 1, for example, changing from 5G to LTE. Further or alternatively, the operation for improving the communication quality may include changing or restricting the lane when the mobile body 1 moves in the sub - area. Further or alternatively, the operation for improving the communication quality may include changing or restricting the time period when the mobile body 1 moves in the sub - area, for example, avoiding moving in the sub - area during the time period when the network is congested. Further or alternatively, the operation for improving the communication quality may include changing the direction of the antenna provided on the mobile body 1.

[0045] According to the operation described with reference to FIG. 3, the information processing apparatus 3 uses an image of the environment captured in a sub-area within the movable area of the mobile body 1 in conjunction with the communication quality information of the sub-area in the movement support of the mobile body 1. As can be understood from the above description, the information obtained from the image data can be useful, for example, for estimating the cause of deterioration of communication quality, for determining whether the deterioration of communication quality is temporary, or for predicting the possibility of future deterioration of communication quality. Therefore, the operation described with reference to FIG. 3 can easily estimate the cause of deterioration of communication quality, determine whether the deterioration of communication quality is temporary, or predict the possibility of future deterioration of communication quality, for example.

[0046] <Second Embodiment> FIG. 4 shows an example of the operation of the information processing apparatus 3. In step 401, the information processing apparatus 3 acquires communication quality information in each sub-area within the movable area of the mobile body 1. As described above, the communication quality information may indicate, for example, the communication quality measured by the mobile body 1, but is not limited thereto. The information processing apparatus 3 may acquire communication quality information based on the measured communication quality by receiving this from another system, or may acquire this by reading it from a storage device. When at least a part of the functions of the information processing apparatus 3 is arranged in the remote monitoring system 2, the information processing apparatus 3 may receive the communication quality information measured and stored by the mobile body 1 from the mobile body 1 via the network 5. When at least a part of the functions of the information processing apparatus 3 is arranged in the mobile body 1, the information processing apparatus 3 may use the communication quality information measured and stored by the mobile body 1, or may receive the communication quality information from the remote monitoring system 2 via the network 5.

[0047] In step 402, when the information processing apparatus 3 evaluates or uses the communication quality in each sub-area for the movement support of the mobile body 1, it considers one or both of the type and number of frequency bands available for wireless communication in each sub-area. Generally, communication in a low frequency band (e.g., sub-6 GHz, FR1) has higher communication quality stability than communication in a high frequency band (e.g., millimeter wave band, FR2). Therefore, the information processing apparatus 3 may use the type of frequency band available in the sub-area to evaluate the communication quality or communication stability of the sub-area. Also, if the number of available frequency bands is large, the mobile body 1 has a chance to continue communication using another frequency band even if the communication quality of one of the frequency bands deteriorates. Therefore, the information processing apparatus 3 may use the number of frequency bands available in the sub-area to evaluate the communication quality or communication stability of the sub-area.

[0048] According to the operation described with reference to FIG. 4, when evaluating or using the communication quality in a sub-area within the movable area of the mobile body 1, the information processing apparatus 3 considers one or both of the type and number of frequency bands available in the sub-area. This contributes to more accurately evaluating the communication quality (e.g., communication stability) of the sub-area for the movement support of the mobile body 1.

[0049] <Third Embodiment> FIG. 5 shows an example of the operation of the information processing apparatus 3. In step 501, the information processing apparatus 3 specifies communication quality requirements according to the automatic driving level of the mobile body 1. In other words, the information processing apparatus 3 determines the required communication quality (e.g., communication stability) level according to the automatic driving level of the mobile body 1. The automatic driving level of the mobile body 1 may be defined by six levels of automatic driving levels 0 to 5 based on international standards defined by SAE International. The information processing apparatus 3 may specify the communication quality requirements so that the required communication quality (e.g., communication stability) increases as the automatic driving level of the mobile body 1 increases.

[0050] In step 502, the information processing device 3 determines the planned movement route or operation of the mobile body 1 in consideration of the specified communication quality requirements. For example, the information processing device 3 may exclude a sub-area whose communication quality does not meet the communication quality requirements, and generate a planned movement route of the mobile body 1 that moves to a sub-area that meets the communication quality requirements. Further or alternatively, when the mobile body 1 moves to a sub-area that does not meet the communication quality requirements, the information processing device 3 may determine the operation of the mobile body 1 (for example, deceleration, stop at a specific location), and instruct the mobile body 1 to do so.

[0051] According to the operation described with reference to FIG. 5, the information processing device 3 can use different communication quality requirements according to the automatic driving level of the mobile body 1. Further, the information processing device 3 can determine the planned movement route or operation of the mobile body 1 based on the communication quality requirements corresponding to the automatic driving level of the mobile body 1.

[0052] <Fourth Embodiment> FIG. 6 shows an example of the operation of the information processing device 3. In step 601, the information processing device 3 acquires communication quality information regarding each sub-area within the movable area of the mobile body 1. As described above, the communication quality information may indicate, for example, the communication quality measured by the mobile body 1, but is not limited thereto. The information processing device 3 may acquire the communication quality information based on the measured communication quality by receiving this from another system, or may acquire this by reading it from a storage device. When at least a part of the functions of the information processing device 3 is arranged in the remote monitoring system 2, the information processing device 3 may receive the communication quality information measured and stored by the mobile body 1 from the mobile body 1 via the network 5. When at least a part of the functions of the information processing device 3 is arranged in the mobile body 1, the information processing device 3 may use the communication quality information measured and stored by the mobile body 1, or may receive the communication quality information from the remote monitoring system 2 via the network 5.

[0053] In step 602, the information processing apparatus 3 acquires the image data of the environment captured in each sub - area within the movable area of the mobile body 1, or the information obtained from the image data. The image data includes one or both of still - image data and moving - image data. The information obtained from the image data may be information about the image of an object detected from the image data using image recognition technology. The process of obtaining information from the image data, for example, the process of detecting the captured object from the image data, may be performed by the information processing apparatus 3, the mobile body 1, the remote monitoring system 2, or another system.

[0054] The information processing apparatus 3 may acquire the image data (or the information obtained from the image data) of each sub - area by receiving this from another system, or may acquire this by reading it from a storage device. When at least a part of the functions of the information processing apparatus 3 is arranged in the remote monitoring system 2, the information processing apparatus 3 may receive the image data (or the information obtained from the image data) captured and stored by the mobile body 1 from the mobile body 1 via the network 5. When at least a part of the functions of the information processing apparatus 3 is arranged in the mobile body 1, the information processing apparatus 3 may use the image data (or the information obtained from the image data) captured and stored by the mobile body 1, or may receive this from the remote monitoring system 2 via the network 5.

[0055] In step 603, the information processing apparatus 3 displays a display representing the communication quality of each sub - area on the map screen of the movable area, and associates and displays the image data of each sub - area or the information about the image of an object detected therefrom with the display of the communication quality. The information processing apparatus 3 may display these on a display device included in the computer in the remote monitoring system 2. Further or alternatively, the information processing apparatus 3 may display these on a display device mounted on the mobile body 1.

[0056] The information processing apparatus 3 may display, by overlaying a transparency color layer corresponding to the communication quality level of the sub-area, within the division of the sub-areas in the map screen. FIG. 7 illustrates an example of overlaying a transparency color corresponding to the communication quality level in each of a plurality of sub-area divisions 710 within the map screen 700. In FIG. 7, for the sake of illustration, different colors are represented by different hatching. The information processing apparatus 3 may display, in response to an input operation by an operator (e.g., an operation manager who operates the remote monitoring system 2, a driver or a pilot of the moving body 1), image data of each sub-area or information about an image of an object to be detected therefrom on a display device. For example, in response to the operator clicking on the location of a sub-area displayed in a color corresponding to its communication quality on the map screen, the information processing apparatus 3 may additionally display (e.g., by a pop-up window) image data of the sub-area or information about an image of an object to be detected therefrom.

[0057] According to the operation described with reference to FIG. 6, the information processing apparatus 3 can display image data of each sub-area or information about an image of an object to be detected therefrom, in association with a display representing the communication quality of each sub-area within the movable area of the moving body 1. This can assist the operator, for example, in inferring the cause of deterioration of the communication quality of the sub-area based on the image data.

[0058] The above-described embodiments are merely examples regarding the application of the technical idea obtained by the present inventor. That is, the technical idea is not limited to only the above-described embodiments, and it goes without saying that various modifications are possible.

[0059] For example, some or all of the above-described embodiments may be described as follows in the appended claims, but are not limited thereto. Some or all of the elements (e.g., configurations and functions) described in the appended claims directed to an apparatus (e.g., an information processing system) may naturally be described as appended claims directed to a method and a program. For example, some or all of the elements described in Appendices 2-26 subordinate to Appendix 1 may be described as appendices subordinate to Appendix 53 with the same subordinate relationship as in Appendix 2-26. Some or all of the elements described in any appendix may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0060] (Appendix 1) Means for acquiring communication quality information in each of a plurality of sub-areas within the movable area of the moving body, Means for acquiring image data of the environment captured in each of the plurality of sub-areas, Means for considering the information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for the movement support of the moving body, An information processing system comprising the same. (Appendix 2) The means for consideration is configured to consider whether an image of a specific object is detected from the image data of each sub-area when evaluating the communication quality in the sub-area. The information processing system according to Appendix 1. (Appendix 3) The means for consideration is configured to adjust the communication quality-related score of the sub-area according to whether an image of a specific object is detected from the image data of each sub-area. The information processing system according to Appendix 2. (Appendix 4) If an image of a specific object is detected from the image data of each sub-area, the means for consideration is configured to adjust the communication quality-related score of the sub-area such that the communication quality of the sub-area is considered to be lower than when the image of the specific object is not detected. The information processing system described in Supplementary Note 3. (Supplementary Note 5) The specific object includes at least one of a street tree and a power transmission line. The information processing system described in Supplementary Note 4. (Supplementary Note 6) When an image of a specific object is detected from the image data of each sub - area, the means for consideration is configured to adjust the communication quality - related score of the sub - area so that the communication quality of the sub - area is considered to be higher than when the image of the specific object is not detected. The information processing system described in Supplementary Note 3. (Supplementary Note 7) The specific object includes a running train. The information processing system described in Supplementary Note 6. (Supplementary Note 8) The means for consideration is configured to determine an operation for assisting the movement of the moving body based on the communication quality of each sub - area and whether an image of a specific object is detected from the image data of the sub - area. The information processing system described in Supplementary Note 1. (Supplementary Note 9) The means for consideration is configured to infer the cause of deterioration of the communication quality of each sub - area based on the image of the object detected from the image data of the sub - area. The information processing system described in Supplementary Note 1. (Supplementary Note 10) The object includes at least one of a street tree, a power transmission line, and a running train. The information processing system described in Supplementary Note 8 or 9. (Supplementary Note 11) The means for consideration is configured to adjust the communication quality - related score of each sub - area used for assisting the movement of the moving body based on the information obtained from the image data. The information processing system described in Supplementary Note 1. (Supplementary Note 12) The means for consideration is configured to use the information obtained from the image data in order to determine whether to perform a specific operation for assisting the movement of the mobile body based on the communication quality of each sub-area. The information processing system according to Supplementary Note 1. (Supplementary Note 13) The means for consideration is configured to further consider sound data acquired within each sub-area in association with the image data when evaluating or using the communication quality in each sub-area for assisting the movement of the mobile body. The information processing system according to any one of Supplementary Notes 1 to 12. (Supplementary Note 14) The means for consideration is configured to further consider one or both of the type and number of frequency bands available for wireless communication in the sub-area when evaluating or using the communication quality in each sub-area for assisting the movement of the mobile body. The information processing system according to any one of Supplementary Notes 1 to 13. (Supplementary Note 15) The communication quality of each sub-area is used to determine whether to include the sub-area in the planned movement route of the mobile body, or to determine the operation of the mobile body within the sub-area or in the surrounding sub-areas. The information processing system according to any one of Supplementary Notes 1 to 14. (Supplementary Note 16) The communication quality includes a parameter representing the stability of communication. The information processing system according to any one of Supplementary Notes 1 to 15. (Supplementary Note 17) The information processing system further includes means for displaying a display representing the communication quality of each sub-area on a map screen of the movable area and displaying the image data of each sub-area in association with the display. The information processing system according to any one of Supplementary Notes 1 to 16. (Supplementary Note 18) Means for displaying, on the map screen of the movable area, a display representing the communication quality of each sub-area, and for displaying information about the image of an object detected from the image data of each sub-area in association with the display. The information processing system according to any one of Appendices 1 to 16. (Appendix 19) Means for performing an operation for improving the communication quality of each sub-area according to the communication quality of the sub-area. [[ID=z]] The information processing system according to any one of Appendices 1 to 18. (Appendix 20) The operation for improving the communication quality includes at least one of changing the cellular network used by the communication device of the moving body, changing the wireless communication method used by the communication device, changing the frequency band used by the communication device, changing or restricting the driving lane of the moving body, changing or restricting the time period during which the moving body travels, and changing the orientation of the antenna of the moving body. The information processing system according to Appendix 19. (Appendix 21) Means for specifying communication quality requirements according to the automatic driving level of the moving body, and for determining the planned movement route or the operation of the moving body in consideration of the specified communication quality requirements. The information processing system according to any one of Appendices 1 to 20. (Appendix 22) The information processing system is mounted on the moving body. The information processing system according to any one of Appendices 1 to 21. (Appendix 23) The information processing system is arranged in one or more computers in a remote monitoring system for assisting the movement of the moving body. The information processing system according to any one of Appendices 1 to 21. (Appendix 24) One or more means provided by the information processing system are distributedly arranged between the moving body and one or more computers in the remote monitoring system. The information processing system according to any one of Supplementary Notes 1 to 21. (Supplementary Note 25) The moving body is an autonomous ground vehicle. The information processing system according to any one of Supplementary Notes 1 to 24. (Supplementary Note 26) The image data includes one or both of still image data and moving image data. The information processing system according to any one of Supplementary Notes 1 to 25. (Supplementary Note 27) Obtaining communication quality information in each of a plurality of sub-areas within the movable area of the moving body, Obtaining image data of the environment captured in each of the plurality of sub-areas, and Considering the information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for the movement support of the moving body. A method performed by an information processing system comprising the above. (Supplementary Note 28) The above consideration includes considering whether an image of a specific object is detected from the image data of each sub-area when evaluating the communication quality in the sub-area. The method according to Supplementary Note 27. (Supplementary Note 29) The above consideration includes adjusting a communication quality related score of the sub-area according to whether an image of a specific object is detected from the image data of each sub-area. The method according to Supplementary Note 28. (Supplementary Note 30) The above consideration includes adjusting the communication quality related score of the sub-area such that when an image of a specific object is detected from the image data of each sub-area, the communication quality of the sub-area is considered to be lower than when the image of the specific object is not detected. The method according to Supplementary Note 29. (Supplementary Note 31) The specific object includes at least one of street trees and power transmission lines. The method according to Supplementary Note 30. (Appendix 32) The consideration includes adjusting the communication quality related score of the sub - area such that if an image of a specific object is detected from the image data of each sub - area, the communication quality of the sub - area is considered to be higher than when the image of the specific object is not detected. The method according to Appendix 29. (Appendix 33) The specific object includes a running train. The method according to Appendix 32. (Appendix 34) The consideration includes determining an operation for assisting the movement of the moving body based on the communication quality of each sub - area and whether an image of a specific object is detected from the image data of the sub - area. The method according to Appendix 27. (Appendix 35) The consideration includes inferring the cause of the deterioration of the communication quality of each sub - area based on the image of the object detected from the image data of the sub - area. The method according to Appendix 27. (Appendix 36) The object includes at least one of street trees, power transmission lines, and running trains. The method according to Appendix 34 or 35. (Appendix 37) The consideration includes adjusting the communication quality related score of each sub - area used for assisting the movement of the moving body based on the information obtained from the image data. The method according to Appendix 27. (Appendix 38) The consideration includes using the information obtained from the image data to determine whether to perform a specific operation for assisting the movement of the moving body based on the communication quality of each sub - area. The method according to Appendix 27. (Appendix 39) The consideration includes further considering sound data acquired within each sub - area in association with the image data when evaluating or using the communication quality in each sub - area for the movement support of the moving body. The method according to any one of Appendices 27 to 38. (Appendix 40) The consideration includes further considering one or both of the type and number of frequency bands available for wireless communication in the sub - area when evaluating or using the communication quality in each sub - area for the movement support of the moving body. The method according to any one of Appendices 27 to 39. (Appendix 41) The communication quality of each sub - area is used to determine whether to include the sub - area in the planned movement route of the moving body, or to determine the operation of the moving body within the sub - area or in the surrounding sub - areas. The method according to any one of Appendices 27 to 40. (Appendix 42) The communication quality includes parameters representing the stability of communication. The method according to any one of Appendices 27 to 41. (Appendix 43) Further comprising displaying a display representing the communication quality of each sub - area on a map screen of the movable area, and displaying the image data of each sub - area in association with the display. The method according to any one of Appendices 27 to 42. (Appendix 44) Further comprising displaying a display representing the communication quality of each sub - area on a map screen of the movable area, and displaying information about the image of the object detected from the image data of each sub - area in association with the display. The method according to any one of Appendices 27 to 42. (Appendix 45) Further comprising performing an operation to improve the communication quality of each sub - area according to the communication quality of the sub - area. The method according to any one of Appendices 27 to 44. (Appendix 46) The operation for enhancing the communication quality includes at least one of changing the cellular network used by the communication device of the moving body, changing the wireless communication method used by the communication device, changing the frequency band used by the communication device, changing or restricting the driving lane of the moving body, changing or restricting the time period during which the moving body travels, and changing the orientation of the antenna of the moving body. The method according to Supplementary Note 45. (Supplementary Note 47) Further comprising specifying communication quality requirements according to the autonomous driving level of the moving body, and determining the planned movement route or the operation of the moving body in consideration of the specified communication quality requirements. The method according to any one of Supplementary Notes 27 to 46. (Supplementary Note 48) The information processing system is mounted on the moving body. The method according to any one of Supplementary Notes 27 to 47. (Supplementary Note 49) The information processing system is arranged in one or more computers in the remote monitoring system for assisting the movement of the moving body. The method according to any one of Supplementary Notes 27 to 47. (Supplementary Note 50) One or more functions provided by the information processing system are distributed between the moving body and one or more computers in the remote monitoring system. The method according to any one of Supplementary Notes 27 to 47. (Supplementary Note 51) The moving body is an autonomous driving ground vehicle. The method according to any one of Supplementary Notes 27 to 50. (Supplementary Note 52) The image data includes one or both of still image data and moving image data. The method according to any one of Supplementary Notes 27 to 51. (Supplementary Note 53) Obtaining communication quality information in each of a plurality of sub-areas within the movable area of the moving body. Obtaining image data of the environment captured in each of the plurality of sub-areas, and Considering information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for the movement support of the mobile body, A program for causing a computer to perform a method comprising the above.

Explanation of Signs

[0061] 1 Mobile body 2 Remote monitoring system 3 Information processing device 5 Network 210 Processor 220 Memory 230 Mass storage 240 Output device 250 Input device 260 Peripheral device

Claims

1. Means for obtaining communication quality information in each of a plurality of sub-areas within the movable area of the mobile body; Means for obtaining image data of the environment captured in each of the plurality of sub-areas; Means for considering the information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for the movement support of the mobile body; An information processing system comprising the above.

2. The means for consideration is configured to adjust the communication quality related score of the sub-area according to whether an image of a specific object is detected from the image data of each sub-area. The information processing system according to Claim 1.

3. The means for consideration is configured to adjust the communication quality related score of the sub-area such that when an image of a specific object is detected from the image data of each sub-area, the communication quality of the sub-area is considered to be higher than when the image of the specific object is not detected. The information processing system according to Claim 2.

4. The means for consideration is configured to determine an operation for the movement support of the mobile body based on the communication quality of each sub-area and whether an image of a specific object is detected from the image data of the sub-area. The information processing system according to Claim 1.

5. The means for consideration is configured to infer the cause of the deterioration of the communication quality of each sub-area based on the image of the object detected from the image data of the sub-area. The information processing system according to Claim 1.

6. The means for consideration is configured to adjust the communication quality related score of each sub-area used for the movement support of the mobile body based on the information obtained from the image data. The information processing system according to Claim 1.

7. The means for consideration is configured to use the information obtained from the image data to determine whether to perform a specific operation for the movement support of the mobile body based on the communication quality of each sub-area. The information processing system according to Claim 1.

8. The communication quality includes a parameter representing the stability of communication. The information processing system according to any one of Claims 1 to 7.

9. Obtaining communication quality information in each of a plurality of sub-areas within the movable area of the mobile body; Obtaining image data of the environment captured in each of the plurality of sub-areas, and Considering information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for the movement support of the mobile body. A method performed by an information processing system comprising the above.

10. Obtaining communication quality information in each of a plurality of sub-areas within the movable area of the mobile body. Obtaining image data of the environment captured in each of the plurality of sub-areas, and Considering information obtained from the image data of each sub-area when evaluating or using the communication quality in the sub-area for the movement support of the mobile body. A program for causing a computer to perform a method comprising the above.

Images