Information processing device, information processing method and information processing program

Abstract

To accurately determine a state of an object.SOLUTION: An information processing device 100 comprises: an acquisition section 133 which acquires time-series data of a signal level which is acquired from a sensor and time-series data of a risk based on a position of an object which is acquired from a sensor; and a generation section 136 in which information indicating a state of the object is generated by a learned model based on the time-series data of the signal level and the time-series data of the risk which are acquired by the acquisition section 133.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for determining the state of an object from position data of the object recognized by a sensor (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5547913 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the prior art, it was sometimes impossible to accurately determine the state of an object. For example, when determining the state of an object from a change in the position of the object obtained using a motion sensor, factors such as the accuracy of the motion sensor, the material of the reflective object, and the moving speed of the object can cause a discrepancy between the actual position of the object and the position of the object recognized by the motion sensor, making it impossible to accurately recognize the position of the object and to correctly determine the state of the object. Thus, the above-mentioned problem is one example of the problem that the present invention aims to solve. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems and achieve the object, the invention described in claim 1 is characterized by having an acquisition unit that acquires time series data of signal levels acquired from a sensor, time series data of an object's position acquired from the sensor, and time series data of a risk based on the object's position, and a generation unit that generates information indicating the state of the object using a trained model based on the time series data of the signal levels, the time series data of the object's position, and the time series data of the risk acquired by the acquisition unit.

[0006] The invention described in claim 7 is a method executed by a computer, characterized in that it includes an acquisition step of acquiring time series data of signal levels acquired from a sensor, time series data of an object's position acquired from the sensor, and time series data of a risk based on the object's position, and a generation step of generating information indicating the state of the object using a trained model based on the time series data of signal levels, the time series data of the object's position, and the time series data of the risk acquired by the acquisition step.

[0007] The invention described in claim 8 is characterized in that it has a computer execute an acquisition step of acquiring time series data of signal levels acquired from a sensor, time series data of object positions acquired from the sensor, and time series data of risk based on the object positions, and a generation step of generating information indicating the state of the object using a trained model based on the time series data of signal levels, the time series data of object positions, and the time series data of risk acquired by the acquisition step. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 3] FIG. 3 is a diagram for explaining distance risk. [Figure 4]FIG. 4 is a diagram for explaining directional risk. [Figure 5] FIG. 5 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of the flow of processing by the information processing device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of processing by the information processing device according to the embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a creation process performed by the creation unit according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of the flow of processing by the information processing device according to the embodiment. [Figure 15] FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[0010] [First embodiment] [System configuration] First, the configuration of an information processing system 1 according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of the information processing system 1 according to the embodiment. As shown in Fig. 1, an example of the information processing system 1 according to the embodiment is a vehicle 10 equipped with an information processing device 100. The information processing device 100 may be, for example, a device that is built into the vehicle 10, brought in, or externally attached and communicatively connected.

[0011] The information processing device 100 determines the state of an object present around the vehicle 10, such as approach, reconnaissance, or intrusion into the vehicle 10, based on information obtained by a sensor, such as a motion sensor, that is built into the information processing device 100 or communicably connected to the information processing device 100. For example, the information processing device 100 obtains time-series data of signal levels obtained from the sensor and time-series data of distance risks based on the position of the object obtained from the sensor, and generates information indicating the state of the object using a first trained model based on the obtained time-series data of signal levels and time-series data of distance risks.

[0012] As a result, the information processing device 100 inputs the time-series data of the signal level and the time-series data of the distance risk into a trained model to generate the state of the object, thereby preventing a decrease in the accuracy of determining the state of the object due to an error in the recognition position of the sensor, and making it possible to determine the state of the object with high accuracy. That is, the information processing device 100 in the first embodiment does not use object position information as input to the model, and is therefore characterized in that the effect of position detection errors on object state determination is small, making it possible to determine the state of the object with high accuracy.

[0013] On the other hand, the information processing device 100 in the first embodiment does not use object position information as input to the model, which means that it has the disadvantage of being unable to detect position-specific risks such as those near the tires, or to exclude position-specific risks caused by objects such as flags that are often found behind the vehicle.

[0014] [Configuration of information processing device] Next, an information processing device 100 according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. As shown in Fig. 2, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Each unit of the information processing device 100 will be described below.

[0015] The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from, for example, an in-vehicle device mounted on the vehicle 10.

[0016] The storage unit 120 is realized by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk. The storage unit 120 stores the position where the motion sensor is installed, the signal level, the angle calculated from the received signal, the distance, the coordinates of the motion object (object), image data, the direction of travel, various thresholds, distance risk, directional risk, risk, trained model, and other information necessary for creating image data and determining the state of the object. The trained model is a model stored after training by the training unit 135, which will be described later, is completed.

[0017] The control unit 130 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in memory. As shown in Fig. 2, the control unit 130 has a sensor unit 131, a calculation unit 132, an acquisition unit 133, a creation unit 134, a learning unit 135, and a generation unit 136. Each unit of the control unit 130 will be described below.

[0018] The sensor unit 131 acquires information from various sensors. For example, the sensor unit 131 detects a moving object using a motion sensor such as a distance sensor, a microwave sensor, or a LiDAR (light detection and ranging). At this time, the sensor unit 131 can acquire the position of the detected object using the information acquired from the motion sensor. The sensor unit 131 also detects the position of the vehicle 10 using a positioning sensor such as a GNSS (Global Navigation Satellite System) sensor or a GPS (Global Positioning System) sensor. The sensor unit 131 also detects the acceleration of the vehicle 10 using an acceleration sensor. For example, the sensor unit 131 also detects the angular velocity of the vehicle 10 using a gyro sensor. For example, the sensor unit 131 also acquires image data (moving image data and still image data) of the surroundings of the vehicle 10 using an imaging device.

[0019] Here, as an example of the location where the sensors of the sensor unit 131 may be installed, the motion sensor may be installed in a location inside or outside the vehicle 10 according to the purpose, such as the front pillar, center pillar, rear pillar, room mirror, side mirror, ceiling, seat, rear window, etc.

[0020] The calculation unit 132 calculates a risk (including a distance risk and a direction risk) based on the position of the object acquired by the sensor unit 131. Here, the risk calculated by the calculation unit 132 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 shows a distance risk, which is a degree of danger related to the position of an object depending on the distance from the vehicle 10. For example, the distance risk takes a value of 0 to 1 point depending on the distance between the vehicle 10 and the object, and as shown in Fig. 3, the closer the distance between the vehicle 10 and the object, the greater the distance risk, and conversely, the farther the distance between the vehicle 10 and the object, the smaller the distance risk.

[0021] For example, the calculation unit 132 calculates that the distance risk is "1" from the distance between the vehicle 10 and the position of the object acquired by the sensor unit 131. Note that, for the sake of explanation, the distance risk is shown only to the right of the vehicle 10 in Fig. 3, but the distance risk is calculated based on the distance from the vehicle 10 to objects present in the front, rear, left, and right of the vehicle 10.

[0022] 4 shows directional risk, which is the degree of danger related to the traveling direction of an object depending on its positional relationship with the vehicle 10. For example, the directional risk takes on a value of 1 to 4 depending on the traveling direction of the object, and as shown in FIG. 4, the directional risk is large when the traveling direction of the object is toward the vehicle 10, and conversely, the directional risk is small when the traveling direction of the object is away from the vehicle 10.

[0023] For example, the calculation unit 132 calculates that the directional risk is "4" from the traveling direction of the object relative to the position of the vehicle 10, which is identified from a change in the position of the object acquired by the sensor unit 131. The traveling direction of the object may also be identified from a change in the coordinate position of the object identified by the sensor. Also, for the sake of explanation, FIG. 4 shows the directional risk only to the right of the vehicle 10, but the directional risk is calculated based on the traveling directions of objects present in the front, rear, left, and right of the vehicle 10.

[0024] The risk is the degree of danger of the object to the vehicle 10, and may be the distance risk itself, the directional risk itself, or the product of the distance risk and the directional risk. For example, the calculation unit 132 calculates the risk as "4", which is the product of the distance risk "1" calculated from the distance between the object's position acquired by the sensor unit 131 and the vehicle 10, and the directional risk "4" calculated from the object's traveling direction relative to the vehicle 10, which is identified from changes in the object's position acquired by the sensor unit 131.

[0025] The acquisition unit 133 acquires time-series data of the signal level acquired from the sensor and time-series data of the distance risk based on the position of the object acquired from the sensor. For example, the acquisition unit 133 acquires time-series data of the signal level acquired from the sensor “Count0: 2.9, Count1: 3, Count2: 3.1, Count3...” from the sensor unit 131, and time-series data of the distance risk based on the position of the object acquired from the sensor “Count0: 0.3, Count1: 0.2, Count2: 0.5, Count3...” from the calculation unit 132.

[0026] Note that Count is an element that represents time in time-series data and may repeat at a predetermined cycle. For example, Count takes a value from 0 to 39 every 50 milliseconds, and when 2000 milliseconds have passed and Count reaches 39, it returns to Count 0 again.

[0027] The acquisition unit 133 further acquires time-series data of directional risk based on the position of an object acquired from the sensor. For example, the acquisition unit 133 acquires time-series data of signal levels acquired from the sensor, “Count0:3.0, Count1:3.1, Count2:3.1, . . . ,” from the sensor unit 131, and time-series data of distance risk based on the position of an object acquired from the sensor, “Count0:0.2, Count1:0.2, Count2:0.4, . . . ,” and time-series data of directional risk based on the position of an object acquired from the sensor, “Count0:3, Count1:3, Count2:3, . . . ,” from the calculation unit 132.

[0028] The creation unit 134 creates image data that chronologically represents the time-series data of the signal level and the time-series data of the distance risk acquired by the acquisition unit 133. For example, the creation unit 134 creates image data in which the vertical axis represents the integrated value of the signal level and the distance risk acquired by the acquisition unit 133 and the horizontal axis represents time.

[0029] Here, the creation process by the creation unit 134 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the creation process by the creation unit 134. Fig. 5 shows an example of a display mode of information included in image data created by the creation unit 134. For example, the creation unit 134 creates image data in which the vertical axis represents the integrated value of the signal level and the distance risk, and the horizontal axis represents time. That is, in the case of the above example, the creation unit 134 creates image data by plotting points at pixels at positions identified from the elements of the integrated value of the signal level and the distance risk, and time.

[0030] As another example, the creation unit 134 can create image data that further includes information on directional risk. For example, the creation unit 134 creates image data in which the vertical axis represents the integrated value of the signal level and distance risk acquired by the acquisition unit 133, the horizontal axis represents time, and the directional risk represents brightness.

[0031] Here, the creation process by the creation unit 134 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the creation process by the creation unit 134. Fig. 6 shows an example of a display mode of information included in image data created by the creation unit 134. For example, the creation unit 134 creates image data in which the vertical axis represents the integrated value of the signal level and the distance risk, the horizontal axis represents time, and a third axis perpendicular to the vertical and horizontal axes represents the directional risk. That is, in the case of the above example, the creation unit 134 creates image data by plotting points representing the directional risk in a predetermined format such as density or brightness at pixels at positions specified by the elements of the integrated value of the signal level and the distance risk and time.

[0032] Next, an example of image data created by the creation unit 134 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the creation process by the creation unit 134. Fig. 7 shows examples of image data created by the creation unit 134. Figs. 7(1) to (3) are image data including a signal level, a distance risk, and a direction risk. The image data in Fig. 7(1) includes a signal level, a distance risk, and a direction risk when an object approaches the vehicle 10, and shows an increase over time in the integrated value of the signal level and the distance risk due to the approach of the object. In addition, a change in the traveling direction of the object is shown as a change in brightness.

[0033] The image data in Figure 7(2) includes the signal level, distance risk, and direction risk when an object spying on the vehicle 10, and the integrated value of the signal level and distance risk shows a certain range of values ​​because the object is near the vehicle 10 and spying on the vehicle 10. Also, changes in the object's traveling direction are shown as changes in brightness.

[0034] The image data in Figure 7(3) includes the signal level, distance risk, and directional risk when an object enters the vehicle 10, and the integrated value of the signal level and distance risk is higher than that of the reconnaissance in Figure 7(2) because the object has entered the vehicle 10. In addition, the change in the object's direction of travel is shown as a change in brightness.

[0035] The above is an example of image data created by the creation unit 134, and image data can be created in which each element corresponds to each axis and brightness, such as image data with directional risk on the vertical axis, time on the horizontal axis, and the integrated value of signal level and distance risk on the brightness.

[0036] The creation unit 134 creates image data so that one piece of image data includes time-series data of signal levels for a predetermined period and time-series data of distance risks. For example, the creation unit 134 creates image data so that one piece of image data includes time-series data of signal levels for 2000 milliseconds and time-series data of distance risks for 2000 milliseconds.

[0037] The creation unit 134 creates image data so that one image data set includes a predetermined number of time-series data of signal levels and time-series data of distance risks. For example, the creation unit 134 creates image data so that one image data set includes information on 40 pieces of time-series data of signal levels and 40 pieces of time-series data of distance risks.

[0038] The learning unit 135 causes the first learning model to learn the relationship between the information on the time-series data of the signal level and the time-series data of the distance risk and the state of the object. For example, the learning unit 135 causes the first learning model to learn the state of the object, such as approach, reconnaissance, or intrusion, in response to changes in the signal level and the distance risk over a certain period of time. Note that the time-series data of the signal level and the time-series data of the distance risk may be included in the image data. For example, the learning unit 135 causes the first learning model to learn the relationship between the image data including the time-series data of the signal level and the time-series data of the distance risk and the state of the object for each image data.

[0039] Furthermore, the learning unit 135 causes the second learning model to learn the relationship between the information on the time-series data of the signal level, the time-series data of the distance risk, and the time-series data of the directional risk, and the state of the object. For example, the learning unit 135 causes the second learning model to learn the state of the object, such as approach, reconnaissance, or intrusion, in response to changes in the signal level, distance risk, and directional risk over a certain period of time.

[0040] The time-series data of the signal level, the time-series data of the distance risk, and the time-series data of the direction risk may be included in the image data. For example, the learning unit 135 causes the second learning model to learn the relationship between the image data including the time-series data of the signal level, the time-series data of the distance risk, and the time-series data of the direction risk and the state of the object for each image data.

[0041] Each learning model that has been learned by the learning unit 135 is stored in the storage unit 120 as a learned model.

[0042] The generation unit 136 generates information indicating the state of the object using a first trained model based on the time series data of the signal level and the time series data of the distance risk acquired by the acquisition unit 133. For example, the generation unit 136 receives the time series data of the signal level and the time series data of the distance risk acquired by the acquisition unit 133 as input, and generates information indicating the state of the object using a first trained model that outputs the state of the object in response to the input of the time series data of the signal level and the time series data of the distance risk.

[0043] More specifically, the generation unit 136 inputs the signal level time series data "Count0:1, Count1:2, Count2:3, Count3..." and the distance risk time series data "Count0:0.3, Count1:0.3, Count2:0.4, Count3..." acquired by the acquisition unit 133 into a first trained model that outputs object information in response to the input of the signal level time series data and the distance risk time series data, thereby generating object information "approaching."

[0044] As another example, the time-series data of the signal level and the time-series data of the distance risk input to the generation unit 136 may be included in the image data. For example, the generation unit 136 receives the image data created by the creation unit 134 as input, and generates information indicating the state of the object using a first trained model that outputs information indicating the state of the object in response to input image data that represents the time-series data of the signal level and the time-series data of the distance risk in a time-series manner.

[0045] More specifically, the generation unit 136 inputs image data (for example, the image data shown in Figure 7(2)) that represents the signal level time series data "Count0:3, Count1:3, Count2:3, Count3..." and the distance risk time series data "Count0:0.8, Count1:0.8, Count2:0.8, Count3..." acquired by the acquisition unit 133 in a time series manner into a first trained model that outputs object information in response to input of image data that represents the signal level time series data and the distance risk time series data in a time series manner, thereby generating object information "reconnaissance."

[0046] As another example, the generation unit 136 may further use image data including time-series data of directional risk as input. For example, the generation unit 136 uses the image data created by the creation unit 134 as input and generates information indicating the state of the object using a second trained model that outputs information indicating the state of the object in response to input of image data that time-series data of the signal level, time-series data of the distance risk, and time-series data of the directional risk.

[0047] More specifically, the generation unit 136 inputs image data (for example, the image data shown in Figure 7(3)) that represents the signal level time series data "Count0:8.0, Count1:7.9, Count2:8.1, Count3...", the distance risk time series data "Count0:1.0, Count1:1.0, Count2:1.0, Count3...", and the direction risk time series data "Count0:4.0, Count1:4, Count2:3, Count3..." acquired by the acquisition unit 133 in a time series manner into a first trained model that outputs object information in response to input of image data that represents the signal level time series data and the distance risk time series data in a time series manner, thereby generating object information "intrusion".

[0048] 〔flowchart〕 Next, processing by information processing device 100 configured as described above will be described with reference to the flowchart in Fig. 8. The flowchart in Fig. 8 is mainly executed by control unit 130. Furthermore, this flowchart can be configured as a program executed by a CPU included in control unit 130, thereby forming an information processing program.

[0049] First, the acquisition unit 133 acquires time-series data of the signal level acquired from the sensor and time-series data of the distance risk based on the position of the object acquired from the sensor (step S101).

[0050] Next, the creating unit 134 creates image data that represents the time-series data of the signal level and the time-series data of the distance risk acquired by the acquiring unit 133 in a time-series manner (step S102).

[0051] Then, the generation unit 136 takes the image data created by the creation unit 134 as input and generates information indicating the state of the object using a first trained model that outputs information indicating the state of the object in response to input image data that represents time series data of signal level and time series data of distance risk in a time series manner (step S103).

[0052] 〔effect〕 The information processing device 100 according to the embodiment includes an acquisition unit 133 that acquires time series data of signal levels acquired from a sensor and time series data of distance risk based on the position of an object acquired from the sensor, and a generation unit 136 that generates information indicating the state of an object using a first trained model based on the time series data of signal levels and the time series data of distance risk acquired by the acquisition unit 133.

[0053] As a result, the information processing device 100 generates information on the state of an object using time series data of signal level and time series data of distance risk as inputs to a model, thereby preventing a decrease in the accuracy of determining the state of an object due to errors in the sensor's recognition position, and can determine the state of an object with higher accuracy than when only the object's position data is used.

[0054] In addition, the information processing device 100 according to the embodiment further includes a creation unit 134 that creates image data that represents the time series data of the signal level and the time series data of the distance risk in a time series manner acquired by the acquisition unit 133, and the generation unit 136 uses the image data created by the creation unit 134 as input and generates information that indicates the state of the object using a first trained model that outputs information that indicates the state of the object in response to the input of image data that represents the time series data of the signal level and the time series data of the distance risk in a time series manner.

[0055] As a result, the information processing device 100 determines the state of an object by using image data including time series data of signal level and time series data of distance risk as input to the model, enabling processing using an image recognition model and enabling the state of an object to be determined with high accuracy.

[0056] Furthermore, the creation unit 134 of the information processing device 100 according to the embodiment creates image data in which the vertical axis represents the integrated value of the signal level and distance risk acquired by the acquisition unit 133, and the horizontal axis represents time.

[0057] As a result, the information processing device 100 creates image data with the integrated value of the signal level and distance risk on the vertical axis and time on the horizontal axis, and can use the image data that largely reflects changes in accordance with the state of the object as input to the image recognition model, thereby accurately determining the state of the object.

[0058] Furthermore, the acquisition unit 133 of the information processing device 100 according to the embodiment further acquires time series data of directional risk based on the position of the object acquired from the sensor, and the creation unit 134 creates image data that time-series represents the time series data of the signal level, the time series data of the distance risk, and the time series data of the directional risk acquired by the acquisition unit 133, and the generation unit 136 uses the image data created by the creation unit 134 as input and generates information that indicates the state of the object using a second trained model that outputs information that indicates the state of the object in response to input of image data that time-series represents the time series data of the signal level, the time series data of the distance risk, and the time series data of the directional risk.

[0059] As a result, the information processing device 100 determines the state of an object by using image data including time series data of signal level, time series data of distance risk, and time series data of directional risk as input to the model, thereby enabling processing using an image recognition model that inputs image data containing more information, and enabling the state of an object to be determined with higher accuracy.

[0060] The creation unit 134 of the information processing device 100 according to the embodiment creates image data in which the vertical axis represents the integrated value of the signal level and distance risk acquired by the acquisition unit 133, the horizontal axis represents time, and the directional risk represents brightness.

[0061] As a result, the information processing device 100 creates image data with the integrated value of the signal level and distance risk on the vertical axis, time on the horizontal axis, and directional risk on the brightness, thereby including more information in the image data and using image data that largely reflects changes in accordance with the state of the object as input to the image recognition model, thereby making it possible to more accurately determine the state of the object.

[0062] Moreover, the creation unit 134 of the information processing apparatus 100 according to the embodiment creates image data so that one image data includes time-series data of the signal level for a predetermined period and time-series data of the distance risk.

[0063] As a result, the information processing device 100 creates image data so that one image data contains data on signal levels and distance risks for a specified period, and by using image data divided at an appropriate granularity as input to the model, it is possible to more accurately determine the state of an object.

[0064] Moreover, the creation unit 134 of the information processing apparatus 100 according to the embodiment creates image data so that one image data includes a predetermined number of time-series data of signal levels and time-series data of distance risks.

[0065] As a result, the information processing device 100 can create one image data set to include a predetermined number of signal level and distance risk data, and use the image data divided at an appropriate granularity as input to the model, thereby making it possible to more accurately determine the state of an object.

[0066] [Second embodiment] In the first embodiment, an example was described in which image data including time-series data of signal level and time-series data of distance risk is used as input to generate information indicating the state of an object, but in the second embodiment described below, an example will be described in which image data including time-series data of signal level, time-series data of object position, and time-series data of risk is used as input to a model to generate information indicating the state of an object. Note that descriptions of content common to the first embodiment will be omitted as appropriate.

[0067] [System configuration] The information processing device 100 acquires, for example, time series data of signal levels acquired from a sensor, time series data of object positions acquired from the sensor, and time series data of risks based on the object positions, and generates information indicating the state of the object using a trained model based on the acquired time series data of signal levels, time series data of object positions, and time series data of risks.

[0068] As a result, the information processing device 100 inputs the time series data of the signal level, the time series data of the object's position, and the time series data of the risk into a trained model to generate the state of the object, thereby preventing a decrease in the accuracy of determining the state of the object due to errors in the sensor's recognition position compared to when only the object's position information is used, and enabling the state of the object to be determined with high accuracy.

[0069] Furthermore, the information processing device 100 in the first embodiment does not use object position information as input to the model, which has the disadvantage of not being able to detect location-specific risks, such as those near the tires, or to exclude location-specific risks caused by objects such as flags that are often found behind the vehicle. However, the information processing device 100 in the second embodiment uses object position information as input to the model, making it possible to detect or exclude location-specific risks.

[0070] On the other hand, the information processing device 100 in the second embodiment uses the position information of an object as input to the model, and therefore has the disadvantage that errors in position detection have a large effect on determining the state of an object.

[0071] [Configuration of information processing device] The acquisition unit 133 acquires information on time series data of signal levels acquired from sensors, time series data of object positions acquired from sensors, and time series data of risks based on the object positions.

[0072] For example, the acquisition unit 133 acquires time-series data of signal levels acquired from the sensor, "Count0: 5.9, Count1: 6.0, Count2: 6.1, Count3...," and object coordinates acquired from the sensor, "Count0: (X0, Y0), Count1: (X1, Y1), Count2: (X2, Y2), Count3...," from the sensor unit 131, and time-series data of risk based on the object's position, "Count0: 3, Count1: 3, Count2: 4, Count3...," from the calculation unit 132. Note that the risk acquired by the acquisition unit 133 from the calculation unit 132 may be the distance risk itself, the directional risk itself, or a product of the distance risk and the directional risk.

[0073] The creation unit 134 creates image data that time-series represents the time-series data of the signal level, the time-series data of the object position, and the time-series data of the risk acquired by the acquisition unit 133. For example, the creation unit 134 creates image data by plotting, at the position of the object, a color that combines components that make up a color corresponding to each of the elements of the signal level, risk, and time acquired by the acquisition unit 133.

[0074] Here, the image data created by the creation unit 134 will be described with reference to Figs. 9 and 10. Figs. 9 and 10 are diagrams showing an example of the creation process by the creation unit 134. Fig. 9 shows an example of the correspondence between each piece of information included in the image data created by the creation unit 134 and a color. For example, as shown in Fig. 9, the creation unit 134 converts the signal level to red, the time (Count) to green, and the risk to blue. Then, the creation unit 134 creates image data by plotting the color obtained by combining the converted RGB colors at the position of the object.

[0075] More specifically, the creation unit 134 converts signal level: 5 to "R: 130", time: Count 0 to "G: 0", and risk: 4 to "B: 240", and plots the color identified from "R: 130, G: 0, B: 240" on the object's position coordinates "XY".

[0076] 10 shows image data in which colors composed of components corresponding to the signal level, risk, and time are plotted at the position of an object. For example, when an object approaches the vehicle 10 and the signal level is converted to red, the time to green, and the risk to blue, the creation unit 134 creates image data in which the closer the position of each object is to the vehicle 10, the stronger the red and blue colors (for example, purple) are plotted, since the signal level and risk values ​​increase as the object approaches the vehicle 10.

[0077] Also, for example, if an object spying on the vehicle 10 converts the signal level to red, the time to green, and the risk to blue, the creation unit 134 creates image data in which strong red and blue colors are plotted at the position of the object located near the vehicle 10, since the signal level value and risk value have increased due to the spying.

[0078] Also, for example, if an object intrudes into the interior of the vehicle 10 and the signal level is converted to red, the time to green, and the risk to blue, the creation unit 134 creates image data in which extremely strong red and blue colors are plotted at the position of the object present inside the vehicle 10, since the signal level value and risk value have become extremely high due to the intrusion.

[0079] Note that using red, green, and blue as colors to correspond to the signal level, time, and risk contained in the image data is merely an example, and color combinations such as cyan, magenta, and yellow may also be used.

[0080] When converting the signal level, risk, and time elements acquired by the acquisition unit 133 into components that make up a color, the creation unit 134 multiplies the value indicating each element by a predetermined coefficient. For example, in the RGB format, while each color takes on 256 values ​​from 0 to 255, if the dynamic range is different, such that the signal level takes on values ​​from 0 to 10 and the risk takes on values ​​from 0 to 4, the signal level may be multiplied by 25 and the risk by 60, for example, to convert into components that make up a color in order to use a wide range of values ​​that each color can take.

[0081] The creation unit 134 creates image data so that one image data piece includes time-series data of signal levels for a predetermined period, time-series data of object positions, and time-series data of risk. For example, the creation unit 134 creates image data so that one image data piece includes time-series data of signal levels for 2000 milliseconds, time-series data of object positions for 2000 milliseconds, and time-series data of risk for 2000 milliseconds.

[0082] The creation unit 134 creates image data so that one image data set includes time-series data of a predetermined number of signal levels, time-series data of object positions, and time-series data of risks. For example, the creation unit 134 creates image data so that one image data set includes time-series data of 40 signal levels, time-series data of 40 object positions, and time-series data of 40 risks.

[0083] The learning unit 135 causes the third learning model to learn the relationship between the information on the time-series data of the signal level, the time-series data of the object's position, and the time-series data of the risk, and the state of the object. For example, the learning unit 135 causes the third learning model to learn the state of the object, such as approach, reconnaissance, or intrusion, in response to changes in the signal level, the object's position, and the risk over a certain period of time. Note that the time-series data of the signal level, the time-series data of the object's position, and the time-series data of the risk may be included in the image data. For example, the learning unit 135 causes the third learning model to learn the relationship between the image data including the time-series data of the signal level, the time-series data of the object's position, and the time-series data of the risk, and the state of the object for each image data.

[0084] The generation unit 136 generates information indicating the state of the object using a trained model based on the time-series data of the signal level, the time-series data of the object position, and the time-series data of the risk acquired by the acquisition unit 133. For example, the generation unit 136 generates information indicating the state of the object using a third trained model that receives as input the time-series data of the signal level, the time-series data of the object position, and the time-series data of the risk acquired by the acquisition unit 133, and outputs the state of the object in response to the input of the time-series data of the signal level, the time-series data of the object position, and the time-series data of the risk.

[0085] More specifically, the generation unit 136 inputs the signal level time series data "Count0:3, Count1:3.1, Count2:3.1, Count3...", the object position time series data "Count0:(X0,Y0), Count1:(X1,Y1), Count2:(X2,Y2), Count3...", and the risk time series data "Count0:3, Count1:3, Count2:4, Count3..." acquired by the acquisition unit 133 into a third trained model that outputs object information in response to the input of the signal level time series data, the object position time series data, and the risk time series data, to generate object information "approaching".

[0086] The generation unit 136 receives the image data created by the creation unit 134 as input, and generates information indicating the state of the object using a third trained model that outputs information indicating the state of the object in response to input image data that represents time series data of the signal level, time series data of the object's position, and time series data of the risk in a time series manner.

[0087] More specifically, the generation unit 136 inputs image data (for example, the image data shown in Figure 10) that represents the signal level time series data "Count0:4, Count1:4.1, Count2:4.1, Count3...", the object position time series data "Count0:(X0,Y0), Count1:(X1,Y1), Count2:(X2,Y2), Count3...", and the risk time series data "Count0:3, Count1:3, Count2:4, Count3..." acquired by the acquisition unit 133 in a time series manner into a third trained model that outputs object information in response to input of image data representing the signal level time series data, the object position time series data, and the risk time series data, thereby generating object information "reconnaissance".

[0088] 〔flowchart〕 Next, processing by information processing device 100 configured as described above will be described with reference to the flowchart in Fig. 11. The flowchart in Fig. 11 is mainly executed by control unit 130. Furthermore, this flowchart can be configured as a program executed by a CPU included in control unit 130, thereby forming an information processing program.

[0089] First, the acquisition unit 133 acquires time-series data of signal levels acquired from sensors, time-series data of object positions acquired from sensors, and time-series data of risks based on the object positions (step S201).

[0090] Next, the creating unit 134 creates image data representing the time series data of the signal level, the time series data of the object position, and the time series data of the risk acquired by the acquiring unit 133 (step S202).

[0091] Then, the generation unit 136 takes the image data created by the creation unit 134 as input and generates information indicating the state of the object using a trained model that outputs information indicating the state of the object in response to input image data representing time series data of the signal level, time series data of the object's position, and time series data of the risk (step S203).

[0092] 〔effect〕 The information processing device 100 according to the embodiment is characterized by having an acquisition unit 133 that acquires time series data of signal levels acquired from a sensor, time series data of an object's position acquired from the sensor, and time series data of a risk based on the object's position, and a generation unit 136 that generates information indicating the state of an object using a trained model based on the time series data of signal levels, the time series data of the object's position, and the time series data of the risk acquired by the acquisition unit 133.

[0093] As a result, the information processing device 100 generates information on the state of an object using time series data of the signal level, time series data of the object's position, and time series data of the risk as inputs to a model, thereby preventing a decrease in the accuracy of determining the state of an object due to errors in the sensor's recognition position compared to when only the object's position information is used, and can determine the state of an object with higher accuracy compared to when only the object's position data is used.

[0094] In addition, the information processing device 100 according to the embodiment further includes a creation unit 134 that creates image data that time-series represents the time-series data of the signal level acquired by the acquisition unit 133, the time-series data of the object's position, and the time-series data of the risk, and the generation unit 136 uses the image data created by the creation unit 134 as input and generates information that indicates the state of the object using a trained model that outputs information that indicates the state of the object in response to input of image data that represents the time-series data of the signal level, the time-series data of the object's position, and the time-series data of the risk.

[0095] As a result, the information processing device 100 determines the state of an object by using image data including time series data of signal level, time series data of object position, and time series data of risk as input to the model, thereby enabling processing using an image recognition model and enabling the state of the object to be determined with high accuracy.

[0096] In addition, the creation unit 134 of the information processing device 100 according to the embodiment creates image data by plotting a color that combines components that make up a color corresponding to each of the elements of signal level, risk, and time acquired by the acquisition unit 133 at the position of the object.

[0097] As a result, the information processing device 100 creates image data for each object position that represents differences in the values ​​of each element, namely signal level, risk, and time, as differences in color, and enables processing using an image recognition model, thereby enabling the state of the object to be determined with high accuracy.

[0098] In addition, when converting each element of the signal level, risk, and time acquired by the acquisition unit 133 into components that make up a color, the creation unit 134 of the information processing device 100 according to the embodiment multiplies the value indicating each element by a predetermined coefficient.

[0099] As a result, the information processing device 100 performs scaling when converting each element of signal level, risk, and time into components that make up color, creates image data in which differences in the values ​​of each element are more clearly expressed as differences in color, and enables processing using an image recognition model, thereby making it possible to accurately determine the state of an object.

[0100] Furthermore, the creation unit 134 of the information processing device 100 according to the embodiment creates image data so that one piece of image data includes time-series data of signal levels for a predetermined period, time-series data of object positions, and time-series data of risks.

[0101] As a result, the information processing device 100 can create a single image data set to include data on the signal level for a specified period, the object's position, and risk, and by using the image data divided at an appropriate granularity as input to the model, it can more accurately determine the state of the object.

[0102] Furthermore, the creation unit 134 of the information processing device 100 according to the embodiment creates image data so that one image data includes time series data of a predetermined number of signal levels, time series data of object positions, and time series data of risks.

[0103] As a result, the information processing device 100 creates a single image data set to include a predetermined number of signal levels, object position, and risk data, and by using the image data divided at an appropriate granularity as input to the model, it is possible to more accurately determine the state of the object.

[0104] [Third embodiment] In the second embodiment, an example was described in which image data including information on signal level time series data, object position time series data, and risk time series data is used as input to generate information indicating the state of an object, but in the following third embodiment, an example will be described in which image data including information on signal level time series data, object position time series data, distance risk time series data, and directional risk time series data is used as input to a model to generate information indicating the state of an object. Note that descriptions of content common to the first or second embodiment will be omitted as appropriate.

[0105] [System configuration] The information processing device 100 acquires, for example, time series data of signal levels acquired from a sensor, time series data of object positions acquired from the sensor, time series data of distance risk based on the object positions, and time series data of directional risk based on the object positions, and generates information indicating the state of the object using a trained model based on the acquired time series data of signal levels, time series data of object positions, time series data of distance risk, and time series data of directional risk.

[0106] As a result, the information processing device 100 inputs the time series data of the signal level, the time series data of the object's position, the time series data of the distance risk, and the time series data of the directional risk into a trained model to generate the state of the object, thereby preventing a decrease in the accuracy of determining the state of the object due to errors in the sensor's recognition position and enabling the state of the object to be determined with high accuracy compared to when only the object's position information is used.

[0107] Furthermore, the information processing device 100 in the first embodiment does not use object position information as input to the model, which has the disadvantage of not being able to detect location-specific risks such as those near the tires, or to exclude location-specific risks caused by objects such as flags that are often found behind the vehicle. However, the information processing device 100 in the third embodiment uses object position information as input to the model, making it possible to detect or exclude location-specific risks.

[0108] On the other hand, the information processing device 100 in the third embodiment uses the position information of an object as input to the model, and therefore has the disadvantage that errors in position detection have a large effect on determining the state of an object.

[0109] [Configuration of information processing device] The acquisition unit 133 acquires time series data of signal levels acquired from the sensor, time series data of object positions acquired from the sensor, time series data of distance risk based on the object positions, and time series data of directional risk based on the object positions.

[0110] For example, the acquisition unit 133 acquires from the sensor unit 131 time series data of the signal level acquired from the sensor, "Count0: 2.1, Count1: 2.1, Count2: 2.2, Count3...", and the coordinates of the object acquired from the sensor, "Count0: (X0, Y0), Count1: (X1, Y1), Count2: (X2, Y2), Count3...", and from the calculation unit 132 time series data of the distance risk based on the object's position, "Count0: 0.3, Count1: 0.3, Count2: 0.4, Count3...", and time series data of the direction risk based on the object's position, "Count0: 3, Count1: 3, Count2: 4, Count3...".

[0111] The creation unit 134 creates image data that represents, in a time series, the time series data of the signal level, the time series data of the object position, the time series data of the distance risk, and the time series data of the directional risk acquired by the acquisition unit 133. For example, the creation unit 134 creates image data that represents a change in the object position using a color that combines components that make up a color that corresponds to each of the elements of the signal level, the distance risk, and the directional risk acquired by the acquisition unit 133.

[0112] Here, the image data creation process by the creation unit 134 will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the creation process by the creation unit 134. Fig. 12 shows the correspondence between each piece of information and color included in the image data created by the creation unit 134, and image data that represents a change in the position of an object using colors that are combinations of the corresponding colors.

[0113] For example, first, the creation unit 134 converts the directional risk into red, the distance risk into green, and the signal level into blue. Then, the creation unit 134 creates image data that displays lines connecting the position coordinates of a predetermined number of consecutive objects in a color that combines the converted RGB colors.

[0114] More specifically, the creation unit 134 converts the direction risk: 4 to "R: 240", the distance risk: 0.8 to "G: 200", and the signal level: 8 to "B: 80", and creates image data showing a line connecting the position coordinates of two consecutive objects in a color determined from "R: 240, G: 200, B: 80".

[0115] Taking the case where an object approaches the vehicle 10 as an example, the creation unit 134 creates image data that displays a line connecting successive position coordinates of the object approaching the vehicle 10 using a color with strong red, green, and blue hues (for example, white), as shown in Figure 12, because the values ​​of the signal level, distance risk, and directional risk increase due to the approach.

[0116] Next, as an example of a case where an object is spying on the vehicle 10, the creation unit 134 creates image data that displays a line connecting consecutive position coordinates of objects present near the vehicle 10 using a color with strong green and blue hues (e.g., yellow), since the signal level and distance risk values ​​have become extremely high due to the intrusion.

[0117] Next, as an example of a case where an object has entered the interior of the vehicle 10, the creation unit 134 creates image data that displays lines connecting the successive position coordinates of objects present inside the vehicle 10 using colors with extremely strong green and blue hues, since the signal level and distance risk values ​​have become extremely high due to the intrusion.

[0118] When displaying a line connecting the position coordinates of a predetermined number of consecutive objects in a color specified by each of the elements of directional risk, distance risk, and signal level, the creation unit 134 can use a color specified by each of the elements of directional risk, distance risk, and signal level at a time corresponding to any of the position coordinates of the predetermined number of consecutive objects. For example, the creation unit 134 can use a color specified by each of the elements of directional risk, distance risk, and signal level at a time corresponding to the position coordinates of the object that is earliest or latest in time among the position coordinates of the predetermined number of consecutive objects.

[0119] Furthermore, although the above example shows a case where a change in the position of an object is displayed using one color specified from each of the RGB components, for example, a change in the position of an object may be displayed using three colors specified from each of the RGB components. More specifically, the creation unit 134 creates image data that displays lines connecting the position coordinates of a predetermined number of consecutive objects using three colors: a color specified from "R:240, G:0, B:0" corresponding to the directional risk, a color specified from "R:0, G:200, B:0" corresponding to the distance risk, and a color specified from "R:0, G:0, B:80" corresponding to the signal level.

[0120] Furthermore, using red, green, and blue as colors to correspond to the directional risk, distance risk, and signal level contained in the image data is merely an example, and color combinations such as cyan, magenta, and yellow may also be used to correspond.

[0121] There are many possible variations in the display mode of the change in the position of an object. For example, the creation unit 134 creates image data that represents the change in the position of an object using a graphic of each component color that makes up the color corresponding to each element. More specifically, the creation unit 134 creates image data that represents the directional risk with an arrow of the red component, the distance risk with an arrow of the green component, and the signal level with an arrow of the blue component. Note that the graphic that represents each element can be a line, an arrow, a polygon, a circle, or any other shape depending on the purpose.

[0122] Here, the image data creation process by the creation unit 134 will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the creation process by the creation unit 134. Fig. 13 illustrates changes in the positions of the vehicle 10 and objects using graphics. For example, Fig. 13 illustrates changes in the position of an object approaching and invading the vehicle 10, and changes in the position of an object reconnaissance the vehicle 10 and then moving away from the vehicle 10, using arrows and rectangles.

[0123] The arrow is, for example, a line connecting the position coordinates of a predetermined number of consecutive objects, and indicates the position coordinates of the object with the smallest Count value among the predetermined number of consecutive objects as the start point of the arrow and the position coordinates of the object with the largest Count value as the end point of the arrow. For example, the arrow is displayed in a color specified by the value R, which corresponds to red for directional risk, and a color specified by the value B, which corresponds to blue for signal level.

[0124] The rectangle is, for example, a quadrangle with a diagonal line passing through the start point and end point of the arrow, and is displayed in a color specified by the value of G, which corresponds to the distance risk as green.

[0125] That is, in the example of Figure 13, when the position of an object changes in a direction approaching vehicle 10, the directional risk is indicated by a reddish arrow, the distance risk is indicated by a greenish arrow, and the signal level is indicated by a blueish rectangle.

[0126] On the other hand, if the object's position changes in a direction away from the vehicle 10, the directional risk is indicated by a reddish arrow with a weaker red tint, the distance risk is indicated by a greenish arrow with a weaker green tint, and the signal level is indicated by a bluish rectangle with a weaker blue tint.

[0127] The creation unit 134 further creates image data including time information. For example, the creation unit 134 creates image data that represents time information using gradation of components that make up the color corresponding to each element. For example, the creation unit 134 creates image data that displays a red arrow representing a directional risk by setting the value of R representing the directional risk to the maximum value of the red component, decreasing the value of the red component when the value of Count is small, and increasing the value of the red component when the value of Count is large.

[0128] For example, for a rectangle showing a signal level displayed in blue, image data is created in which B, which represents the signal level, is set to the maximum value of the blue component, and if the Count value is small, the value of the blue component is small, and if the Count value is large, the value of the blue component is large.

[0129] As another example, the creation unit 134 creates image data that represents time information by the thickness of the lines of a graphic. More specifically, the creation unit 134 creates image data that displays a red arrow representing a directional risk so that the smaller the value of Count, the thinner it is, and the larger the value, the thicker it is.

[0130] When converting each element of the signal level, distance risk, and directional risk acquired by the acquisition unit 133 into components constituting a color, the creation unit 134 multiplies the value indicating each element by a predetermined coefficient. For example, in the RGB format, while each color takes on 256 values ​​from 0 to 255, if the dynamic range is different, such as the signal level taking on a value from 0 to 10, the distance risk taking on a value from 0 to 1, and the directional risk taking on a value from 0 to 4, in order to use a wide range of values ​​that each color can take, the signal level may be multiplied by 25, the distance risk by 255, and the directional risk by 60, for example, to convert into components constituting a color.

[0131] The creation unit 134 creates image data so that one image data set includes time-series data of signal levels for a predetermined period, time-series data of object positions, time-series data of distance risks, and time-series data of directional risks. For example, the creation unit 134 creates image data so that one image data set includes time-series data of signal levels for 2000 milliseconds, time-series data of object positions for 2000 milliseconds, time-series data of distance risks for 2000 milliseconds, and time-series data of directional risks for 2000 milliseconds.

[0132] The creation unit 134 creates image data so that one image data set includes a predetermined number of time-series data of signal levels, time-series data of object positions, time-series data of distance risks, and time-series data of directional risks. For example, the creation unit 134 creates image data so that one image data set includes 40 time-series data of signal levels, 40 time-series data of object positions, 40 time-series data of distance risks, and 40 time-series data of directional risks.

[0133] The learning unit 135 causes the fourth learning model to learn the relationship between the information on the time-series data of the signal level, the time-series data of the object position, the time-series data of the distance risk, and the time-series data of the direction risk, and the state of the object. For example, the learning unit 135 causes the fourth learning model to learn the state of the object, such as approach, reconnaissance, or intrusion, in response to changes in the signal level, the object position, the distance risk, and the direction risk over a certain period of time.

[0134] The time-series data of the signal level, the time-series data of the object position, the time-series data of the distance risk, and the time-series data of the directional risk may be included in the image data. For example, the learning unit 135 causes the fourth learning model to learn the relationship between the image data including the time-series data of the signal level, the time-series data of the object position, the time-series data of the distance risk, and the time-series data of the directional risk, and the state of the object for each image data.

[0135] The generation unit 136 generates information indicating the state of the object using a trained model based on the time series data of the signal level acquired by the acquisition unit 133, the time series data of the object's position, the time series data of the distance risk, and the time series data of the directional risk.

[0136] For example, the generation unit 136 receives as input the time series data of the signal level, the time series data of the object's position, the time series data of the distance risk, and the time series data of the directional risk acquired by the acquisition unit 133, and generates information indicating the state of the object using a fourth trained model that outputs the state of the object in response to the input of the time series data of the signal level, the time series data of the object's position, the time series data of the distance risk, and the time series data of the directional risk.

[0137] More specifically, the generation unit 136 inputs the signal level time series data "Count0:3.5, Count1:3.5, Count2:3.6, Count3...", the object position time series data "Count0:(X0,Y0), Count1:(X1,Y1), Count2:(X2,Y2), Count3...", the distance risk direction time series data "Count0:0.4, Count1:0.5, Count2:0.6, Count3...", and the risk time series data "Count0:4, Count1:4, Count2:4, Count3..." acquired by the acquisition unit 133 into a fourth trained model that outputs object information in response to input of the signal level time series data, the object position time series data, the distance risk time series data, and the direction risk time series data, to generate object information "approaching".

[0138] The generation unit 136 receives the image data created by the creation unit 134 as input, and generates information indicating the state of the object using a trained model that outputs information indicating the state of the object in response to input image data that represents time series data of the signal level, time series data of the object's position, time series data of the distance risk, and time series data of the direction risk in a time series manner.

[0139] More specifically, the generation unit 136 inputs image data (for example, image data shown in Figure 13 or 14) that represents the signal level time series data "Count0:1, Count1:1.5, Count2:2, Count3...", the object position time series data "Count0:(X0,Y0), Count1:(X1,Y1), Count2:(X2,Y2), Count3...", the distance risk time series data "Count0:0.2, Count1:0.2, Count2:0.2, Count3...", and the directional risk time series data "Count0:4, Count1:4, Count2:4, Count3..." acquired by the acquisition unit 133, into a fourth trained model that outputs object information in response to input of image data that represents the signal level time series data, the object position time series data, the distance risk time series data, and the directional risk time series data in time series, thereby generating object information "approaching".

[0140] 〔flowchart〕 Next, processing by the information processing device 100 configured as described above will be described with reference to the flowchart in Fig. 14. The flowchart in Fig. 14 is mainly executed by the control unit 130. Furthermore, this flowchart can be configured as a program executed by the CPU of the control unit 130, thereby forming an information processing program.

[0141] First, the acquisition unit 133 acquires time series data of the signal level acquired from the sensor, time series data of the object position acquired from the sensor, time series data of the distance risk based on the object position, and time series data of the direction risk based on the object position (step S301).

[0142] Next, the creation unit 134 creates image data that represents the time series data of the signal level acquired by the acquisition unit 133, the time series data of the object position, the time series data of the distance risk, and the time series data of the direction risk in a time series manner (step S302).

[0143] Then, the generation unit 136 takes the image data created by the creation unit 134 as input, and generates information indicating the state of the object using a trained model that outputs information indicating the state of the object in response to input image data that time-series data of the signal level, time-series data of the object's position, time-series data of the distance risk, and time-series data of the direction risk (step S303).

[0144] 〔effect〕 The information processing device 100 according to the embodiment includes an acquisition unit 133 that acquires time series data of signal levels acquired from a sensor, time series data of an object's position acquired from the sensor, time series data of a distance risk based on the object's position, and time series data of a directional risk based on the object's position, and a generation unit 136 that generates information indicating the state of an object using a trained model based on the time series data of signal levels, the time series data of the object's position, the time series data of the distance risk, and the time series data of the directional risk acquired by the acquisition unit 133.

[0145] As a result, the information processing device 100 generates information on the state of the object using time series data of the signal level, time series data of the object's position, time series data of the distance risk, and time series data of the directional risk as inputs to the model, thereby preventing a decrease in the accuracy of determining the state of the object due to errors in the sensor's recognition position compared to when only the object's position information is used, and can determine the state of the object with higher accuracy compared to when only the object's position data is used.

[0146] In addition, the information processing device 100 according to the embodiment further includes a creation unit 134 that creates image data that time-series represents the time series data of the signal level acquired by the acquisition unit 133, the time series data of the object position, the time series data of the distance risk, and the time series data of the directional risk. The generation unit 136 receives the image data created by the creation unit 134 as input and generates information that indicates the state of the object using a trained model that outputs information that indicates the state of the object in response to input of image data that time-series represents the time series data of the signal level, the time series data of the object position, the time series data of the distance risk, and the time series data of the directional risk.

[0147] As a result, the information processing device 100 determines the state of an object by using an image including time series data of signal level, time series data of object position, time series data of distance risk, and time series data of directional risk as input to the model, thereby enabling processing using an image recognition model and enabling the state of an object to be determined with high accuracy.

[0148] In addition, the creation unit 134 of the information processing device 100 according to the embodiment creates image data that represents changes in the position of an object using colors that combine components that make up colors corresponding to the signal level, distance risk, and directional risk elements acquired by the acquisition unit 133.

[0149] As a result, the information processing device 100 creates image data in which the differences in the values ​​of each element, namely signal level, distance risk, and directional risk, are represented as different colors in response to changes in the object's position, and by enabling processing using an image recognition model, it is possible to accurately determine the state of the object.

[0150] Furthermore, the creation unit 134 of the information processing apparatus 100 according to the embodiment creates image data that represents a change in the position of an object using a graphic of the color of each component that makes up the color corresponding to each element.

[0151] As a result, the information processing device 100 can create image data that more clearly shows the differences between each element by displaying the figure in a color corresponding to each element, and by enabling processing using an image recognition model, it can accurately determine the state of the object.

[0152] Moreover, the creation unit 134 of the information processing apparatus 100 according to the embodiment further creates image data including time information.

[0153] As a result, the information processing device 100 creates image data that further includes time information, and represents changes in each element of signal level, object position, distance risk, and directional risk over time in an image, enabling processing using an image recognition model, thereby enabling the state of the object to be determined with high accuracy.

[0154] Furthermore, the creation unit 134 of the information processing apparatus 100 according to the embodiment creates image data that represents time information using gradation of components that make up the color corresponding to each element.

[0155] As a result, the information processing device 100 can represent time information using a gradation of the components that make up the color corresponding to each element, thereby creating image data that more clearly shows the changes in each element of signal level, object position, distance risk, and directional risk over time, and by enabling processing using an image recognition model, it can accurately determine the state of the object.

[0156] Moreover, the creation unit 134 of the information processing apparatus 100 according to the embodiment creates image data that expresses time information by the thickness of lines of a graphic.

[0157] As a result, the information processing device 100 can represent time information by the thickness of the lines of the figure, and in addition to changes in the elements of signal level, object position, distance risk, and directional risk over time, it can create image data that more clearly shows the passage of time, and by enabling processing using an image recognition model, it can accurately determine the state of the object.

[0158] In addition, when the creation unit 134 of the information processing device 100 according to the embodiment converts each element of the signal level, distance risk, and directional risk acquired by the acquisition unit 133 into components that make up a color, it multiplies the value indicating each element by a predetermined coefficient.

[0159] As a result, the information processing device 100 performs scaling when converting each element of the signal level, distance risk, and directional risk into components that make up color, creates image data in which differences in the values ​​of each element are more clearly expressed as differences in color, and enables processing using an image recognition model, thereby making it possible to accurately determine the state of an object.

[0160] In addition, the creation unit 134 of the information processing device 100 according to the embodiment creates image data so that one image data includes time series data of signal levels for a predetermined period of time, time series data of object positions, time series data of distance risk, and time series data of directional risk.

[0161] As a result, the information processing device 100 creates a single image data set to include data on the signal level for a specified period, the object's position, distance risk, and directional risk, and by using the image data divided at an appropriate granularity as input to the model, it is possible to more accurately determine the state of the object.

[0162] In addition, the creation unit 134 of the information processing device 100 according to the embodiment creates image data so that one image data includes time series data of a predetermined number of signal levels, time series data of an object's position, time series data of distance risk, and time series data of direction risk.

[0163] As a result, the information processing device 100 creates one image data set to include a predetermined number of signal levels, object position, distance risk, and directional risk data, and by using the image data divided at an appropriate granularity as input to the model, it is possible to more accurately determine the state of the object.

[0164] [Variations] [System configuration] Up to this point, an example has been described in which the information processing according to the first to third embodiments is realized by the information processing device 100 provided in the vehicle 10. Below, a modified example of the above embodiments will be described. As a modified example, the information processing according to the first to third embodiments may be realized by communication between the information processing device 100 existing on the cloud and an in-vehicle device provided in the vehicle 10.

[0165] For example, if the in-vehicle device is an edge computer that performs edge processing near the user, the information processing device 100 may be, for example, a cloud computer that performs processing on the cloud side. That is, the information processing device 100 may be a server device, and the information processing according to the embodiment is realized in the information processing system 1 by transmitting and receiving information between the information processing device 100, which is a server device existing on the cloud, and the in-vehicle device provided in the vehicle 10.

[0166] The in-vehicle device may be a dedicated sensor device built into or externally attached to the vehicle 10, or may be a recording device (drive recorder) or other device installed in the vehicle 10 for crime prevention or to combat aggressive driving.

[0167] The in-vehicle device may also be configured with a sensor device and a notification device. As an example, the in-vehicle device may be a composite device in which a sensor device and a notification device that are independent of each other are connected to each other so as to be able to communicate with each other. As another example, the in-vehicle device may be a single device having a sensor function and a notification function.

[0168] Furthermore, a user can connect a predetermined sensor to a portable terminal device (e.g., a smartphone, tablet terminal, notebook PC, desktop PC, PDA, etc.) that they use on a daily basis and install a predetermined application to use it as an in-vehicle device. For example, a portable terminal device that is equipped with a predetermined sensor or to which a predetermined sensor is connected can be understood as an in-vehicle device as used herein. When a portable terminal device is used as an in-vehicle device, it is installed, for example, on the dashboard of the vehicle 10 while driving.

[0169] [others] [Hardware configuration] The information processing device 100 according to the embodiment described above is realized, for example, by a computer 1000 configured as shown in Fig. 15. Fig. 15 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0170] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0171] The HDD 1400 stores programs executed by the CPU 1100 and data used by the programs. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0172] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0173] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0174] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0175] 〔others〕 Although one example of an embodiment of the present invention has been described above, the present invention is not limited to the above example. In other words, a person skilled in the art can implement various modifications in accordance with conventionally known knowledge without departing from the gist of the present invention. As long as such modifications still include the information processing device of the present invention, they are of course included in the scope of the present invention. [Explanation of symbols]

[0176] 1. Information Processing Systems 10 vehicles 100 Information processing device 110 Communications Department 120 Storage section 130 control section 131 Sensor unit 132 Calculation Unit 133 Acquisition Department 134 Creation Department 135 Learning Department 136 Generation part

Claims

1. an acquisition unit that acquires time-series data of signal levels acquired from a sensor, time-series data of object positions acquired from the sensor, and time-series data of risks based on the object positions; a generation unit that generates information indicating a state of the object using a trained model based on the time series data of the signal level, the time series data of the object's position, and the time series data of the risk acquired by the acquisition unit; and An information processing device comprising:

2. a creating unit that creates image data that time-series represents the time-series data of the signal level acquired by the acquiring unit, the time-series data of the position of the object, and the time-series data of the risk; Furthermore, The generation unit The image data created by the creation unit is input, and the trained model outputs information indicating the state of an object in response to input of image data that time-series data of a signal level, time-series data of an object's position, and time-series data of a risk, and generates information indicating the state of the object using the trained model.

2. The information processing apparatus according to claim 1, wherein:

3. The creation unit The image data is created by plotting a color obtained by combining components that constitute a color corresponding to each of the elements of the signal level, the risk, and time acquired by the acquisition unit at the position of the object.

3. The information processing apparatus according to claim 2, wherein:

4. The creation unit When converting each element of the signal level, the risk, and the time acquired by the acquisition unit into a component constituting a color, a value indicating each element is multiplied by a predetermined coefficient.

4. The information processing apparatus according to claim 3,

5. The information processing device according to claim 2, characterized in that the creation unit creates the image data so that one image data includes time series data of the signal level for a predetermined period, time series data of the object's position, and time series data of the risk.

6. The information processing device according to claim 2, characterized in that the creation unit creates the image data so that one image data includes time series data of a predetermined number of the signal levels, time series data of the object's position, and time series data of the risk.

7. 1. A computer-implemented method comprising: an acquisition step of acquiring time series data of signal levels acquired from sensors, time series data of object positions acquired from the sensors, and time series data of risks based on the object positions; a generation step of generating information indicating the state of the object using a trained model based on the time series data of the signal level, the time series data of the object's position, and the time series data of the risk acquired in the acquisition step; An information processing method comprising:

8. an acquisition step of acquiring time series data of a signal level acquired from a sensor, time series data of an object position acquired from the sensor, and time series data of a risk based on the object position; a generation step of generating information indicating a state of the object using a trained model based on the time series data of the signal level, the time series data of the object position, and the time series data of the risk acquired in the acquisition step; An information processing program characterized by causing a computer to execute the above.

Images