Information processing device, information processing method, and information processing program

Abstract

To realize an information processing device that makes it easier for users to make judgments about the trajectory of an object. [Solution] The information processing device includes: an object detection unit that detects an object present in a space using sensor information indicating sensing results from a sensor that senses the space; an object tracking unit that tracks the object detected by the object detection unit; a text generation unit that generates text describing the trajectory of the object using the tracking results from the object tracking unit, environmental information relating to the environment of the object, and spatial information representing the space; and an output unit that outputs an image representing the trajectory of the object in the space and the text generated by the text generation unit.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Techniques for tracking the movement of an object or the like are known. As an example of a technique for tracking the movement of an object, for example, the technique described in Patent Document 1 can be cited. The imaging device described in Patent Document 1 images an object, extracts a plurality of feature amounts of the imaged object, determines the priorities of the plurality of extracted feature amounts, determines a feature amount according to the degree of priority and the allowable amount of the output destination, and associates the feature amount with the moving direction and outputs them. Also, a technique for displaying the tracking result of the movement of an object is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When displaying the tracking result of the movement of an object, although the user can grasp the trajectory of the movement of the object, it is difficult to determine whether the displayed trajectory is important (whether it needs to be watched, etc.) or how important the trajectory is just by checking the displayed trajectory. In particular, for example, when a plurality of trajectories are displayed on one screen, it is difficult for the user to determine which trajectory is important. The technique described in Patent Document 1 also has the same problem.

[0005] The present disclosure has been made in view of the above problems, and an exemplary object thereof is to provide a technique that makes it easier for a user to judge the trajectory of an object.

Means for Solving the Problems

[0006] An information processing device relating to an exemplary aspect of this disclosure includes: an object detection means for detecting an object present in a space using sensor information indicating sensing results from a sensor that senses the space; an object tracking means for tracking the object detected by the object detection means; a text generation means for generating text describing the trajectory of the object using the tracking results from the object tracking means, environmental information relating to the environment of the object, and spatial information representing the space; and an output means for outputting an image representing the trajectory of the object in the space and the text generated by the text generation means.

[0007] An information processing device relating to an exemplary aspect of this disclosure comprises: an object detection means for detecting an object present in space using sensor information; an object tracking means for tracking the object detected by the object detection means; a labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking by the object tracking means; and a training means for training a learning model that takes the trajectory and environmental information as input and outputs the importance of the trajectory, using training data including the trajectory labeled by the labeling means and environmental information relating to the environment of the object.

[0008] An information processing method relating to an exemplary aspect of this disclosure includes: an object detection process in which at least one processor detects an object present in a space using sensor information indicating sensing results from a sensor that senses the space; an object tracking process in which the at least one processor tracks the object detected in the object detection process; a text generation process in which the at least one processor generates text describing the trajectory of the object using the tracking results from the object tracking process, environmental information relating to the environment of the object, and spatial information representing the space; and an output process in which the at least one processor outputs an image representing the trajectory of the object in the space and the text generated in the text generation process.

[0009] An illustrative aspect of the present disclosure is an information processing program for causing a computer to function as an information processing device, wherein the computer functions as: an object detection means for detecting an object present in a space using sensor information indicating sensing results from a sensor that senses the space; an object tracking means for tracking the object detected by the object detection means; a text generation means for generating text describing the trajectory of the object using the tracking results from the object tracking means, environmental information relating to the environment of the object, and spatial information representing the space; and an output means for outputting an image representing the trajectory of the object in the space and the text generated by the text generation means.

[0010] An example of an information processing method relating to this disclosure includes: an object detection process in which at least one processor detects an object present in space using sensor information; an object tracking process in which the at least one processor tracks the object detected in the object detection process; a labeling process in which the at least one processor assigns a label indicating importance to the trajectory of the object obtained by tracking in the object tracking process; and a training process in which the at least one processor trains a learning model that takes the trajectory and environmental information as input and outputs the importance of the trajectory, using training data that includes the trajectory labeled by the labeling process and environmental information relating to the environment of the object.

[0011] An illustrative aspect of the present disclosure is an information processing program for causing a computer to function as an information processing device, wherein the computer functions as: an object detection means for detecting an object present in space using sensor information; an object tracking means for tracking the object detected by the object detection means; a labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking by the object tracking means; and a training means for training a learning model that takes the trajectory and environmental information as input and outputs the importance of the trajectory, using training data including the trajectory labeled by the labeling means and environmental information relating to the environment of the object. [Effects of the Invention]

[0012] One illustrative aspect of this disclosure is that it provides a technology that makes it easier for users to make judgments about the trajectory of an object. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing the configuration of the information processing device related to this disclosure. [Figure 2] This is a flowchart showing the flow of the information processing method related to this disclosure. [Figure 3] This is a block diagram showing the configuration of the information processing device related to this disclosure. [Figure 4] This is a flowchart showing the flow of the information processing method related to this disclosure. [Figure 5] This is a block diagram showing the configuration of the information processing device related to this disclosure. [Figure 6] This is a block diagram showing an example of the functional configuration of the information processing device related to this disclosure. [Figure 7] This figure shows specific examples of images output by the image output unit and text output by the text output unit related to this disclosure. [Figure 8] This is a block diagram showing the configuration of a computer that functions as an information processing device related to this disclosure. [Figure 9] This figure shows an example of an image representing the trajectory of the conventional technology. [Modes for carrying out the invention]

[0014] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0015] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0016] (Configuration of information processing device) The configuration of the information processing apparatus 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing apparatus 1. As shown in FIG. 1, the information processing apparatus 1 includes a target detection unit 11, a target tracking unit 12, a text generation unit 13, and an output unit 14. The target detection unit 11 detects a target existing in the space using sensor information indicating the sensing result of a sensor that senses the space. The target tracking unit 12 tracks the target detected by the target detection unit 11. The text generation unit 13 generates text explaining the trajectory of the target using the tracking result by the target tracking unit 12, environmental information regarding the environment of the target, and space information representing the space. The output unit 14 outputs an image representing the trajectory of the target in the space and the text generated by the text generation unit 13.

[0017] (Effect of the information processing apparatus) As described above, in the information processing apparatus 1, there is adopted a configuration including a target detection unit 11 that detects a target existing in the space using sensor information indicating the sensing result of a sensor that senses the space, a target tracking unit 12 that tracks the target detected by the target detection unit 11, a text generation unit 13 that generates text explaining the trajectory of the target using the tracking result by the target tracking unit 12, environmental information regarding the environment of the target, and space information representing the space, and an output unit 14 that outputs an image representing the trajectory of the target in the space and the text generated by the text generation unit 13. Therefore, according to the information processing apparatus 1, there is obtained an effect that it is possible to facilitate the user's judgment regarding the trajectory of the target.

[0018] (Flow of the information processing method) The flow of the information processing method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the information processing method S1. As shown in Figure 2, the information processing method S1 includes an object detection process S11, an object tracking process S12, a text generation process S13, and an output process S14. In the object detection process S11, at least one processor detects an object present in a space using sensor information indicating the sensing result from a sensor that senses the space. In the object tracking process S12, the at least one processor tracks the object detected in the object detection process S11. In the text generation process S13, the at least one processor generates text describing the trajectory of the object using the tracking result from the object tracking process S12, environmental information about the environment of the object, and spatial information representing the space. In the output process S14, the at least one processor outputs an image representing the trajectory of the object in the space and the text generated in the text generation process S13.

[0019] (Effects of information processing methods) As described above, the information processing method S1 employs a configuration that includes: an object detection process S11 in which at least one processor detects an object present in a space using sensor information indicating the sensing result from a sensor that senses the space; an object tracking process S12 in which the at least one processor tracks the object detected in the object detection process S11; a text generation process S13 in which the at least one processor generates text describing the trajectory of the object using the tracking result from the object tracking process S12, environmental information relating to the environment of the object, and spatial information representing the space; and an output process S14 in which the at least one processor outputs an image representing the trajectory of the object in the space and the text generated in the text generation process S13. Therefore, the information processing method S1 has the effect of making it easier for the user to make a judgment about the trajectory of an object.

[0020] (Configuration of information processing device) The configuration of the information processing device 2 will be explained with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the information processing device 2. As shown in Figure 3, the information processing device 2 comprises an object detection unit 21, an object tracking unit 22, a labeling unit 23, and a training unit 24. The object detection unit 21 detects objects present in space using sensor information. The object tracking unit 22 tracks the objects detected by the object detection unit 21. The labeling unit 23 assigns labels indicating importance to the trajectories of the objects obtained by the tracking of the objects by the object tracking unit 22. The training unit 24 uses training data, which includes the trajectories labeled by the labeling unit 23 and environmental information about the environment of the objects, to train a learning model that takes the trajectories and environmental information as input and outputs the importance of the trajectories.

[0021] (Effects of information processing equipment) As described above, the information processing device 2 employs a configuration comprising: an object detection unit 21 that detects objects present in space using sensor information; an object tracking unit 22 that tracks the objects detected by the object detection unit 21; a labeling unit 23 that assigns labels indicating importance to the trajectories of the objects obtained by the tracking of the objects by the object tracking unit 22; and a training unit 24 that trains a learning model that takes the trajectories and environmental information as input and outputs the importance of the trajectories, using training data that includes the trajectories labeled by the labeling unit 23 and environmental information about the environment of the objects. Therefore, the information processing device 2 has the effect of being able to generate a learning model that makes it easier for users to make judgments about the trajectories of objects.

[0022] (Information processing flow) The flow of the information processing method S2 will be explained with reference to Figure 4. Figure 2 is a flowchart showing the flow of the information processing method S2. As shown in Figure 2, the information processing method S2 includes the target detection process S21, the target tracking process S22, the labeling process S23, and the training process S24.

[0023] In the object detection process S21, at least one processor detects an object present in space using sensor information. In the object tracking process S22, the at least one processor tracks the object detected in the object detection process S21. In the labeling process S23, the at least one processor assigns a label indicating importance to the trajectory of the object obtained through tracking in the object tracking process S22. The at least one processor trains a learning model using training data that includes the trajectory labeled in the labeling process S23 and environmental information about the object's environment, taking the trajectory and environmental information as input and outputting the importance of the trajectory.

[0024] (Effects of information processing methods) As described above, the information processing method S2 employs a configuration that includes: an object detection process S21 in which at least one processor detects an object present in space using sensor information; an object tracking process S22 in which at least one processor tracks the object detected in the object detection process S21; a labeling process S23 in which at least one processor assigns a label indicating importance to the trajectory of the object obtained by tracking in the object tracking process S22; and a training process S24 in which at least one processor trains a learning model that takes the trajectory and environmental information as input and outputs the importance of the trajectory, using training data that includes the trajectory labeled by the labeling process S23 and environmental information about the environment of the object. Therefore, the information processing method S2 has the effect of generating a learning model that makes it easier for the user to make judgments about the trajectory of an object.

[0025] [Second exemplary embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0026] (Configuration of information processing device) Figure 5 is a block diagram showing the configuration of the information processing device 1A according to this disclosure. The information processing device 1A is a device that tracks an object and presents the object's movement trajectory to the user. Examples of objects include aircraft, ships, drones, automobiles, robots, people, and animals. However, the objects are not limited to these. The information processing device 1A also highlights and presents important trajectories among multiple trajectories to the user, and presents information corresponding to the important trajectories to the user in text. Examples of text corresponding to the trajectories include reports and instructions related to operations in aircraft control towers, reports and instructions related to operations in security operations in commercial facilities and hospitals, etc. More specifically, the text corresponding to the trajectory may include, for example, text that explains how the aircraft moved and text that shows the predicted result of how it will move. The user makes a decision about the object's trajectory (to issue a warning, to perform rescue, etc.) by reviewing the presented information.

[0027] As shown in Figure 5, the information processing device 1A comprises a control unit 10A, a storage unit 20A, a communication unit 30A, an input unit 40A, and an output unit 50A. The communication unit 30A communicates with external devices of the information processing device 1A via a communication line N. The communication unit 30A transmits data supplied from the control unit 10A to other devices and supplies data received from other devices to the control unit 10A.

[0028] (Input section / Output section) The input unit 40A is configured to receive input to the information processing device 1A, and may include, for example, an input device such as a keyboard, mouse, touch panel, camera, or microphone. The input unit 40A may also be configured to receive data from the input device via an interface such as USB (Universal Serial Bus). The output unit 50A is configured to output from the information processing device 1A, and may include, for example, an output device such as a display, printer, touch panel, or speaker. The output unit 50A may also be configured to have an interface such as USB and output data to the output device via that interface.

[0029] (Storage part) The memory unit 20A stores various types of information referenced by the control unit 10A. The memory unit 20A specifically includes an observation data storage unit 201A. The observation data storage unit 201A stores observation data, including the trajectories of previously observed objects and the environmental information at the time those trajectories were observed. In other words, the observation data storage unit 201A stores previously observed object trajectories and environmental information in association. Furthermore, the observation data storage unit 201A stores the ground truth data generated by the ground truth data generation unit 115A (described later) in association with previously observed object trajectories and environmental information. The ground truth data indicates which trajectories are important and which are not. The data stored in the observation data storage unit 201A is used to train a learning model used by the trajectory permutation calculation unit 106A (described later) to calculate importance. In other words, the observation data storage unit 201A can also be said to store training data that includes multiple sets of past tracking results, environmental information corresponding to those tracking results, and correct answer data generated by the correct answer data generation unit 115A, which will be described later.

[0030] (Control Unit) The control unit 10A includes a sensor information acquisition unit 101A, a spatial information acquisition unit 102A, an environmental information acquisition unit 103A, an object detection unit 104A, an object tracking unit 105A, a trajectory sequence calculation unit 106A, an image enhancement unit 107A, an image output unit 108A, a text generation unit 109A, a text output unit 110A, an audio acquisition unit 111A, a text conversion unit 112A, and a training unit 113A. The object detection unit 104A, the object tracking unit 105A, the trajectory sequence calculation unit 106A, the image enhancement unit 107A, and the text generation unit 109A are examples of object detection means, object tracking means, importance calculation means, selection means, and text generation means according to this disclosure, respectively. The image output unit 108A and the text output unit 110A are examples of output means according to this disclosure. The training unit 113A is an example of labeling means and training means according to this disclosure.

[0031] (Sensor information acquisition unit) Figure 6 is a block diagram showing an example of the functional configuration of the information processing device 1A. The sensor information acquisition unit 101A acquires sensor information indicating the sensing results from a sensor that senses space. Examples of sensors that sense space include radar, LIDAR (Laser Imaging Detection and Ranging), event cameras, infrared cameras, surveillance cameras, and in-vehicle cameras. Examples of sensor information include information indicating measurement results from radar or LIDAR, and image data (multispectral images, SAR (Synthetic Aperture Radar) images, infrared images, surveillance images, in-vehicle images, etc.).

[0032] As an example, the sensor information acquisition unit 101A acquires sensor information input to the input unit 40A. The sensor information acquisition unit 101A may also receive sensor information from other devices via the communication unit 30A. The sensor information acquisition unit 101A may also acquire sensor information by reading it from a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A). The sensor information acquisition unit 101A may also perform preprocessing such as noise reduction on the sensor information.

[0033] (Spatial information acquisition unit) The spatial information acquisition unit 102A acquires spatial information representing spatial information. Spatial information may include, for example, data representing a map, satellite image, or aerial image of the target area. Spatial information may also include information representing the geography of the space (for example, information indicating latitude and longitude). The spatial information acquisition unit 102A acquires spatial information input to the input unit 40A, for example. The spatial information acquisition unit 102A may also receive spatial information from other devices via the communication unit 30A. The spatial information acquisition unit 102A may also acquire spatial information by reading spatial information from a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A).

[0034] (Environmental Information Acquisition Department) The environmental information acquisition unit 103A acquires environmental information about the target environment. Examples of environmental information include temperature, climate, current events (external news such as an aircraft taking off from xx airport), observation information from other locations (such as sensor information from another location), and the date and time when the sensor information was acquired. Examples of observation information from other locations include satellite data from other locations and information indicating the weather in the surrounding environment. As an example, the environmental information is used by the text generation unit 109A, which will be described later, to generate text.

[0035] As an example, the environmental information acquisition unit 103A acquires environmental information input to the input unit 40A. The environmental information acquisition unit 103A may also receive environmental information from other devices via the communication unit 30A. Alternatively, the environmental information acquisition unit 103A may acquire environmental information by reading it from a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A).

[0036] (Target detection unit) The object detection unit 104A uses the sensor information acquired by the sensor information acquisition unit 101A to detect objects in space and generates data indicating the detection result. The data generated by the object detection unit 104A is, for example, coordinate data that shows the area of ​​the object as a rectangle. For example, if the sensor information is information indicating measurement results from radar or LiDAR, the object detection unit 104A detects the object based on the measurement results from radar or LiDAR. Furthermore, if the sensor information is image data (multispectral image, infrared image, etc.), the object detection unit 104A detects the object using, for example, a method employing an object detection model such as YOLOX. The method employing an object detection model is not limited to YOLOX; the object detection unit 104A may also detect objects using other methods such as YOLO (You Only Look Once), ViT (Vision Transformer), Faster R-CNN (Regions with CNN features), or SSD (Single Shot MultiBox Detector).

[0037] (Target tracking unit) The object tracking unit 105A tracks the object detected by the object detection unit 104A by correlating the objects in a time series and generates data indicating the tracking results. The data indicating the tracking results is, for example, the time-series coordinates of each trajectory obtained by the tracking of the object tracking unit 105A. For example, if the sensor information is information indicating measurement results from radar or LiDAR, the object tracking unit 105A tracks the object using a method such as a Kalman filter. If the sensor information is image data, the object tracking unit 105A tracks the object using, for example, the ByteTrack method.

[0038] (Locus permutation calculation unit) The trajectory permutation calculation unit 106A calculates the importance of each of the multiple trajectories obtained by the tracking of the target tracking unit 105A, and prioritizes the trajectories in order of importance. The importance calculated by the trajectory permutation calculation unit is, for example, a vector in which the number of trajectories is the number of dimensions and the importance of each trajectory is each component.

[0039] The trajectory permutation calculation unit 106A calculates importance using, for example, a learning model generated by machine learning. Examples of learning models include deep neural networks. More specifically, deep neural networks that take time-series data as input include LSTM (Long Short-Term Memory) and 1DCNN (One-Dimensional Convolutional Neural Network). The learning model is, for example, trained in the trajectory permutation learning unit 114A, which will be described later.

[0040] The input data to be input to the learning model includes data showing the tracking results obtained by the target tracking unit 105A. In addition to the data showing the tracking results, the input data may also include at least one of environmental information and spatial information. In other words, the learning model can be said to be a learning model that takes trajectories, environmental information, and spatial information as inputs and outputs the importance of the target trajectories.

[0041] (Image enhancement section) The image enhancement unit 107A selects from among multiple trajectories obtained by the tracking of the target tracking unit 105A that have a higher importance than other targets. For example, the image enhancement unit 107A selects trajectories with a priority higher than a predetermined threshold. The image enhancement unit 107A also generates a superimposed image by superimposing an image representing the trajectory onto an image representing the spatial information acquired by the spatial information acquisition unit 102A.

[0042] As an example, the image enhancement unit 107A generates a superimposed image by superimposing an image representing the selected trajectory onto an image representing spatial information. In this case, the superimposed image can also be described as an image on which important trajectories on a map or satellite image are superimposed.

[0043] As another example, the image enhancement unit 107A may generate a superimposed image in which the selected trajectory is emphasized more than the other trajectories. As a method for emphasizing the selected trajectory, for example, the image enhancement unit 107A may make the color of the high-priority trajectory different from the color of the other trajectories, or make the thickness of the high-priority trajectory thicker than the other trajectories. Alternatively, the image enhancement unit 107A may emphasize the selected trajectory by making the type of line used to draw the high-priority trajectory different from the type of line used for the other trajectories.

[0044] Furthermore, the image enhancement unit 107A may superimpose predicted future trajectories onto the image representing space, in addition to the trajectories obtained by the tracking of the target tracking unit 105A. In this case, the image enhancement unit 107A may, for example, select one or more trajectories similar to those obtained by the tracking of the target tracking unit 105A from previously accumulated observation data (a collection of data indicating which trajectories are important and which are not), and superimpose the selected trajectories onto the image representing space as predicted trajectories.

[0045] (Image output section) The image output unit 108A outputs the image data generated by the image enhancement unit 107A. For example, the image output unit 108A outputs the image data to a display connected to the output unit 50A, and displays the image represented by the image data on the display. Alternatively, the image output unit 108A may transmit the image data to another device connected via the communication unit 30A, and display the image represented by the image data on the display of that other device.

[0046] The image output unit 108A may also output the image data by writing it to a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A). The image output unit 108A may also output the data to an output device such as a speaker or a printer.

[0047] (Text generation unit) The text generation unit 109A generates text describing the trajectory of an object using environmental information and spatial information. Alternatively, the text generation unit 109A may generate text using the trajectory obtained by the object tracking unit 105A, in addition to the environmental and spatial information. In this case, the text generation unit 109A may generate text describing the trajectory using the trajectory selected by the image enhancement unit 107A from among multiple trajectories, along with environmental information and spatial information.

[0048] The text generation unit 109A generates text using, for example, a large-scale language model. Examples of large-scale language models include, but are not limited to, generative AIs such as ChatGPT (Chat Generative Pre-trained Transformer), GPT-4 (Generative Pre-trained Transformer 4), and GPT-4o, or generative AIs that have been fine-tuned using environmental information and spatial information.

[0049] The large-scale language model may be stored in the storage unit 20A of the information processing device 1A, or it may be stored in a device other than the information processing device 1A. Here, when we say that the large-scale language model is stored in a memory device (such as the storage unit 20A), we mean that the parameters that define the large-scale language model are stored in the memory device. When the large-scale language model is stored in a device other than the information processing device 1A, the text generation unit 109A transmits input data to the device via the communication unit 30A, receives output data transmitted from the device, and generates the above text based on the received output data.

[0050] (Input to large-scale language models) The input data fed into the large-scale language model includes environmental information and spatial information. The input data may also include tracking results from the target tracking unit 105A. In other words, the text generation unit 109A can generate text describing the trajectory of the target based on output data obtained by inputting the input data, which includes the tracking results, the environmental information, and the spatial information, into the large-scale language model.

[0051] Furthermore, the input data may include text converted by the text conversion unit 112A, which will be described later. In this case, the text generation unit 109A generates text explaining the trajectory using the tracking results, the environmental information, the spatial information, and the text converted by the text conversion unit 112A. In other words, the text generation unit 109A can also generate text explaining the trajectory of the target using the tracking results, the environmental information, the spatial information, and text representing the user's spoken voice.

[0052] Furthermore, the input data may include superimposed images generated by the image enhancement unit 107A. In this case, the text generation unit 109A can generate text explaining the trajectory using the tracking results, the environmental information, the spatial information, and the image output by the image output unit 108A.

[0053] Furthermore, the input data may include instructional text. For example, the instructional text may be: "Below are images with the tracked object superimposed, the date, the tracked object's trajectory, the importance of that trajectory, and environmental information (such as temperature). Please summarize these using past response texts as a reference." In addition to the above data, the input data may also include the importance calculated by the trajectory permutation calculation unit 106A.

[0054] Furthermore, the input data may include environmental information corresponding to past trajectories similar to the tracking results obtained by the target tracking unit 105A. In this case, the text generation unit 109A searches the observation data storage unit 201A for one or more trajectories similar to the trajectory of the target tracked by the target tracking unit 105A, and includes the environmental information stored in association with the searched trajectories as input data for the large-scale language model.

[0055] Furthermore, the input data may include response text obtained in the past for similar trajectories. In this case, the text generation unit 109A inputs the previously observed target trajectories and environmental information into a large-scale language model and stores the response text obtained in the observation data storage unit 201A in association with the previously observed target trajectories and the environmental information corresponding to those trajectories. The past response text stored in association with the retrieved trajectories is then included in the input data of the large-scale language model.

[0056] (Output of a large-scale language model) The output of the large-scale language model includes response text. An example of response text would be: "At xx day yy hour zz minute, bb (target object) passed near point aa with a speed of cc (velocity, etc.). It may pass through dd in the future. In a similar past case, at ee year ff month gg day hh hour jj minute, it passed through point kk and then point mm. At that time, the decision was made to take action nn (e.g., rescue)." The output of the large-scale language model may also include data other than text (image data, audio data, etc.).

[0057] (Pre-training of large-scale language models) The text generation unit 109A may pre-train the large-scale language model through fine-tuning, instruction tuning, etc., so that the large-scale language model outputs more desirable text. In this case, the training data used for fine-tuning, instruction tuning, etc. may include, as an example, at least one of the following for past cases: text converted by the text conversion unit 112A, spatial information acquired by the spatial information acquisition unit 102A, environmental information acquired by the environmental information acquisition unit 103A, superimposed images generated by the image enhancement unit 107A, trajectories obtained by the target tracking unit 105A, and importance calculated by the trajectory permutation calculation unit 106A.

[0058] Furthermore, the training data includes response texts corresponding to past trajectories (response texts output by large-scale language models in the past). The response texts corresponding to past trajectories are response texts corresponding to similar trajectories (reports, work instructions, etc.). The training data may also include similar environmental information corresponding to trajectories predicted from similar past data. Here, similar environmental information refers to environmental information from previously observed data that corresponds to data with similar trajectories (similar number of trajectories, similar movement, passing through the same points, etc.).

[0059] (Text output section) The text output unit 110A outputs the text generated by the text generation unit 109A. For example, the text output unit 110A outputs the above text to a display connected to the output unit 50A, and displays the text on the display. Alternatively, the text output unit 110A may transmit the above text to another device connected via the communication unit 30A, and display the text on the display of that other device.

[0060] Furthermore, the text output unit 110A may output the text by writing it to a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A). Alternatively, the text output unit 110A may output the text to an output device such as a speaker or a printer.

[0061] Figure 7 shows a specific example of an image output by the image output unit 108A and text output by the text output unit 110A. As shown in Figure 7, the image output unit 108A and the text output unit 110A output an image A11 representing the trajectory of an object in space and text A12 explaining the trajectory of the object. In Figure 7, image A11 is an image in which trajectories A111 to A113 selected by the image enhancement unit 107A are enhanced compared to other trajectories. Figure 9 shows an example of an image representing a trajectory according to the prior art. In the example in Figure 9, image A113 containing multiple trajectories is displayed. Image A113 contains a mixture of unimportant and important trajectories, making it difficult for the user to determine which trajectories are important or whether any trajectories require any action, even after viewing image A113. As is clear from comparing image A113 in Figure 9 with image A11 in Figure 7, the image output by the information processing device 1A according to this disclosure makes it easier for the user to grasp important trajectories.

[0062] (Speech acquisition unit / Text conversion unit) The voice acquisition unit 111A acquires voice data representing the user's speech. For example, the voice data may be data recorded by a microphone recorder or similar device when the user is identifying important trajectories. For example, the voice data may be used to reduce errors in visual inspection. Furthermore, the voice data may also be used for recording the visual inspection process. The text conversion unit 112A converts the voice data into text. For example, the text conversion unit 112A performs the conversion using a deep learning-based voice conversion method.

[0063] (Training Unit / Correct Answer Data Generation Unit) The training unit 113A trains the regression function of the trajectory permutation calculation unit 106A. The training unit 113A comprises a trajectory permutation learning unit 114A and a ground truth data generation unit 115A. The ground truth data generation unit 115A assigns labels (ground truth data) indicating importance to the trajectories of the target obtained by tracking by the target tracking unit 105A.

[0064] As an example, the correct answer data generation unit 115A extracts words corresponding to important trajectories from the text representing the user's spoken voice, selects a trajectory corresponding to the extracted words from among multiple trajectories obtained by tracking by the target tracking unit 105A, and assigns a label indicating importance to the selected trajectory. In this case, for example, the administrator of the information processing device 1A uses an input device connected to the input unit 40A to select important words from the text representing the spoken voice. The correct answer data generation unit 115A extracts the selected words as important words and assigns a label indicating high importance to the trajectory corresponding to the extracted words. Here, examples of trajectories corresponding to the extracted words include, but are not limited to, the trajectory of an object that passed through the area indicated by the extracted words, or the trajectory of an object that passed through a predetermined area at the date and time indicated by the extracted words.

[0065] As another example, the correct answer data generation unit 115A may present environmental information and text to an administrator or other person, who may then review the presented environmental information and text and extract keywords corresponding to important trajectories. In this case, the correct answer data generation unit 115A may generate the correct answer data by assigning importance levels to the trajectories corresponding to the keywords extracted by the administrator or other person.

[0066] Alternatively, the ground truth data generation unit 115A may generate ground truth data by extracting words corresponding to important trajectories from environmental information and text information using a learning model such as a large-scale language model, and assigning importance levels to the trajectories corresponding to those words.

[0067] (Learning Unit for Trajectory Permutations) The trajectory permutation learning unit 114A trains a learning model using training data that shows trajectories with the above labels. The training data may also include environmental information and spatial information in addition to the data showing labeled trajectories. In other words, the trajectory permutation learning unit 114A can also be said to train a learning model using training data that includes the above-mentioned labeled trajectories, environmental information, and spatial information. The weights that the trajectory permutation learning unit 114A optimizes are, for example, a regression function using a deep neural network. More specifically, the weights of a deep neural network that takes time-series data as input are LSTM or 1DCNN (one-dimensional convolutional neural network). These weights are calculated, for example, based on the training data. More specifically, for example, a loss function is defined based on the difference between the importance (estimated importance) regressed with tracking results, environmental information, and spatial information as input variables and the correct importance, and the trajectory permutation learning unit 114A learns the weights to minimize this function.

[0068] (Examples of practical applications) The information processing device 1A relating to this disclosure is applicable to various technical fields such as robotics, logistics systems, and drone control. For example, in the case of robotics, the subject of this disclosure is, as an example, a robot that moves objects, and the text generated by the text generation unit 109A is, as an example, a report explaining the work performed by the robot or instructions related to the work. Also, for example, in the case of logistics systems, the subject of this disclosure is, as an example, a delivery person that delivers products, and the text generated by the text generation unit 109A is, as an example, a work report or instructions related to delivery work using a delivery vehicle.

[0069] (Effects of information processing equipment) As described above, the information processing device 1A includes a trajectory permutation calculation unit 106A that calculates the importance of each of the multiple trajectories obtained by tracking by the target tracking unit 105A, and an image enhancement unit 107A that selects from the multiple trajectories that have a higher importance than other targets. The text generation unit 109A generates text describing the trajectory using the trajectory selected by the image enhancement unit 107A, the environmental information, and the spatial information. The image output unit 108A and the text output unit 110A output an image in which the trajectory selected by the image enhancement unit 107A is enhanced compared to other trajectories, and the text generated by the text generation unit 109A. Therefore, with the information processing device 1A, even when the monitoring area includes the trajectories of multiple targets, it is easy for the user to understand what characteristics the trajectory that should be focused on has. As a result, the information processing device 1A can support the user's decision-making regarding the trajectory of a target.

[0070] Furthermore, in the information processing device 1A, the text generation unit 109A is configured to generate text describing the trajectory of the target using the tracking results, the environmental information, the spatial information, and text representing the user's spoken voice. Therefore, the information processing device 1A has the effect of being able to generate text describing the trajectory of the target with greater accuracy.

[0071] Furthermore, in the information processing device 1A, the text generation unit 109A is configured to generate text describing the target's trajectory based on output data obtained by inputting the tracking results, environmental information, and spatial information into a large-scale language model. Therefore, the information processing device 1A can output text that makes it easier for the user to make judgments about the target's trajectory.

[0072] Furthermore, in the information processing device 1A, the text generation unit 109A searches for trajectories similar to those tracked by the target tracking unit 105A from the observation data storage unit 201A, which stores previously observed target trajectories and environmental information in association with each other, and includes the environmental information corresponding to the searched trajectories in the input data. Therefore, the information processing device 1A can generate text that takes previously observed trajectories into consideration, thereby enabling the generation of more accurate text.

[0073] Furthermore, in the information processing device 1A, the text generation unit 109A inputs previously observed target trajectories and environmental information into a large-scale language model to obtain response text, which is then stored in the observation data storage unit 201A in association with the previously observed target trajectories and the environmental information corresponding to those trajectories. The past response text stored in association with the retrieved trajectories is then included in the input data. As a result, the information processing device 1A can generate text that takes into account responses to previously observed trajectories (for example, reports and work instructions), thereby enabling the generation of text desired by the user (for example, text that conforms to the format of previously generated reports and work instructions).

[0074] Furthermore, the information processing device 1A includes a ground truth data generation unit 115A that assigns labels indicating importance to the trajectories of targets obtained by tracking by the target tracking unit 105A, and a trajectory permutation learning unit 114A that trains a learning model that takes trajectories and environmental information as input and outputs the importance of the target trajectories using training data that includes labeled trajectories and environmental information. The trajectory permutation calculation unit 106A calculates the importance using the learning model. As a result, the information processing device 1A can calculate the importance of trajectories with greater accuracy.

[0075] [Examples of implementation using software] Some or all of the functions of the information processing devices 1, 1A, and 2 (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software. In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that implements each of the above functions. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 8. Figure 8 is a block diagram showing the hardware configuration of computer C, which functions as each of the above devices.

[0076] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0077] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0078] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0079] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0080] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0081] [Additional Note A] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0082] (Note A1) An object detection means for detecting an object present in a space using sensor information that shows the sensing results from a sensor that senses the space, The object tracking means tracks the object detected by the object detection means, A text generation means generates text describing the trajectory of the object using the tracking results from the object tracking means, environmental information relating to the environment of the object, and spatial information representing the space. An output means that outputs an image representing the trajectory of the object in the space and text generated by the text generation means, An information processing device equipped with the following features.

[0083] (Appendix A2) A means for calculating the importance of each of the multiple trajectories obtained by tracking the aforementioned target tracking means, The system further comprises a selection means for selecting from among the multiple trajectories that have a higher importance than the other objects, The text generation means generates text describing the trajectory using the trajectory selected by the selection means, the environmental information, and the spatial information. The output means outputs an image in which the trajectory selected by the selection means is emphasized compared to other trajectories, and the text generated by the text generation means. The information processing device described in Appendix A1.

[0084] (Note A3) The text generation means generates text describing the trajectory of the target using the tracking results, the environmental information, the spatial information, and text representing the user's spoken voice. The information processing device described in Appendix A1 or A2.

[0085] (Note A4) The text generation means generates text describing the trajectory of the target based on output data obtained by inputting the tracking results, the environmental information, and the spatial information into a large-scale language model. An information processing device as described in any one of the appendices A1 to A3.

[0086] (Note A5) The text generation means searches for a trajectory similar to the trajectory of the object tracked by the object tracking means from a storage device that stores previously observed object trajectories and environmental information in association, and includes the environmental information corresponding to the searched trajectory in the input data. The information processing device described in Appendix A4.

[0087] (Note A6) The text generation means inputs previously observed object trajectories and environmental information into a large-scale language model to obtain response text, stores the response text obtained by associating it with the previously observed object trajectories and the environmental information corresponding to those trajectories in the storage device, and includes the previously stored response text associated with the retrieved trajectories in the input data. The information processing device described in Appendix A5.

[0088] (Note A7) A labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking the object using the object tracking means, The system further comprises a training means for training a learning model that takes trajectories and environmental information as input and outputs the importance of the target trajectories, using training data that includes the trajectories to which the labels have been assigned and the environmental information. The importance calculation means calculates the importance using the learning model. The information processing device described in Appendix A2.

[0089] (Note A8) The labeling means extracts words corresponding to important trajectories from text representing the user's spoken audio, selects trajectories corresponding to the extracted words from among the multiple trajectories, and assigns a label indicating the importance to the selected trajectories. The information processing device described in Appendix A7.

[0090] (Note A9) The text generation means generates text describing the trajectory using the tracking results, the environmental information, the spatial information, and the image output by the output means. The information processing device described in Appendix A2, 7, or 8.

[0091] (Note A10) A voice acquisition means for acquiring voice data representing the user's spoken voice, The system further comprises a text conversion means for converting the aforementioned audio data into text, The text generation means generates text describing the trajectory using the tracking results, the environmental information, the spatial information, and the text converted by the text conversion means. The information processing device described in Appendix A3.

[0092] (Note A11) A means for detecting objects present in space using sensor information, The object tracking means tracks the object detected by the object detection means, A labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking the object using the object tracking means, A training means for training a learning model that takes trajectories and environmental information as inputs and outputs the importance of the trajectories, using training data that includes trajectories labeled by the labeling means and environmental information relating to the target environment. An information processing device equipped with the following features.

[0093] [Additional Note B] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0094] (Note B1) At least one processor performs an object detection process that detects objects present in a space using sensor information indicating the sensing results from a sensor that senses the space, The at least one processor performs an object tracking process that tracks the object detected in the object detection process, The at least one processor performs a text generation process that generates text describing the trajectory of the object using the tracking results from the object tracking process, environmental information relating to the environment of the object, and spatial information representing the space. The at least one processor outputs an image representing the trajectory of the object in the space and the text generated in the text generation process, Information processing methods including

[0095] (Note B2) The at least one processor performs an importance calculation process for each of the multiple trajectories obtained by tracking the target tracking process, and calculates the importance of each trajectory. The at least one processor further includes a selection process that selects from the plurality of trajectories that have a higher importance than the others, In the text generation process, the at least one processor generates text describing the trajectory using the trajectory selected in the selection process, the environmental information, and the spatial information. In the output process, the at least one processor outputs an image in which the trajectory selected in the selection process is emphasized more than other trajectories, and the text generated in the text generation process. The information processing method described in Appendix B1.

[0096] (Note B3) In the text generation process, the at least one processor generates text describing the trajectory of the target using the tracking result, the environmental information, the spatial information, and text representing the user's spoken voice. The information processing method described in Appendix B1 or B2.

[0097] (Note B4) In the text generation process, the at least one processor generates text describing the trajectory of the target based on output data obtained by inputting the tracking results, the environmental information, and the spatial information into a large-scale language model. The information processing method described in any one of the appendices B1 to B3.

[0098] (Note B5) In the text generation process, the at least one processor searches a storage device that stores previously observed object trajectories and environmental information in association with each other for trajectories similar to the trajectory of the object tracked in the object tracking process, and includes the environmental information corresponding to the searched trajectory in the input data. The information processing method described in Appendix B4.

[0099] (Note B6) In the text generation process, the at least one processor inputs the previously observed trajectory of an object and environmental information into a large-scale language model to obtain a response text, stores the response text obtained by associating the previously observed trajectory of the object with the environmental information corresponding to the trajectory in the storage device, and includes the previously stored response text associated with the retrieved trajectory in the input data. The information processing method described in Appendix B5.

[0100] (Note B7) The at least one processor performs a labeling process that assigns a label indicating importance to the trajectory of the object obtained by tracking the object tracking process, The at least one processor further includes a training process that uses training data including the labeled trajectories and the environmental information to train a learning model that takes trajectories and environmental information as inputs and outputs the importance of the target trajectories, In the importance calculation process, the at least one processor calculates the importance using the learning model. The information processing method described in Appendix B2.

[0101] (Note B8) In the labeling process described above, at least one processor extracts words corresponding to important trajectories from text representing the user's spoken voice, selects a trajectory corresponding to the extracted words from among the multiple trajectories, and assigns a label indicating the importance to the selected trajectory. The information processing method described in Appendix B7.

[0102] (Note B9) In the text generation process, the at least one processor generates text describing the trajectory using the tracking result, the environmental information, the spatial information, and the image output in the output process. The information processing method described in Appendix B2, 7, or 8.

[0103] (Note B10) The aforementioned at least one processor performs a speech acquisition process to acquire speech data representing the user's speech, The at least one processor further includes a text conversion process that converts the audio data into text, In the text generation process, the at least one processor generates text describing the trajectory using the tracking result, the environmental information, the spatial information, and the text converted in the text conversion process. The information processing method described in Appendix B3.

[0104] (Note B11) The aforementioned at least one processor performs an object detection process that detects an object present in space using sensor information, The at least one processor performs an object tracking process that tracks the object detected in the object detection process, The at least one processor performs a labeling process that assigns a label indicating importance to the trajectory of the object obtained by tracking the object tracking process, The at least one processor performs a training process that uses training data including the trajectories labeled by the labeling process and environmental information relating to the target environment to train a learning model that takes the trajectories and environmental information as inputs and outputs the importance of the trajectories. Information processing methods including

[0105] [Additional Note C] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0106] (Note C1) A program for causing a computer to function as an information processing device, wherein the computer, An object detection means for detecting an object present in a space using sensor information that shows the sensing results from a sensor that senses the space, The object tracking means tracks the object detected by the object detection means, A text generation means generates text describing the trajectory of the object using the tracking results from the object tracking means, environmental information relating to the environment of the object, and spatial information representing the space. An output means that outputs an image representing the trajectory of the object in the space and text generated by the text generation means, An information processing program designed to function as such.

[0107] (Note C2) The aforementioned computer, A means for calculating the importance of each of the multiple trajectories obtained by tracking the aforementioned target tracking means, Furthermore, it functions as a selection means for selecting from the aforementioned multiple trajectories that have a higher importance than the others. The text generation means generates text describing the trajectory using the trajectory selected by the selection means, the environmental information, and the spatial information. The output means outputs an image in which the trajectory selected by the selection means is emphasized compared to other trajectories, and the text generated by the text generation means. The information processing program described in Appendix C1.

[0108] (Note C3) The text generation means generates text describing the trajectory of the target using the tracking results, the environmental information, the spatial information, and text representing the user's spoken voice. The information processing program described in Appendix C1 or C2.

[0109] (Note C4) The text generation means generates text describing the trajectory of the target based on output data obtained by inputting the tracking results, the environmental information, and the spatial information into a large-scale language model. An information processing program described in any one of the appendices C1 to C3.

[0110] (Note C5) The text generation means searches for a trajectory similar to the trajectory of the object tracked by the object tracking means from a storage device that stores previously observed object trajectories and environmental information in association, and includes the environmental information corresponding to the searched trajectory in the input data. The information processing program described in Appendix C4.

[0111] (Appendix C6) The text generation means inputs previously observed object trajectories and environmental information into a large-scale language model to obtain response text, stores the response text obtained by associating it with the previously observed object trajectories and the environmental information corresponding to those trajectories in the storage device, and includes the previously stored response text associated with the retrieved trajectories in the input data. The information processing program described in Appendix C5.

[0112] (Note C7) The aforementioned computer, A labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking the object using the object tracking means, The training means further functions as a means for training a learning model that takes trajectories and environmental information as input and outputs the importance of the target trajectory, using training data including the trajectories to which the labels have been assigned and the environmental information. The importance calculation means calculates the importance using the learning model. The information processing program described in Appendix C2.

[0113] (Note C8) The labeling means extracts words corresponding to important trajectories from text representing the user's spoken audio, selects trajectories corresponding to the extracted words from among the multiple trajectories, and assigns a label indicating the importance to the selected trajectories. The information processing program described in Appendix C7.

[0114] (Note C9) The text generation means generates text describing the trajectory using the tracking results, the environmental information, the spatial information, and the image output by the output means. The information processing program described in Appendix C2, 7, or 8.

[0115] (Note C10) The aforementioned computer, A voice acquisition means for acquiring voice data representing the user's spoken voice, The aforementioned audio data is further provided to function as a text conversion means for converting audio data into text, The text generation means generates text describing the trajectory using the tracking results, the environmental information, the spatial information, and the text converted by the text conversion means. The information processing program described in Appendix C3.

[0116] (Note C11) An information processing program for causing a computer to function as an information processing device, wherein the computer, A means for detecting objects present in space using sensor information, The object tracking means tracks the object detected by the object detection means, A labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking the object using the object tracking means, A training means for training a learning model that takes trajectories and environmental information as inputs and outputs the importance of the trajectories, using training data that includes trajectories labeled by the labeling means and environmental information relating to the target environment. An information processing program designed to function as such.

[0117] [Additional Note D] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0118] (Note D1) It comprises at least one processor, and the at least one processor is A process for detecting objects present in a space using sensor information that shows the sensing results from a sensor that senses the space, A target tracking process that tracks the target detected in the aforementioned target detection process, A text generation process that generates text describing the trajectory of the object using the tracking results from the aforementioned object tracking process, environmental information relating to the environment of the object, and spatial information representing the space, An output process that outputs an image representing the trajectory of the object in the aforementioned space and the text generated in the text generation process, An information processing device that performs the following actions.

[0119] The information processing device may also include memory. Furthermore, the memory may store a program that causes at least one processor to execute each of the aforementioned processes.

[0120] (Note D2) The aforementioned at least one processor, For each of the multiple trajectories obtained through the aforementioned target tracking process, an importance calculation process is performed to calculate the importance of each trajectory. Further, a selection process is performed to select the one with a higher importance than the others from among the aforementioned multiple trajectories. In the text generation process, the at least one processor generates text describing the trajectory using the trajectory selected in the selection process, the environmental information, and the spatial information. In the output process, the at least one processor outputs an image in which the trajectory selected in the selection process is emphasized more than other trajectories, and the text generated in the text generation process. The information processing device described in Appendix D1.

[0121] (Note D3) In the text generation process, the at least one processor generates text describing the trajectory of the target using the tracking result, the environmental information, the spatial information, and text representing the user's spoken voice. The information processing device described in Appendix D1 or D2.

[0122] (Note D4) In the text generation process, the at least one processor generates text describing the trajectory of the target based on output data obtained by inputting the tracking results, the environmental information, and the spatial information into a large-scale language model. An information processing device as described in any one of the appendices D1 to D3.

[0123] (Note D5) In the text generation process, the at least one processor searches a storage device that stores previously observed object trajectories and environmental information in association with each other for trajectories similar to the trajectory of the object tracked in the object tracking process, and includes the environmental information corresponding to the searched trajectory in the input data. The information processing device described in Appendix D4.

[0124] (Note D6) In the text generation process, the at least one processor inputs the previously observed trajectory of an object and environmental information into a large-scale language model to obtain a response text, stores the response text obtained by associating the previously observed trajectory of the object with the environmental information corresponding to the trajectory in the storage device, and includes the previously stored response text associated with the retrieved trajectory in the input data. The information processing device described in Appendix D5.

[0125] (Note D7) The aforementioned at least one processor, A labeling process that assigns labels indicating importance to the trajectory of the object obtained by tracking the object in the aforementioned object tracking process, Further, a training process is performed to train a learning model that takes trajectories and environmental information as input and outputs the importance of the target trajectories, using training data that includes the trajectories to which the labels have been assigned and the environmental information. In the importance calculation process, the at least one processor calculates the importance using the learning model. The information processing device described in Appendix D2.

[0126] (Note D8) In the labeling process described above, at least one processor extracts words corresponding to important trajectories from text representing the user's spoken voice, selects a trajectory corresponding to the extracted words from among the multiple trajectories, and assigns a label indicating the importance to the selected trajectory. The information processing device described in Appendix D7.

[0127] (Note D9) In the text generation process, the at least one processor generates text describing the trajectory using the tracking result, the environmental information, the spatial information, and the image output in the output process. The information processing device described in Appendix D2, 7, or 8.

[0128] (Note D10) The aforementioned at least one processor, A voice acquisition process that obtains audio data representing the user's spoken voice, The process further involves performing a text conversion process that converts the aforementioned audio data into text, In the text generation process, the at least one processor generates text describing the trajectory using the tracking result, the environmental information, the spatial information, and the text converted in the text conversion process. The information processing device described in Appendix D3.

[0129] (Note D11) It comprises at least one processor, and the at least one processor is Object detection processing that detects objects present in space using sensor information, A target tracking process that tracks the target detected in the aforementioned target detection process, A labeling process that assigns labels indicating importance to the trajectory of the object obtained by tracking the object in the aforementioned object tracking process, A training process is performed to train a learning model that takes trajectories and environmental information as inputs and outputs the importance of the trajectories, using training data that includes trajectories labeled by the labeling process and environmental information about the target environment. An information processing device that performs the following actions.

[0130] [Additional Note E] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0131] (Note E1) A program that causes a computer to function as an information processing device, wherein the computer, A process for detecting objects present in a space using sensor information that shows the sensing results from a sensor that senses the space, A target tracking process that tracks the target detected in the aforementioned target detection process, A text generation process that generates text describing the trajectory of the object using the tracking results from the aforementioned object tracking process, environmental information relating to the environment of the object, and spatial information representing the space, An output process that outputs an image representing the trajectory of the object in the aforementioned space and the text generated in the text generation process, A non-temporary recording medium that stores an information processing program that executes that program.

[0132] (Note E2) An information processing program for causing a computer to function as an information processing device, wherein the computer, Object detection processing that detects objects present in space using sensor information, A target tracking process that tracks the target detected in the aforementioned target detection process, A labeling process that assigns labels indicating importance to the trajectory of the object obtained by tracking through the aforementioned object tracking process, A training process is performed to train a learning model that takes trajectories and environmental information as inputs and outputs the importance of the trajectories, using training data that includes trajectories labeled by the labeling process and environmental information about the target environment. A non-temporary recording medium that stores an information processing program that executes that program. [Explanation of Symbols]

[0133] 1, 1A, 2 Information Processing Devices 11, 21, 104A Target detection unit 12, 22, 105A Target Tracking Section 13, 109A Text generation unit 14. 50A output section 23 Labeling section 24, 113A Training Department 115A Correct Answer Data Generation Unit 106A Trajectory Permutation Calculation Unit 107A Image enhancement section 108A Image Output Unit 110A Text Output Unit 111A Voice acquisition unit 112A Text Conversion Unit 114A Trajectory Permutation Learning Unit

Claims

1. An object detection means for detecting an object present in a space using sensor information that shows the sensing results from a sensor that senses the space, The object tracking means tracks the object detected by the object detection means, A text generation means generates text describing the trajectory of the object using the tracking results from the object tracking means, environmental information relating to the environment of the object, and spatial information representing the space. An output means that outputs an image representing the trajectory of the object in the space and text generated by the text generation means, An information processing device equipped with the following features.

2. A means for calculating the importance of each of the multiple trajectories obtained by tracking the aforementioned target tracking means, The system further comprises a selection means for selecting from among the multiple trajectories that have a higher importance than the other objects, The text generation means generates text describing the trajectory using the trajectory selected by the selection means, the environmental information, and the spatial information. The output means outputs an image in which the trajectory selected by the selection means is emphasized compared to other trajectories, and the text generated by the text generation means. The information processing apparatus according to claim 1.

3. The text generation means generates text describing the trajectory of the target using the tracking results, the environmental information, the spatial information, and text representing the user's spoken voice. The information processing apparatus according to claim 1 or 2.

4. The text generation means generates text describing the trajectory of the target based on output data obtained by inputting the tracking results, the environmental information, and the spatial information into a large-scale language model. The information processing apparatus according to claim 1 or 2.

5. The text generation means searches for a trajectory similar to the trajectory of the object tracked by the object tracking means from a storage device that stores previously observed object trajectories and environmental information in association, and includes the environmental information corresponding to the searched trajectory in the input data. The information processing apparatus according to claim 4.

6. The text generation means inputs the previously observed trajectory of an object and environmental information into a large-scale language model to obtain a response text, which is then stored in the storage device in association with the previously observed trajectory of the object and the environmental information corresponding to that trajectory. The past answer text stored in association with the searched trajectory is included in the input data. The information processing apparatus according to claim 5.

7. A labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking the object using the object tracking means, The system further comprises a training means for training a learning model that takes trajectories and environmental information as input and outputs the importance of the target trajectories, using training data that includes the trajectories to which the labels have been assigned and the environmental information. The importance calculation means calculates the importance using the learning model. The information processing apparatus according to claim 2.

8. A means for detecting objects present in space using sensor information, The object tracking means tracks the object detected by the object detection means, A labeling means for assigning a label indicating importance to the trajectory of the object obtained by tracking the object using the object tracking means, A training means for training a learning model that takes trajectories and environmental information as inputs and outputs the importance of the trajectories, using training data that includes trajectories labeled by the labeling means and environmental information relating to the target environment. An information processing device equipped with the following features.

9. At least one processor performs an object detection process that detects an object present in a space using sensor information indicating the sensing result from a sensor that senses the space, The at least one processor performs an object tracking process that tracks the object detected in the object detection process, The at least one processor performs a text generation process that generates text describing the trajectory of the object using the tracking results from the object tracking process, environmental information relating to the environment of the object, and spatial information representing the space. The at least one processor outputs an image representing the trajectory of the object in the space and the text generated in the text generation process, Information processing methods including

10. An information processing program for causing a computer to function as an information processing device, wherein the computer, An object detection means for detecting an object present in a space using sensor information that shows the sensing results from a sensor that senses the space, The object tracking means tracks the object detected by the object detection means, A text generation means generates text describing the trajectory of the object using the tracking results from the object tracking means, environmental information relating to the environment of the object, and spatial information representing the space. An output means that outputs an image representing the trajectory of the object in the space and text generated by the text generation means, An information processing program designed to function as such.

Images