Processing device, processing method, and recording medium

Abstract

A processing device according to the present disclosure comprises a situation data acquisition unit, a trajectory data acquisition unit, a reception unit, and a display control unit. The situation data acquisition unit acquires situation data indicating the situation in a real space. The trajectory data acquisition unit acquires trajectory data indicating the movement trajectory of an object in the real space. The reception unit receives an input designating a specific point on the movement trajectory of the object in the real space. The display control unit displays a virtual space in which the situation in the real world at the time at which the object is positioned at the specific point is reproduced.

Description

Processing device, processing method, and recording medium

[0001] The present disclosure relates to a processing device, a processing method, and a program.

[0002] A technology related to the present disclosure is disclosed in Patent Document 1. The technology disclosed in Patent Document 1 detects an object from images captured by multiple imaging means. The technology then predicts the future state of the object based on the detection results. The technology then renders the estimated position of the object in a virtual three-dimensional space.

[0003] Japanese Patent Application Laid-Open No. 2023-81582

[0004] There is a need for a technology that can reproduce in a virtual space a situation in real space at a timing desired by a user. An example of an objective of the present disclosure is to provide a technology that can reproduce in a virtual space a situation in real space at a timing desired by a user.

[0005] According to the present disclosure, there is provided a processing device having: a situation data acquisition means for acquiring situation data indicating a situation in real space; a trajectory data acquisition means for acquiring trajectory data indicating a movement trajectory of an object in real space; a receiving means for receiving an input specifying a specific point on the movement trajectory of the object in real space; and a display control means for displaying a virtual space that reproduces the situation in the real world when the object is located at the specific point.

[0006] Furthermore, according to the present disclosure, a processing method is provided in which one or more computers acquire situation data indicating a situation in real space, acquire trajectory data indicating a movement trajectory of an object in real space, accept input specifying a specific point on the movement trajectory of the object in real space, and display a virtual space that reproduces the situation in the real world when the object is located at the specific point.

[0007] Furthermore, according to the present disclosure, a program is provided that causes a computer to function as: situation data acquisition means for acquiring situation data indicating the situation in real space; trajectory data acquisition means for acquiring trajectory data indicating the movement trajectory of an object in real space; reception means for receiving input specifying a specific point on the movement trajectory of the object in real space; and display control means for displaying a virtual space that reproduces the situation in the real world when the object is located at the specific point.

[0008] According to one aspect of the present disclosure, a technology is realized that reproduces in a virtual space a situation in a real space at a timing desired by a user.

[0009] FIG. 1 is a diagram showing an example of a functional block diagram of a processing device according to the present disclosure. FIG. 2 is a flowchart showing an example of a processing flow of a processing device according to the present disclosure. FIG. 3 is a diagram showing an example of a hardware configuration of a processing device according to the present disclosure. FIG. 4 is a diagram showing an example of information displayed by a processing device according to the present disclosure. FIG. 5 is a diagram showing another example of information displayed by a processing device according to the present disclosure. FIG. 6 is a flowchart showing another example of a processing flow of a processing device according to the present disclosure. FIG. 7 is a diagram showing another example of information displayed by a processing device according to the present disclosure. FIG. 8 is a diagram showing an example of a functional block diagram of a processing device according to the present disclosure. FIG. 9 is a diagram showing another example of information displayed by a processing device according to the present disclosure.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0011] <<First Embodiment>> Fig. 1 is a functional block diagram showing an overview of a processing device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the processing device 10.

[0012] 1, the processing device 10 includes a situation data acquisition unit 11, a trajectory data acquisition unit 12, a reception unit 13, and a display control unit 14. These functional units execute the processing of the flowchart in FIG.

[0013] In S10, the situation data acquisition unit 11 acquires situation data indicating the situation in real space. In S11, the trajectory data acquisition unit 12 acquires trajectory data indicating the movement trajectory of the object in real space. Note that the processing order of S10 and S11 is not limited to the example of FIG. 2 . S10 and S11 may be performed in parallel. Furthermore, S10 may be performed after S11. In S12, the reception unit 13 receives input specifying a specific point on the movement trajectory of the object in real space. In S13, the display control unit 14 displays a virtual space that reproduces the situation in the real world when the object is located at the specific point.

[0014] According to such a processing device 10, a user can reproduce in a virtual space the situation in the real space at a timing desired by the user by specifying a specific point on the movement trajectory of an object in the real space. That is, the user can specify the desired timing, i.e., the timing to be reproduced (hereinafter, sometimes referred to as "reproduction timing"), by specifying a specific point on the movement trajectory of an object in the real space. The timing desired by the user, i.e., the reproduction timing, is the timing when the object is at the specific point.

[0015] For example, a user may want to observe the behavior of an object based on a virtual space that reproduces a real-world situation. In this case, the user may want to confirm the real-world situation when the object was in a specific position. In addition, the user may want to confirm the real-world situation when the object was in each of multiple positions. In such cases, the user needs to specify the time when the object was in the specific position as the reproduction time.

[0016] The reproduction timing can be specified by specifying the time, for example. However, if the time when the object was at the specified position is unknown, it is not easy to specify the timing (time) when the object was at the specified position as the reproduction timing.

[0017] In contrast, with the processing device 10, the user simply designates a predetermined position (specific point) on the movement trajectory of the object. This input designates the time when the object was at the predetermined position as the reproduction time. With such processing device 10, the user can easily designate the time when the object was at the predetermined position as the reproduction time, and the situation in real space at that reproduction time can be reproduced in the virtual space.

[0018] <<Second Embodiment>> <Overview> A processing apparatus 10 according to a second embodiment is a specific embodiment of the configuration of the processing apparatus 10 according to the first embodiment. The processing apparatus 10 will be described in detail below.

[0019] <Hardware Configuration> First, an example of the hardware configuration of the processing device 10 will be described. Each functional unit of the processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are pre-loaded when the device is shipped, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0020] FIG. 3 is a block diagram illustrating an example of the hardware configuration of a processing device 10. As shown in FIG. 3, the processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0021] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, and touch panel. Examples of output devices include a display, projection device, speaker, printer, and mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0022] <Functional Configuration> Next, a detailed description will be given of the functional configuration of the processing device 10. Fig. 1 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has a situation data acquisition unit 11, a trajectory data acquisition unit 12, a reception unit 13, and a display control unit 14.

[0023] The situation data acquisition unit 11 acquires situation data that indicates the situation in real space.

[0024] The situation data indicates the situation in real space at a certain timing. Furthermore, the situation data can indicate the situation in real space at each of a plurality of timings in a time series. Such situation data can indicate changes in the situation in real space over time. The situation data has date and time information indicating the date and time of each timing.

[0025] The situation data is generated by analyzing images captured in real space. The images may be images (still images or video) generated by a surveillance camera installed in real space. Multiple surveillance cameras may be installed. The capture areas of the multiple cameras may partially overlap, or may not overlap at all. Alternatively, the images may be images (still images or video) generated by a camera installed on a moving object moving in real space. Examples of moving objects include, but are not limited to, automobiles, large vehicles, motorcycles, trains, buses, ships, and aircraft (drones, etc.). Alternatively, the images may be images (still images or video) generated by a person capturing images with a camera in real space.

[0026] The camera may detect visible light and generate an image, or may detect other electromagnetic waves such as ultraviolet light, infrared light, X-rays, and radio waves (microwaves, etc.) and generate an image. The camera may also generate an image without two-dimensional depth information, or may generate a distance image with depth information. Examples of cameras that generate distance images include, but are not limited to, stereo cameras and sensors (such as LIDAR) that emit electromagnetic waves such as radar and detect the reflected waves to generate an image. By analyzing such images, situation data is generated.

[0027] The situation data indicates the position and type of an object existing in the real space. There are various ways to classify types, but they are defined so as to be identifiable by image analysis. For example, the objects include at least one of people and non-human objects. People may be further subdivided by, for example, gender, age, nationality, etc. Non-human objects may be further subdivided into dogs, cats, automobiles, bicycles, traffic lights, signs, etc. Dogs and cats may also be further subdivided by type, etc. Automobiles may also be further subdivided by model, manufacturer, year, body type, etc.

[0028] The situation data may also indicate other situations that can be identified by image analysis. For example, the situation data may further indicate at least one of the following: "person's posture," "person's physique," "person's facial expression (smiling, serious, angry, surprised, etc.)," ​​"person's facial orientation," "person's body orientation," "person's hairstyle," "person's hair color," "person's clothing," "person's possessions," "person's clothing," "shape of a non-human object," "size of a non-human object," "color of a non-human object," "orientation of a non-human object," "pattern of a non-human object," "posture of a non-human object," and "color of a traffic light."

[0029] Such situation data can be generated using any well-known image analysis technique, such as a learning model generated by machine learning. Any well-known learning model, such as a classifier or a pose estimation model, can be used.

[0030] Furthermore, the position in real space of an object detected in an image can be determined using any well-known technology. In one example, transformation information for converting a position in the image (coordinates in a coordinate system set in the image) into a position in real space (coordinates in a coordinate system set in real space) is prepared in advance. Then, the position of the object detected in the image in real space can be estimated based on the position in the image of the object and this transformation information. Note that the process of determining the position in real space of the object detected in the image may or may not utilize a distance image. By utilizing depth information (distance from the camera) indicated by the distance image, the position in real space of the object detected in the image can be determined with high accuracy. However, even without using a distance image, the position in real space of the object detected in the image can be determined with a certain degree of accuracy.

[0031] In one example, the situation data acquisition unit 11 acquires images from an external device. Then, the situation data acquisition unit 11 analyzes the acquired images to generate situation data. The external device is the above-mentioned camera or a device that stores images generated by the above-mentioned camera. The processing device 10 and the external device are connected to each other so that they can communicate with each other. The situation data acquisition unit 11 may acquire images by real-time processing or by batch processing. The acquired images may include metadata indicating the shooting location, shooting time, etc.

[0032] In another example, an external device analyzes an image and generates situation data. Then, the situation data acquisition unit 11 acquires the situation data from the external device. The external device is the above-mentioned camera or a device that acquires and analyzes images generated by the above-mentioned camera. The processing device 10 and the external device are connected to each other so that they can communicate with each other. The situation data acquisition unit 11 may acquire the situation data by real-time processing or by batch processing.

[0033] "Acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request to another device and receiving a response, and accessing and reading information from another device or storage medium. An example of passive acquisition is receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0034] In addition to acquiring situation data through image analysis, the situation data acquisition unit 11 may acquire situation data by other methods. For example, the situation data acquisition unit 11 may acquire situation data from an external server, acquire sensing results from a sensor other than a camera installed in real space, or receive user input. The situation data acquired in such examples includes, but is not limited to, voice, weather, temperature, humidity, wind speed, wind direction, etc. at each position and each timing in real space.

[0035] The trajectory data acquisition unit 12 acquires trajectory data that indicates the movement trajectory of an object in real space.

[0036] The target object is an object that exists in real space. The trajectory data acquisition unit 12 acquires trajectory data that indicates the movement trajectory of at least one object among the objects that exist in real space.

[0037] The trajectory data indicates the movement trajectory of an object in real space. The trajectory data indicates the real space position of the object at each of a plurality of timings in a time series. The trajectory data includes date and time information indicating the date and time of each timing.

[0038] The trajectory data may be generated by analyzing an image captured in real space. For example, a computer tracks an object detected in the image within the image. The computer then identifies the position in real space from the position in the image of the object being tracked, and generates the trajectory data by connecting the identified positions in a chronological order. The method of identifying the position in real space from the position in the image can be, for example, the method described in the generation of situation data.

[0039] In this example, the trajectory data acquisition unit 12 may generate trajectory data by analyzing images acquired from an external device. The external device is the above-mentioned camera or a device that stores images generated by the above-mentioned camera. The processing device 10 and the external device are connected to each other so that they can communicate with each other. The images may be acquired by real-time processing or batch processing. The acquired images may include metadata indicating the shooting location, shooting time, etc.

[0040] Alternatively, an external device may analyze the image and generate the trajectory data. Then, the trajectory data acquisition unit 12 may acquire the trajectory data from the external device. The external device is the above-mentioned camera or a device that acquires and analyzes images generated by the above-mentioned camera. The processing device 10 and the external device are connected to each other so that they can communicate with each other. The trajectory data acquisition unit 12 may acquire the trajectory data by real-time processing or by batch processing.

[0041] Alternatively, the trajectory data may be generated based on the situation data described above. As described above, the situation data can indicate the situation in real space at each of a plurality of timings in a time series. The situation in real space indicates, for example, the position in real space of an object existing in the real space. Based on such situation data, the computer can generate trajectory data for each object by connecting the positions in real space in a time series. Note that the identity of objects appearing in different images (frame images) can be determined using the external appearance characteristics and position information of the objects.

[0042] In this example, the trajectory data acquisition unit 12 may acquire situation data from the situation data acquisition unit 11 and generate trajectory data based on the situation data. Alternatively, an external device may generate trajectory data based on the situation data. Then, the trajectory data acquisition unit 12 may acquire trajectory data from the external device. The external device is the above-mentioned camera or a device that acquires and analyzes images generated by the above-mentioned camera. The processing device 10 and the external device are connected to each other so that they can communicate with each other. The trajectory data acquisition unit 12 may acquire trajectory data by real-time processing or by batch processing.

[0043] Alternatively, a location information acquisition sensor (e.g., a GPS (global positioning system) sensor) mounted on a mobile object or a location information acquisition sensor mounted on a mobile device carried by a person may acquire current location information at predetermined time intervals, and the acquired current location information may be linked in chronological order to generate trajectory data. In this example, the trajectory data acquisition unit 12 acquires trajectory data from an external device that stores the trajectory data generated in this manner. The external device may be a device mounted on a mobile object, a mobile device carried by a person, or a device that acquires and stores trajectory data from these. The processing device 10 and these devices are connected to each other so as to be able to communicate with each other. The trajectory data acquisition unit 12 may acquire trajectory data by real-time processing or batch processing.

[0044] Both the situation data and the trajectory data include date and time information indicating the date and time of each timing. Therefore, both the situation data and the trajectory data indicate the location of each object at each timing. Therefore, by matching the positions of the objects at the same time, it is possible to identify which object's trajectory data indicates, among the objects indicated by the situation data.

[0045] The display control unit 14 generates a virtual space that reproduces a real-world situation using the situation data generated by the situation data acquisition unit 11. The situation data acquisition unit 11 then displays the generated virtual space via an output device. The virtual space is a three-dimensional space. Examples of the output device include a projection device and a display. The display may be a wearable display such as a head-mounted display or VR (virtual reality) goggles, or may be other types of display.

[0046] Fig. 4 shows a schematic example of a virtual space displayed by the display control unit 14. As shown in Fig. 4, the situation in the real space is reproduced in the virtual space.

[0047] In the virtual space, objects that remain in the same position in the real space and objects whose positions change in the real space are arranged in the same positional relationship as in the real space.

[0048] In Fig. 4, signs, traffic lights, buildings, roads, pedestrian crossings, white lines, etc. are shown as objects that remain in the same position in real space. Note that objects that remain in the same position in real space are not limited to these. Also, in Fig. 4, a person P and a car M are shown as objects whose positions change in real space. Note that objects whose positions change in real space are not limited to these.

[0049] The design of an object displayed in a virtual space can be determined using a variety of methods.

[0050] In one example, a design to be displayed in the virtual space is determined in advance for each type of object. Then, when the display control unit 14 identifies the type and position of an object based on the situation data, it displays the object in a design corresponding to the identified type at a position in the virtual space corresponding to the identified position. In this case, objects of the same type are displayed in the virtual space with the same design.

[0051] Furthermore, by categorizing objects into finer types, the designs of the objects displayed in the virtual space can reproduce real-world situations in more detail. If people are subdivided by, for example, gender, age, nationality, etc., people will be displayed in the virtual space with designs corresponding to those types. If dogs and cats are subdivided by type, dogs and cats will be displayed in the virtual space with designs corresponding to those types. If cars are subdivided by model, manufacturer, year, body type, etc., cars will be displayed with designs corresponding to those types. A user viewing the virtual space can grasp the details of an object based on the design of the displayed object.

[0052] As another example, the display control unit 14 may determine the design of an object to be displayed in the virtual space based on the external characteristics of the object indicated in the situation data. Specifically, the display control unit 14 may determine a design that expresses the external characteristics of the object indicated in the situation data. Alternatively, the display control unit 14 may determine a design by modifying the design for each type based on the external characteristics of each object indicated in the situation data. Then, the display control unit 14 may display the object with the determined design at a position in the virtual space that corresponds to the position of the object indicated in the situation data.

[0053] As described above, the situation data can indicate, for example, at least one of the following: "person's posture," "person's physique," "person's facial expression (smiling, serious, angry, surprised, etc.)," ​​"person's facial direction," "person's body direction," "person's hairstyle," "person's hair color," "person's clothing," "person's possessions," "things the person is wearing," "shape of a non-human object," "size of a non-human object," "color of a non-human object," "orientation of a non-human object," "pattern of a non-human object," "posture of a non-human object," and "color of a traffic light."

[0054] The display control unit 14 can display an object at a predetermined position in the virtual space with a design that expresses the external characteristics of the object. In this way, it is possible to reproduce a situation in the real world in more detail. That is, it is possible to reproduce in the virtual space a person's posture, facial expression, facial orientation, hairstyle, hair color, clothing, possessions, and items worn by a person. It is also possible to reproduce in the virtual space the shape, size, color, orientation, pattern, and posture of non-human objects, the color of traffic lights, and the like.

[0055] The display control unit 14 may also reproduce the weather indicated by the situation data in the virtual space. For example, the display control unit 14 may depict a situation in which it is raining or snowing in the virtual space. The display control unit 14 may also adjust the brightness of the displayed virtual space depending on the weather, such as sunny, cloudy, rainy, or snowy. The display control unit 14 may also reproduce wind speed and wind direction in the virtual space. For example, the display control unit 14 may reproduce wind speed and wind direction by displaying an object (e.g., a flag) fluttering in the wind at an arbitrary position in the virtual space and fluttering the object in an appropriate direction and to an appropriate extent. Additionally, the display control unit 14 may display weather, temperature, humidity, wind speed, wind direction, etc. as text information in a separate window or the like at an arbitrary position in the virtual space.

[0056] Furthermore, the display control unit 14 may continue to display the situation in the real space at a certain timing in the virtual space. That is, the display control unit 14 may display a virtual space that reproduces the situation in the real space as a still image. Alternatively, the display control unit 14 may display a change in the situation in the real space over time in the virtual space. That is, the display control unit 14 may display a virtual space that reproduces the situation in the real world as a moving image. When displaying the virtual space as a moving image, the display control unit 14 may output sound collected in the real space in time synchronization with the display of the virtual space.

[0057] The display control unit 14 may also display, in the virtual space, the movement trajectory of the object in the real space. An example is shown in FIG. 5. １ is the movement trajectory of the car M, and the movement trajectory Q ２ is the movement trajectory of person P. In this way, when multiple objects exist in the displayed virtual space, the display control unit 14 may display the movement trajectory of each of the multiple objects. Alternatively, when multiple objects exist in the displayed virtual space, the display control unit 14 may display the movement trajectory of one of the multiple objects selected by the user (not shown). The display control unit 14 can accept a selection by the user via an input device. Examples of input devices include, but are not limited to, a mouse, a keyboard, a touch panel, physical buttons, a microphone, a gesture input device, a laser pointer, a remote controller, etc.

[0058] The display control unit 14 can display a virtual space that reproduces the situation in real space at a position and timing specified by the user. The specification of the position and timing is received by the receiving unit 13. The display control unit 14 changes the content of the virtual space to be displayed based on the content received by the receiving unit 13. In other words, the display control unit 14 changes the position and timing of the real space reproduced in the virtual space based on the content received by the receiving unit 13.

[0059] The receiving unit 13 receives an input specifying a specific point on the movement trajectory of the object in real space. The display control unit 14 then recreates and displays in virtual space the situation in real space at the time the object is at the specific point. For example, the display control unit 14 identifies the date and time when the object is located at the specific point based on the date and time information of the trajectory data. The display control unit 14 then identifies situation data indicating the situation at the identified date and time based on the date and time information of the situation data. The display control unit 14 then recreates the situation in real space in virtual space based on the identified situation data. The display control unit 14 can also recreate and display the situation around the specific point in virtual space.

[0060] The receiving unit 13 can receive an input specifying a specific point on the movement trajectory of an object in real space using either of the following methods 1 and 2.

[0061] Method 1 In Method 1, the receiving unit 13 receives an input to select one of the objects displayed in the virtual space, move the position of the selected object within the virtual space, and specify the new position as a specific point. For example, the receiving unit 13 may receive an input to select one object and move its position within the virtual space by drag-and-drop or an input equivalent thereto. Note that the input method is not limited to this, and other methods may also be used.

[0062] If the position after the movement is not on the movement trajectory of the selected object in real space, the receiving unit 13 may invalidate the input.

[0063] Alternatively, if the position after the movement is not on the movement trajectory of the selected object in real space, the reception unit 13 may convert the position after the movement to a point on the movement trajectory of the selected object in real space and receive the converted point as the specific point. There are various ways of conversion, but for example, the reception unit 13 may convert the position after the movement to "the point closest to the position after the movement" on the movement trajectory of the selected object in real space.

[0064] As shown in the example of FIG. 5, the movement trajectory Q of the object (car M and person P) １ and Q ２ is displayed in the virtual space, the user can easily grasp the positions to which the object can be moved, which is preferable.

[0065] Method 2 In Method 2, the display control unit 14 calculates the movement trajectory Q of the object (car M and person P) as shown in the example of FIG. １ and Q ２ is displayed in the virtual space. Then, the reception unit 13 receives the movement trajectory Q １ and Q ２ The receiving unit 13 receives an input specifying a point on the screen 10. The receiving unit 13 receives the specified point as a specific point.

[0066] As shown in the example of FIG. 5, in the virtual space, the movement trajectories Q of each of the multiple objects (cars M and people P) in the real space are １ and Q ２ In this case, the reception unit 13 may display the movement trajectories Q of each of the plurality of objects (cars M and people P). １ and Q ２ The receiving unit 13 can receive an input specifying a point on any one of the movement trajectories. The receiving unit 13 then receives the point specified in this manner as a specific point.

[0067] Here, an example of how the display of the virtual space changes in response to the reception of input by the reception unit 13 will be described.

[0068] First, assume that, with the virtual space as shown in Fig. 4 displayed, the reception unit 13 receives an input specifying a specific point on the movement trajectory of the automobile M (object) using the method described above. In response to the input, the display control unit 14 changes the display of the virtual space from Fig. 4 to Fig. 6.

[0069] 4 and 6, it can be seen that the position of the automobile M (object), for which a specific point is designated on the movement trajectory, has changed. It can also be seen that the position of the person P (object), which is another object for which a specific point is not designated on the movement trajectory, has also changed. That is, in response to the operation on the automobile M (object), not only the appearance of the automobile M (object) but also of other objects has changed. It can also be seen that the posture and orientation of the person P (object) and the color of the traffic light have changed. In this way, the situation in the real space shown in the virtual space of FIG. 4 differs from the situation in the real space shown in the virtual space of FIG. 6.

[0070] Note that, although the explanation here is based on the assumption that an input specifying a specific point on the movement trajectory of a car M (object) has been received, a similar change in display is also achieved when an input specifying a specific point on the movement trajectory of a person P (object) is received.

[0071] 5 and 7, the movement trajectory Q of the object (car M and person P) before and after the change in the display of the virtual space is １ and Q ２ can also be displayed in the virtual space.

[0072] Next, an example of the flow of processing by the processing device 10 will be described with reference to the flowchart of Fig. 8. Note that the processing device 10 executes processing to acquire situation data and trajectory data at any timing before executing the processing shown in Fig. 8.

[0073] First, the processing device 10 displays a virtual space that reproduces a situation in the real world in response to a user input (S20), resulting in the virtual space being displayed as shown in FIG.

[0074] The position and timing of initial reproduction may be a preset default position and timing. The default may be set / updated by the user, or may be automatically set / updated by the processing device 10. Alternatively, the position and timing of initial display may be specified by an address, place name, date and time, etc., along with a user input to display the virtual space.

[0075] The processing device 10 then waits for an input specifying a specific point on the movement trajectory of the object in real space (S21). When the processing device 10 receives the input (Yes in S21), it recreates and displays in virtual space the situation in the real world when the object is positioned at the specific point (S22). As a result, the content of the virtual space being displayed changes, for example, from FIG. 4 to FIG. 6. Alternatively, the content of the virtual space being displayed changes, for example, from FIG. 5 to FIG. 7.

[0076] Thereafter, the processing device 10 repeats the same process.

[0077] "Operational Effects" According to the processing apparatus 10 of the second embodiment, operational effects similar to those of the processing apparatus 10 of the first embodiment are realized.

[0078] Furthermore, the processing device 10 of the second embodiment allows a user to efficiently check the real-world situation displayed in images generated by multiple cameras in a unified manner in a virtual space. By viewing the virtual space, the user can efficiently check the real-world situation when a certain object is located at a specific point.

[0079] Furthermore, the processing device 10 of the second embodiment can accept an input for specifying a specific point using the characteristic techniques such as the above-described methods 1 and 2. As a result, the user can intuitively and easily input a desired position as a specific point.

[0080] Furthermore, the processing device 10 of the second embodiment can acquire situation data indicating various situations as described above and reproduce the situation indicated by the situation data in a virtual space, thereby enabling the situation in the real world to be reproduced in more detail in the virtual space.

[0081] <<Third Embodiment>> There are cases where part of the trajectory data acquired by the method described in the second embodiment is missing. For example, trajectory data can be generated based on images captured by a camera, but in this case, trajectory data is not generated while moving in an area outside the camera's capture area. As a result, trajectory data is missing while moving in an area outside the camera's capture area.

[0082] Furthermore, even when generating movement trajectory data using a location information acquisition sensor mounted on a moving body or a location information acquisition sensor mounted on a mobile terminal carried by a person, there may be sections where the movement trajectory is not displayed due to unforeseen circumstances such as poor communication.

[0083] The processing device 10 of the third embodiment has a function of interpolating missing portions based on the acquired trajectory data, which will be described in detail below.

[0084] Based on the acquired trajectory data, the trajectory data acquisition unit 12 estimates the movement trajectory of the object in a section not indicated by the acquired trajectory data. Specifically, based on the acquired trajectory data, the trajectory data acquisition unit 12 interpolates the movement trajectory of the object in a section not indicated by the acquired trajectory data. For example, the trajectory data acquisition unit 12 may predict (interpolate) the movement trajectory of the object in a section not indicated by the acquired trajectory data based on trajectory data before and after the section not indicated by the acquired trajectory data. The trajectory data acquisition unit 12 can achieve this interpolation using any data interpolation technique. An example of interpolation will be described below, but is not limited to this.

[0085] As described in the second embodiment, the trajectory data includes date and time information indicating the date and time of each timing. Based on this date and time information, the date and time when the object was at each position on the movement trajectory are identified. The trajectory data acquisition unit 12 can calculate the average movement speed of the object based on the date and time when the object was at each position on the movement trajectory. Then, the trajectory data acquisition unit 12 can generate trajectory data (data indicating the location of the object at each timing) on ​​the assumption that the object moves at that movement speed through a section not indicated in the acquired trajectory data.

[0086] In addition, if there are multiple candidate movement routes for a section not indicated in the acquired trajectory data, the trajectory data acquisition unit 12 can estimate one of the movement routes as the movement trajectory of the object using a predetermined rule.

[0087] The predetermined rule may be, for example, "select the shortest movement route." Alternatively, the predetermined rule may be "select the movement route whose movement distance is closest to a reference value." The reference value may be, for example, but is not limited to, the product of "the calculated average movement speed of the object" and "the length of time for which trajectory data is missing."

[0088] As shown in FIG. 9, the display control unit 14 displays, in the virtual space, a movement trajectory Q of the object in the real space indicated by the acquired trajectory data. １ and Q ２ and the estimated movement trajectory Q of the object in real space. １ ' are distinguishably displayed. In the figure, they are distinguishably displayed by different types of lines, but the method of distinguishing them is not limited to this. For example, the lines may be different in color, brightness, or thickness, or other methods may be used.

[0089] Furthermore, the display control unit 14 may reproduce the situation in real space in a virtual section based on the estimated trajectory data. An area lacking trajectory data is, for example, an area outside the camera's image capture area. For such an area (hereinafter referred to as a "non-image capture area"), no situation data is generated. As a result, the display control unit 14 cannot reproduce the situation in the non-image capture area in the virtual space. Therefore, the display control unit 14 may reproduce the situation in the non-image capture area in the virtual space based on the estimated trajectory data. The estimated trajectory data indicates where the object was located in the non-image capture area and at what time. Therefore, the display control unit 14 may display the object in the virtual space at the timing and position indicated by the estimated trajectory data.

[0090] Other configurations of the processing apparatus 10 of the third embodiment are similar to those of the processing apparatus 10 of the first and second embodiments.

[0091] According to the processing apparatus 10 of the third embodiment, the same effects as those of the processing apparatus 10 of the first and second embodiments are achieved.

[0092] Furthermore, the processing device 10 of the third embodiment can interpolate missing parts of the trajectory data of the object and use the interpolated data to reproduce the situation in the real world in a virtual space. In this way, the processing device 10 can predict and visualize parts that are not shown in the acquired data.

[0093] <<Fourth Embodiment>> A processing device 10 according to a fourth embodiment has a function of displaying notification information in a displayed virtual space. This will be described in detail below.

[0094] 10 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 includes a situation data acquisition unit 11, a trajectory data acquisition unit 12, a reception unit 13, a display control unit 14, and a notification information acquisition unit 15.

[0095] The notification information acquisition unit 15 acquires notification target information indicating a notification target among objects existing in real space. The notification target information indicates the timing (date, time, etc.) when the notification target was detected in real space and the position of the detected notification target.

[0096] A notification target is an object whose presence is to be notified to the user. For example, a person or vehicle that breaks a predetermined rule (e.g., a traffic rule) in the real world may be a notification target. Other notification targets may be people or vehicles that are listed in a pre-generated list. The list contains registered characteristics that appear on the appearance of the person or vehicle (e.g., facial information, gender, age, nationality, physique, license plate information, car model, manufacturer, year, body type, etc.).

[0097] In one example, the notification information acquisition unit 15 can generate notification target information by analyzing an image captured in real space and detecting a notification target in the image. For example, the notification information acquisition unit 15 may analyze an image to detect a vehicle that has committed a speeding violation or changed lanes at a location where a speeding violation has occurred. Alternatively, the notification information acquisition unit 15 may analyze an image to detect people, vehicles, etc. listed in a list from the image. These processes can be realized using any widely known image analysis technology.

[0098] In another example, an external device different from the processing device 10 can analyze an image captured in real space and detect the notification target in the image, thereby generating the notification target information. The notification information acquisition unit 15 then acquires the notification target information from the external device. The external device is the above-mentioned camera or a device that acquires and analyzes an image generated by the above-mentioned camera. The processing device 10 and the external device are connected to each other so that they can communicate with each other.

[0099] Note that the external device may detect a person, vehicle, etc. that has violated a predetermined rule (e.g., a traffic rule, etc.) based on the sensing result of a sensor other than a camera (e.g., a speed sensor, etc.) installed in real space instead of or in addition to an image. Then, the external device may generate notification target information based on the detection result.

[0100] The display control unit 14 displays the notification target in a identifiable manner in the virtual space. An example is shown in Fig. 11. In the example of Fig. 11, the notification target is displayed in an identifiable manner by displaying information "Notification 1" linked to the notification target. The method of displaying the notification target in an identifiable manner is not limited to this.

[0101] Both the situation data and the notification target information include date and time information indicating the date and time of each timing. Therefore, both the situation data and the notification target information indicate the location of each object at each timing. Therefore, by matching the positions of the objects at the same time, it is possible to identify which object among the objects indicated by the situation data the notification target information relates to.

[0102] As a modified example, when the display control unit 14 receives a user input specifying a single notification target, it may display related information related to the notification target. The related information is, for example, the reason why the notification target is selected. Examples of the reason include, but are not limited to, "speeding near XX intersection at XX / XX / XX hour and minute" or "registered on the blacklist." In this example, in response to a user input specifying location information (near XX intersection) included in the related information linked to the notification target, the reception unit 13 may receive the location indicated by the location information (near XX intersection) as a specific point on the movement trajectory of the notification target. Then, the display control unit 14 may change the display in the virtual space in response to the reception.

[0103] Other configurations of the processing apparatus 10 of the fourth embodiment are similar to those of the processing apparatus 10 of the first to third embodiments.

[0104] According to the processing apparatus 10 of the fourth embodiment, the same effects as those of the processing apparatus 10 of the first to third embodiments are realized.

[0105] Furthermore, the processing device 10 of the fourth embodiment can display notification targets in a identifiable manner in the virtual space, allowing the user to easily determine which object among the objects displayed in the virtual space the user should pay attention to.

[0106] <<Modifications>> Modifications applicable to the fourth embodiment having the configuration of the third embodiment will be described.

[0107] The notification information acquisition unit 15 may detect a notification target based on trajectory data indicating a predicted movement trajectory of the target. As described in the third embodiment, the trajectory data indicates the location of the target at each timing. Based on this data, the notification information acquisition unit 15 may calculate the movement speed and the position passed by the target within the section whose movement trajectory is predicted. Then, the notification information acquisition unit 15 may detect a speeding target or an object that has passed through a prohibited area as a notification target. In this modification, the same effects as those of the first to fourth embodiments are achieved.

[0108] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0109] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0110] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. 1. A processing device comprising: situation data acquisition means for acquiring situation data indicating a situation in real space; trajectory data acquisition means for acquiring trajectory data indicating a movement trajectory of an object in real space; reception means for receiving input specifying a specific point on the movement trajectory of the object in real space; and display control means for displaying a virtual space that reproduces the situation in the real world when the object is located at the specific point. 2. The processing device according to 1, wherein the reception means receives input for moving the position of the object displayed in the virtual space within the virtual space and specifying the position after the movement as the specific point. 3. The processing device according to 1 or 2, wherein the display control means displays the movement trajectory of the object in real space in the virtual space, and the reception means receives input for specifying a point on the movement trajectory of the object in real space displayed in the virtual space as the input specifying the specific point. 4. 4. The processing device according to any one of 1 to 3, wherein the display control means displays, in the virtual space, the movement trajectory of each of the plurality of objects in real space, and the receiving means receives, as the input for specifying the specific point, an input for specifying a point on any of the movement trajectories of each of the plurality of objects in real space displayed in the virtual space. 5. The processing device according to any one of 1 to 4, wherein the trajectory data acquisition means estimates, based on the trajectory data, the movement trajectory of the object for a section not indicated by the trajectory data, and the display control means distinguishably displays, in the virtual space, the movement trajectory of the object in real space indicated by the trajectory data from the estimated movement trajectory of the object in real space. 6. The processing device according to any one of 1 to 5, wherein the situation data indicates the position and type of an object existing in real space.7. The processing device according to 6, wherein the situation data further indicates at least one of a person's posture, a person's physique, a person's facial expression, a person's facial orientation, a person's body orientation, a person's hairstyle, a person's hair color, a person's clothing, a person's belongings, an item a person is wearing, a shape of a non-human object, a size of a non-human object, a color of a non-human object, a orientation of a non-human object, a pattern of a non-human object, a posture of a non-human object, and a lit color of a traffic light. 8. The processing device according to any of 1 to 7, further comprising an alarm information acquisition means for acquiring information indicating an alarm target among objects existing in real space, wherein the display control means displays the alarm target in an identifiable manner in the virtual space. 9. A processing method, wherein one or more computers: acquire situation data indicating a situation in real space; acquire trajectory data indicating a movement trajectory of an object in real space; accept input specifying a specific point on the movement trajectory of the object in real space; and display a virtual space that reproduces the situation in the real world when the object is located at the specific point. 10. A program that causes a computer to function as: situation data acquisition means that acquires situation data indicating a situation in real space; trajectory data acquisition means that acquires trajectory data that indicates a movement trajectory of an object in real space; reception means that accepts input specifying a specific point on the movement trajectory of the object in real space; and display control means that displays a virtual space that reproduces the situation in the real world when the object is located at the specific point.

[0111] Some or all of Supplements 2 to 8 that are dependent on the processing device of Supplement 1 described above may also be dependent on the processing method of Supplement 9 and the program of Supplement 10 in the same dependent relationship as Supplement 1 and Supplements 2 to 8. Furthermore, within the scope of each of the above-described embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0112] This application claims priority based on Japanese Patent Application No. 2024-024393, filed February 21, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0113] REFERENCE SIGNS LIST 10 Processing device 11 Situation data acquisition unit 12 Trajectory data acquisition unit 13 Reception unit 14 Display control unit 15 Notification information acquisition unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. A processing device having: a situation data acquisition means for acquiring situation data indicating a situation in real space; a trajectory data acquisition means for acquiring trajectory data indicating a movement trajectory of an object in real space; a receiving means for receiving an input specifying a specific point on the movement trajectory of the object in real space; and a display control means for displaying a virtual space that reproduces the situation in the real world when the object is located at the specific point.

2. The processing device according to claim 1, wherein the receiving means receives an input to move the position of the object displayed in the virtual space within the virtual space and specify the position after the movement as the specific point.

3. A processing device as described in claim 1 or 2, wherein the display control means displays the movement trajectory of the object in real space in the virtual space, and the receiving means receives input specifying a point on the movement trajectory of the object in real space displayed in the virtual space as input specifying the specific point.

4. A processing device according to any one of claims 1 to 3, wherein the display control means displays the movement trajectories of each of the plurality of objects in real space in the virtual space, and the receiving means receives, as the input for specifying the specific point, an input for specifying a point on any of the movement trajectories of each of the plurality of objects in real space displayed in the virtual space.

5. A processing device according to any one of claims 1 to 4, wherein the trajectory data acquisition means estimates the movement trajectory of the object in a section not indicated by the trajectory data based on the trajectory data, and the display control means displays, in the virtual space, the movement trajectory of the object in real space indicated by the trajectory data and the estimated movement trajectory of the object in real space in a distinguishable manner.

6. A processing device according to any one of claims 1 to 5, wherein the situation data indicates the position and type of an object present in real space.

7. The processing device of claim 6, wherein the situation data further indicates at least one of a person's posture, a person's physique, a person's facial expression, a person's facial orientation, a person's body orientation, a person's hairstyle, a person's hair color, a person's clothing, a person's belongings, an item worn by a person, a shape of a non-human object, a size of a non-human object, a color of a non-human object, an orientation of a non-human object, a pattern of a non-human object, a posture of a non-human object, and a color of a traffic light.

8. A processing device according to any one of claims 1 to 7, further comprising a notification information acquisition means for acquiring information indicating a notification target among objects existing in real space, wherein the display control means displays the notification target in an identifiable manner in the virtual space.

9. A processing method in which one or more computers acquire situation data indicating the situation in real space, acquire trajectory data indicating the movement trajectory of an object in real space, accept input specifying a specific point on the movement trajectory of the object in real space, and display a virtual space that reproduces the situation in the real world when the object is located at the specific point.

10. The processing method according to claim 9, wherein the input receiving process receives an input for moving the position of the object displayed in the virtual space within the virtual space and specifying the position after the movement as the specific point.

11. A processing method according to claim 9 or 10, wherein the process of displaying the virtual space displays the movement trajectory of the object in real space in the virtual space, and the process of accepting input accepts input specifying a point on the movement trajectory of the object in real space displayed in the virtual space as the input specifying the specific point.

12. A processing method according to any one of claims 9 to 11, wherein the process of displaying the virtual space displays the movement trajectories of each of the plurality of objects in real space in the virtual space, and the process of accepting input accepts, as the input for specifying the specific point, an input for specifying a point on any of the movement trajectories of each of the plurality of objects in real space displayed in the virtual space.

13. A processing method according to any one of claims 9 to 12, wherein in the process of acquiring the trajectory data, the movement trajectory of the object in a section not indicated by the trajectory data is estimated based on the trajectory data, and in the process of displaying the virtual space, the movement trajectory of the object in real space indicated by the trajectory data and the estimated movement trajectory of the object in real space are displayed in the virtual space in a distinguishable manner.

14. A processing method according to any one of claims 9 to 13, wherein the situation data indicates the position and type of an object present in real space.

15. A recording medium for recording a program that causes a computer to function as: situation data acquisition means for acquiring situation data indicating the situation in real space; trajectory data acquisition means for acquiring trajectory data indicating the movement trajectory of an object in real space; reception means for receiving input specifying a specific point on the movement trajectory of the object in real space; and display control means for displaying a virtual space that reproduces the situation in the real world when the object is located at the specific point.

16. A recording medium according to claim 15, wherein the receiving means receives an input to move the position of the object displayed in the virtual space within the virtual space and to specify the position after the movement as the specific point.

17. A recording medium as described in claim 15 or 16, wherein the display control means displays the movement trajectory of the object in real space in the virtual space, and the receiving means receives, as the input for specifying the specific point, an input for specifying a point on the movement trajectory of the object in real space displayed in the virtual space.

18. A recording medium described in any one of claims 15 to 17, wherein the display control means displays the movement trajectories of each of the plurality of objects in real space in the virtual space, and the receiving means receives, as the input for specifying the specific point, an input for specifying a point on any of the movement trajectories of each of the plurality of objects in real space displayed in the virtual space.

19. A recording medium described in any one of claims 15 to 18, wherein the trajectory data acquisition means estimates the movement trajectory of the object in a section not indicated by the trajectory data based on the trajectory data, and the display control means displays, in the virtual space, the movement trajectory of the object in real space indicated by the trajectory data and the estimated movement trajectory of the object in real space in a distinguishable manner.

20. A recording medium according to any one of claims 15 to 19, wherein the situation data indicates the position and type of an object existing in real space.

Images