Model display device, model display method, and program

Abstract

To appropriately track a moving object in a real space and display the moving object in a virtual space with a simple configuration, even when the moving object cannot be tracked.SOLUTION: A model display device includes: a setting unit which sets an allocation area within a display target range in which a sensor is installed; an obtaining unit which obtains a result of tracking a moving object recognized by the sensor; an update unit which updates information about a model representing the moving object based on the result of tracking the tracked moving object, and when a new moving object is recognized within the allocation area, updates information about the model representing the tracked moving object within the allocation area based on information about the new moving object; and an output unit which displays the model representing the moving object in a virtual space predetermined for the display target range.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a model display device, a model display method, and a program. [Background technology]

[0002] BACKGROUND ART Systems are known that allow a computer system constituting a cyber world to recognize real-world object information (for example, Patent Document 1).

[0003] In the system described in Patent Document 1, the real world is modeled on a mono-basis, and real-world objects are recognized by attaching an accessory element with an ID issuing unit to them. While the issuing unit cannot be recognized, the system continues tracking by tracking movement from past points in time and performing temporary interpolation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-147642 Summary of the Invention [Problem to be solved by the invention]

[0005] In order for the system described in Patent Document 1 to recognize people and objects, it is necessary to attach additional elements to them, which means that people and objects that cannot be controlled cannot be tracked, and people and objects in the real world cannot be represented in virtual space.

[0006] Furthermore, when people and objects are not allowed to transmit information, there is a method of reproducing them using recognition data from infrastructure sensors, etc. However, when tracking cannot be continued due to occlusion by objects or other people, the method of reproducing the model in virtual space becomes an issue.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and aims to appropriately track a moving object in real space and display it in virtual space using a simple configuration, even when it is not possible to track the moving object. [Means for solving the problem]

[0008] In order to achieve the above object, the model display device of the first aspect is configured to include: a setting unit that sets an allocation area within a display target range in which a sensor is installed; an acquisition unit that acquires the results of tracking a moving body recognized by the sensor; an update unit that updates information about a model representing the moving body based on the results of tracking the tracked moving body, and when a new moving body is recognized within the allocation area, updates information about the model representing the tracked moving body within the allocation area based on information about the new moving body; and an output unit that displays the model representing the moving body in a virtual space predetermined for the display target range.

[0009] The model display method of the second aspect is a method executed by a computer, which sets an assigned area within a display target range in which a sensor is installed, obtains the results of tracking a moving object recognized by the sensor, updates information of a model representing the moving object based on the results of tracking the tracked moving object, and when a new moving object is recognized within the assigned area, updates information of a model representing the tracked moving object within the assigned area based on information about the new moving object, and displays the model representing the moving object in a virtual space predetermined for the display target range.

[0010] In addition, a model display program according to a third aspect is a program that causes at least one processor to set an allocation area within a display target range in which a sensor is installed, obtain results of tracking a moving body recognized by the sensor, update information on a model representing the moving body based on the results of tracking the tracked moving body, and when a new moving body is recognized within the allocation area, update information on a model representing the tracked moving body within the allocation area based on information on the new moving body, and display the model representing the moving body in a virtual space predetermined for the display target range. [Effects of the Invention]

[0011] According to the model display device, method, and program disclosed herein, even if it is not possible to track a moving object, a simple configuration can be used to appropriately track a moving object in real space and display it in virtual space. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating a configuration of a model display system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a functional block diagram of a recognition device according to an embodiment of the present disclosure. [Figure 3] FIG. 2 is a functional block diagram of a server according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a hardware configuration of a model display device according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a functional block diagram of a model display device according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is a diagram showing an example in which an allocation area is set based on stationary objects determined in advance within a specific area. [Figure 7] 10A and 10B are diagrams illustrating examples of posture data included in model information and a model in a virtual space. [Figure 8] 10 is a flowchart showing an example of a setting process according to the first embodiment of the present disclosure. [Figure 9]10 is a flowchart illustrating an example of a model display process according to an embodiment of the present disclosure. [Figure 10] 10 is a flowchart illustrating an example process for allocating models representing tracked people within an allocation area. [Figure 11] 10 is a flowchart illustrating an example of a model update process according to an embodiment of the present disclosure. [Figure 12] FIG. 10 is a diagram illustrating an example of identifying an area where occlusion by a moving object occurs. [Figure 13] FIG. 10 is a diagram showing an example in which allocation areas are set based on areas where occlusion by a moving object occurs. [Figure 14] 10 is a flowchart showing an example of a setting process according to a second embodiment of the present disclosure. [Figure 15] FIG. 10 is a diagram showing an example in which an allocation area is set based on an area surrounding a specified group of people. [Figure 16] FIG. 10 illustrates an example of allocating a model representing a tracked person within an allocation area. [Figure 17] 10 is a flowchart showing an example of a setting process according to a third embodiment of the present disclosure. [Figure 18] FIG. 10 is a diagram showing an example in which allocation areas are set based on predicted movement of a person's position. [Figure 19] FIG. 10 illustrates an example of allocating a model representing a tracked person within an allocation area. [Figure 20] 10 is a flowchart showing an example of a setting process according to the fourth embodiment of the present disclosure. [Figure 21] FIG. 10 is a diagram showing an example in which allocation areas overlap each other. [Figure 22] FIG. 10 illustrates an example of allocating a model representing a tracked person within an allocation area. [Figure 23] 13 is a flowchart showing an example of overlap area addition processing according to the fifth embodiment of the present disclosure. [Figure 24] 10 is a flowchart illustrating an example process for allocating models representing tracked people within an allocation area. [Figure 25] FIG. 10 is a diagram illustrating an example in which an allocation area is set based on an overlapping region of detection ranges of a plurality of sensors. DETAILED DESCRIPTION OF THE INVENTION

[0013] Before describing the details of the embodiments of the present disclosure, an overview of the embodiments of the present disclosure will be described.

[0014] <Summary of Embodiments of the Present Disclosure> In a system that recreates a moving object in a virtual space using data (e.g., a person's posture data) recognized by a sensor in a specific area that is the display range to be displayed as virtual space, if it is temporarily unable to track the moving object due to occlusion, etc., there will be no data to provide to the model in the virtual space, so possible responses include turning off the display of the model, or waiting for data while keeping the model displayed at the final value or movement prediction.

[0015] In other words, even if the data is interrupted, the actual moving object remains there, so the model continues to be displayed in the virtual space even if there is no data, and while the data is interrupted, the model in the virtual space is represented based on movement predictions, etc.

[0016] Furthermore, even if the moving object is recognized again and data is obtained, if the unique ID of the moving object being tracked is updated, there remains the issue of reassigning the re-recognized data to the waiting model, making it difficult to represent the real world in virtual space.

[0017] Therefore, in this embodiment, even if tracking is temporarily impossible due to the occurrence of occlusion, etc., when the moving object is recognized again, the tracked model is assigned, thereby properly tracking the moving object in real space and displaying a model representing the moving object in virtual space.

[0018] Specifically, an allocation area is statically or dynamically set in a specific area that is the display target range, and if the position of a recognized moving object during re-recognition is within the allocation area, the data is assigned to the closest model by comparing it with the features of the model corresponding to the allocation area.As a result, even if tracking is temporarily impossible or the ID of a recognized moving object is switched, by assigning the tracked model, it is possible to properly track the moving object in real space and display a model representing the moving object in virtual space.

[0019] [First embodiment] As shown in FIG. 1, the model display system 10 according to the first embodiment includes a plurality of sensors 20 provided for a specific area in real space, a recognition device 22 connected to each sensor 20, a server 40 that collects the results of moving object tracking by the recognition device 22, and a model display device 100 that displays a model representing the tracked moving object in a virtual space predetermined for the specific area.

[0020] In this embodiment, a case will be described in which a person is used as an example of a moving object to be tracked. For simplicity, Fig. 1 shows an example in which two sensors 20 are provided, but three or more sensors 20 may be provided. Furthermore, the moving object to be tracked may be something other than a person, such as an animal or a robot.

[0021] Each sensor 20 is connected to a recognition device 22. The recognition device 22, the server 40, and the model display device 100 are connected via a network N such as a LAN (Local Area Network) or the Internet.

[0022] The sensor 20 is a fixed sensor provided in a specific area in real space, and includes, for example, a camera, a LiDAR (Light Detection and Ranging), and the like.

[0023] The recognition device 22 is a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and functionally includes a data acquisition unit 30, a recognition unit 32, a tracking unit 34, and a communication unit 36, as shown in FIG. 2.

[0024] The data acquisition unit 30 acquires the output of the sensor 20 .

[0025] The recognition unit 32 detects a specific moving object (e.g., a person) based on the output of the sensor 20, acquires data expressed by key points indicating each joint and each part of the detected person, and recognizes posture data from the data expressed by the key points. The posture data includes, for example, the person's position (three-dimensional coordinates), orientation (orientation on the three-dimensional coordinates), height, physique (length of each part), moving speed, stride length, and relationship with surrounding people. Note that the relationship with surrounding people may be obtained by a device other than the recognition device 22 (e.g., the server 40).

[0026] The tracking unit 34 tracks the person based on the posture data of the recognized person and assigns an ID to identify the person. If the person is successfully tracked, the same ID is assigned to the person, and if the person is not successfully tracked, a different ID is assigned to each recognized person. The processing of the tracking unit 34 may be performed by a device other than the recognition device 22 (for example, a server 40).

[0027] The communication unit 36 ​​transmits to the server 40 person tracking information including the posture data and ID of the recognized person.

[0028] The server 40 is a computer having a CPU, a ROM, a RAM, and a storage, and functionally includes a collection unit 42 and an aggregation unit 44, as shown in FIG.

[0029] The collection unit 42 collects the person tracking information transmitted from each recognition device 22 .

[0030] The aggregation unit 44 aggregates the collected person tracking information and transmits it to the model display device 100. Specifically, when it is determined that the person tracking information represents the same person based on the posture data included in the person tracking information, the information is aggregated into one person tracking information.

[0031] Fig. 4 is a block diagram showing the hardware configuration of the model display device 100 of this embodiment. As shown in Fig. 4, the model display device 100 has a CPU 11, a ROM 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0032] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a model display program.

[0033] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0034] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0035] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0036] The communication interface 17 is an interface for communicating with other devices such as terminals, etc. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0037] Next, we will explain each functional configuration of the model display device 100. Figure 5 is a block diagram showing the configuration of the model display device 100 of this embodiment. Each functional configuration is realized by the CPU 11 reading out a model display program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0038] As shown in FIG. 5, the model display device 100 is configured to include a setting unit 102, an acquisition unit 104, an update unit 106, a learning unit 108, an output unit 110, an area database (DB) 112, a model database (DB) 114, and a virtual space database (DB) 116.

[0039] The setting unit 102 sets an allocation area based on a stationary object determined in advance within the specific area. Specifically, as shown in Fig. 6 , the setting unit 102 identifies an area 152 in real space where occlusion occurs based on the positional relationship between the sensor 20 and the stationary object 150 within the specific area and the shape of the stationary object 150, and sets an area including the identified area 152 as the allocation area 160. For example, an area expanded around the identified area 152 by the size of a person is set as the allocation area 160.

[0040] The acquisition unit 104 acquires the results of tracking the moving object recognized by the sensor 20. Specifically, the acquisition unit 104 acquires person tracking information transmitted from the server 40.

[0041] If the person tracking information is the result of tracking a person who has already been tracked, the update unit 106 updates the information of the model representing that person. The information of the model includes posture data.

[0042] When the person tracking information represents a person in the allocation area 160 and indicates that a new person has been recognized, the update unit 106 updates the information of the model representing the person already tracked in the allocation area 160 based on the information of the new person represented by the person tracking information. Specifically, when a new person is recognized in the allocation area 160, the update unit 106 identifies which of the models representing people already tracked in the allocation area 160 the new person represented by the person tracking information is, based on the posture data of the new person represented by the person tracking information and data for identifying the model representing the person already tracked in the allocation area 160, and updates the information of the identified model.

[0043] 6, when a new person is recognized within the allocation area 160, the new person represented by the person tracking information is identified as a model (ID=10) representing a person who has already been tracked within the allocation area 160, and the information of the identified model (ID=10) is updated. Also, when a new person is recognized within the allocation area 160, the new person represented by the person tracking information is identified as a model (ID=11) representing a person who has already been tracked within the allocation area 160, and the information of the identified model (ID=11) is updated.

[0044] Furthermore, for models representing tracked people that have not been updated based on person tracking information, the update unit 106 updates the information of the model based on the results of movement prediction performed on the tracked people.

[0045] The learning unit 108 learns data for identifying each model representing a tracked person based on person tracking information about the person representing the model. Specifically, for each model representing a tracked person, based on the accumulated person tracking information about the person representing the model, the learning unit 108 learns features of pose data or a classifier as data for identifying the model.

[0046] The output unit 110 displays a model representing each person in a predetermined virtual space. Specifically, as shown in Fig. 7, the output unit 110 generates a model in the virtual space from posture data included in the model information, and displays the generated model in the virtual space that reproduces a specific area.

[0047] The area database (DB) 112 stores information about the allocation area 160 set by the setting unit 102. The information about the allocation area 160 includes, for example, the range of the allocation area 160 in real space.

[0048] The model database (DB) 114 stores information about each model representing a tracked person and data for identifying the model. The model information includes a model ID and pose data.

[0049] The virtual space database (DB) 116 stores information about a virtual space in which a specific area is reproduced, which has been obtained in advance. The information about the virtual space includes, for example, three-dimensional data in which the specific area is reproduced in the virtual space.

[0050] Next, the operation of the model display system 10 will be described.

[0051] 8 is a flowchart showing the flow of the setting process by the model display device 100. The setting process is performed in advance by the CPU 11 reading out the model display program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The setting process is an example of a model display method. The model display device 100 receives an allocation area setting instruction as input and performs the following process.

[0052] In step S100, the CPU 11 functions as the setting unit 102 to acquire the position of the sensor 20 within the specific area, the detection range of the sensor 20, the position of the stationary object 150, and the shape of the stationary object 150, which have been determined in advance.

[0053] In step S102, the CPU 11, as the setting unit 102, identifies an area 152 in which occlusion occurs in real space based on the position of the sensor 20 within the specific area, the detection range of the sensor 20, the position of the stationary object 150, and the shape of the stationary object 150, sets an area including the identified area 152 as the assigned area 160, and terminates the setting process.

[0054] In the recognition device 22, a data acquisition unit 30 acquires the output of a sensor 20 installed in a specific area, and a recognition unit 32 detects a person based on the output of the sensor 20 and recognizes posture data of the detected person. A tracking unit 34 tracks the person based on the posture data of the recognized person and assigns an ID to identify the person. A communication unit 36 ​​then transmits person tracking information including the posture data and ID of the recognized person to a server 40.

[0055] 9 is a flowchart showing the flow of model display processing by the model display device 100. The model display processing is performed by the CPU 11 reading out a model display program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The model display processing is an example of a model display method. When person tracking information has started to be transmitted from the recognition device 22 to the model display device 100 via the server 40, the model display device 100 repeatedly performs the following processing.

[0056] In step S110, the CPU 11, functioning as the acquisition unit 104, determines whether or not person tracking information has been received from the server 40. If person tracking information has been received from the server 40, the process proceeds to step S112. On the other hand, if person tracking information has not been received from the server 40, the model display process ends. Note that person tracking information may be received directly from the recognition device 22.

[0057] In step S112, the CPU 11, functioning as the update unit 106, determines whether the ID included in the received person tracking information is the ID of a person who has already been tracked. If the ID included in the received person tracking information is the ID of a person who has already been tracked, the process proceeds to step S114. On the other hand, if the ID included in the received person tracking information is not the ID of a person who has already been tracked, the process determines that a new person has been recognized, and the process proceeds to step S116.

[0058] In step S114, the CPU 11 functions as the update unit 106 and assigns the received person tracking information to the tracked model with the same ID.

[0059] In step S116, the CPU 11, functioning as the update unit 106, determines whether or not the position included in the received person tracking information is within the assigned area. If the position included in the received person tracking information is within the assigned area, the process proceeds to step S118, and if the position included in the received person tracking information is not within the assigned area, the process proceeds to step S120.

[0060] In step S118, the CPU 11, functioning as the update unit 106, allocates the received person tracking information to a model representing a person that has been tracked within the allocation area 160.

[0061] In step S120, the CPU 11, functioning as the update unit 106, generates a new model and assigns the received person tracking information to the new model.

[0062] In step S122, the CPU 11 functions as the update unit 106 to update the information of the model to which the received person tracking information is assigned, using the posture data of the person tracking information.

[0063] In step S124, the CPU 11 functions as the output unit 110 to display the updated model in the virtual space, and then ends the model display process.

[0064] The above step S118 is realized by the processing routine shown in FIG.

[0065] In step S130, the CPU 11 functions as the update unit 106 and acquires the received person tracking information.

[0066] In step S132, the CPU 11, as the update unit 106, identifies the closest model from the models representing the tracked person within the allocation area 160, based on the posture data of the new person represented by the acquired person tracking information and data for identifying the model representing the tracked person within the allocation area 160.

[0067] In step S134, the CPU 11, functioning as the update unit 106, assigns the received person tracking information to the identified closest human model, and ends the processing routine.

[0068] 11 is a flowchart showing the flow of model update processing by the model display device 100. The model update processing is performed by the CPU 11 reading out a model display program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. The model update processing is an example of a model display method. When transmission of person tracking information from the recognition device 22 has begun, the model display device 100 repeatedly performs the following processing for each model representing a tracked person stored in the model database 114, each time the display in the virtual space is updated.

[0069] In step S140, CPU 11 determines whether or not person tracking information has been assigned to the target model since the previous update timing. If person tracking information has not been assigned to the target model, the process proceeds to step S142. On the other hand, if person tracking information has been assigned to the target model, the process proceeds to step S148.

[0070] In step S142, the CPU 11, functioning as the update unit 106, predicts the movement of the target model based on the posture data included in the information about the target model.

[0071] In step S144, the CPU 11 functions as the update unit 106 to generate posture data of the model after movement based on the result of the movement prediction of the target model.

[0072] In step S146, the CPU 11 functions as the update unit 106 to update the information of the target model based on the generated posture data of the model after movement, and then ends the model update process.

[0073] In step S148, the CPU 11, functioning as the learning unit 108, adds the assigned person tracking information for the target model to the past data accumulated for the target model.

[0074] In step S150, the CPU 11 functions as the learning unit 108 to learn data for identifying the target model based on the accumulated person tracking information for the target model, and then ends the model update process.

[0075] As described above, the model display device according to the first embodiment sets an assignment area within a specific area in which sensors are installed, acquires the results of tracking moving objects recognized by the sensors, updates information about a model representing the moving object based on the results of tracking the previously tracked moving object, and when a new moving object is recognized within the assignment area, updates information about a model representing the previously tracked moving object within the assignment area based on information about the new moving object, and displays the model representing the moving object in a virtual space. This makes it possible to appropriately track a moving object in real space and display it in a virtual space with a simple configuration, even when it is not possible to track the moving object.

[0076] Furthermore, by setting an allocation area based on a stationary object, even if an occlusion by a stationary object occurs and a moving object cannot be tracked, the moving object in the real space can be properly tracked and displayed in the virtual space.

[0077] [Second embodiment] Next, a description will be given of a model display system according to a second embodiment. Note that the configuration of the model display system according to the second embodiment is the same as that of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0078] The second embodiment differs from the first embodiment in that the allocation area is dynamically set based on the area where occlusion occurs due to a moving object.

[0079] The recognition unit 32 of the recognition device 22 according to the second embodiment detects a person based on the output of the sensor 20, and recognizes the posture data of the detected person.

[0080] The recognition unit 32 detects all moving objects, including moving objects other than people, based on the output of the sensor 20, and recognizes the size or shape of the detected moving objects.

[0081] The communication unit 36 ​​transmits to the server 40 person tracking information including the posture data and ID of the recognized person, and moving body information including the size or shape of the recognized moving body.

[0082] The collection unit 42 of the server 40 collects the person tracking information and moving object information transmitted from each recognition device 22 .

[0083] The aggregation unit 44 aggregates the collected person tracking information and transmits it to the model display device 100. The aggregation unit 44 also aggregates the collected moving body information and transmits it to the model display device 100.

[0084] The setting unit 102 of the model display device 100 sets an assigned area based on the moving body information received from the server 40. Specifically, as shown in Fig. 12, the setting unit 102 identifies an area 252 in real space where occlusion occurs based on the positional relationship between the sensor 20 and the moving body 250 in the specified area and the size or shape of the moving body 250, and sets an area including the identified area 152 as an assigned area 260, as shown in Fig. 13.

[0085] Furthermore, similarly to the first embodiment, the setting unit 102 sets an allocation area 152 based on stationary objects 150 determined in advance within a specific area.

[0086] Next, the operation of the model display system 10 will be described.

[0087] 14 is a flowchart showing the flow of the setting process by the model display device 100. The setting process is performed by the CPU 11 reading out the model display program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The setting process is an example of a model display method. The model display device 100 receives moving body information from the server 40 as input and performs the following processes.

[0088] In step S160, the CPU 11 functioning as the setting unit 102 acquires the mobile object information received from the server 40.

[0089] In step S162, the CPU 11, as the setting unit 102, identifies an area 252 in which occlusion occurs in real space based on the positional relationship between the sensor 20 and the moving body 250 in the specific area, and the size or shape of the moving body 250, obtained from the received moving body information.

[0090] In step S164, the CPU 11, functioning as the setting unit 102, sets an area including the identified area 252 as the assigned area 260, and ends the setting process.

[0091] Furthermore, the model display device 100 performs the setting process shown in FIG. 8, similarly to the first embodiment.

[0092] The other configurations and operations of the model display system 10 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0093] As described above, the model display device according to the second embodiment sets an assigned area within a specific area in which sensors are installed, acquires the results of tracking moving objects recognized by the sensors, updates information about a model representing the moving object based on the results of tracking the previously tracked moving object, and when a new moving object is recognized within the assigned area, updates information about a model representing the previously tracked moving object within the assigned area based on information about the new moving object, and displays the model representing the moving object in a virtual space. This makes it possible to appropriately track a moving object in real space and display it in a virtual space with a simple configuration, even when it is not possible to track the moving object.

[0094] Furthermore, by setting an allocation area based on a moving object, even if an occlusion by the moving object occurs and the moving object cannot be tracked, the moving object in the real space can be properly tracked and displayed in the virtual space.

[0095] [Third embodiment] Next, a model display system according to a third embodiment will be described. Note that the configuration of the model display system according to the third embodiment is the same as that of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0096] The third embodiment differs from the first embodiment in that the allocation area is dynamically set based on the area where occlusion occurs due to a group of moving objects.

[0097] The setting unit 102 of the model display device 100 identifies a group of people consisting of recognized people based on the person tracking information received from the server 40, and sets an allocation area based on the group of people. Specifically, as shown in Fig. 15 , the setting unit 102 identifies the group of people based on the relationship with surrounding people included in the posture data of the person tracking information, identifies an area 352 surrounding the identified group of people, and sets an area including the identified area 352 as the allocation area 360.

[0098] As a result, as shown in Figure 16, when a new person is recognized within allocation area 360 including area 352 surrounding a group of people, the new person represented by the person tracking information is assigned to one of the models representing people that have already been tracked within allocation area 360 based on the posture data of the new person represented by the person tracking information and data for identifying the model representing the person that has already been tracked within allocation area 360.

[0099] Next, the operation of the model display system 10 will be described.

[0100] 17 is a flowchart showing the flow of the setting process by the model display device 100. The setting process is performed by the CPU 11 reading out the model display program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The setting process is an example of a model display method. The model display device 100 receives person tracking information from the server 40 as input, and performs the following processes.

[0101] In step S170, the CPU 11 functioning as the setting unit 102 acquires the person tracking information received from the server 40.

[0102] In step S172, the CPU 11 as the setting unit 102 identifies a group of people based on the relationships between surrounding people included in the posture data of the received person tracking information, and identifies an area 352 surrounding the identified group of people.

[0103] In step S174, the CPU 11, functioning as the setting unit 102, sets an area including the identified area 352 as the assigned area 360, and ends the setting process.

[0104] The other configurations and operations of the model display system 10 according to the third embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0105] As described above, the model display device according to the third embodiment sets an assigned area within a specific area in which sensors are installed, acquires the results of tracking moving objects recognized by the sensors, updates information about a model representing the moving object based on the results of tracking the previously tracked moving object, and when a new moving object is recognized within the assigned area, updates information about a model representing the previously tracked moving object within the assigned area based on information about the new moving object, and displays the model representing the moving object in a virtual space. This makes it possible to appropriately track a moving object in real space and display it in a virtual space with a simple configuration, even when it is not possible to track the moving object.

[0106] Furthermore, by setting an allocation area based on a group of people, even if occlusion by the group of people occurs and a moving object cannot be tracked, the moving object in the real space can be properly tracked and displayed in the virtual space.

[0107] [Fourth embodiment] Next, a model display system according to a fourth embodiment will be described. Note that the configuration of the model display system according to the fourth embodiment is the same as that of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0108] The fourth embodiment differs from the first embodiment in that allocation areas are dynamically set based on predicted movement of a person's position.

[0109] The setting unit 102 of the model display device 100 of the fourth embodiment predicts the movement of the position of the recognized person based on the person tracking information received from the server 40, and sets an allocation area based on the result of the movement prediction. Specifically, as shown in Fig. 18 , the setting unit 102 predicts the movement of the person's position based on the position, orientation, and movement speed of the person included in the posture data of the person tracking information, identifies an area 452 that represents the predicted destination, and sets an area including the identified area 452 as the allocation area 460.

[0110] As a result, as shown in Figure 19, when a new person is recognized within an allocation area 460 including an area 452 representing the predicted destination of the person, the new person represented by the person tracking information is assigned to one of the models representing people that have already been tracked within the allocation area 460 based on the posture data of the new person represented by the person tracking information and data for identifying the model representing the person that has already been tracked within the allocation area 460.

[0111] Next, the operation of the model display system 10 will be described.

[0112] 20 is a flowchart showing the flow of the setting process by the model display device 100. The setting process is performed by the CPU 11 reading out the model display program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The setting process is an example of a model display method. The model display device 100 receives person tracking information from the server 40 as input, and performs the following processes.

[0113] In step S180, the CPU 11 functioning as the setting unit 102 acquires the person tracking information received from the server 40.

[0114] In step S182, the CPU 11, as the setting unit 102, predicts the movement of the person's position based on the position, orientation, and movement speed of the person contained in the posture data of the received person tracking information, and identifies an area 452 representing the predicted destination.

[0115] In step S184, the CPU 11, functioning as the setting unit 102, sets an area including the identified area 452 as the allocated area 460, and ends the setting process.

[0116] The other configurations and operations of the model display system 10 according to the fourth embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0117] As described above, the model display device according to the fourth embodiment sets an assigned area within a specific area in which sensors are installed, acquires the results of tracking moving objects recognized by the sensors, updates information about a model representing the moving object based on the results of tracking the previously tracked moving object, and when a new moving object is recognized within the assigned area, updates information about a model representing the previously tracked moving object within the assigned area based on information about the new moving object, and displays the model representing the moving object in a virtual space. This makes it possible to appropriately track a moving object in real space and display it in a virtual space with a simple configuration, even when it is not possible to track the moving object.

[0118] Furthermore, by setting an allocation area based on the predicted movement of a person, even if it is temporarily impossible to track a moving object, the moving object in the real space can be properly tracked and displayed in the virtual space.

[0119] [Fifth embodiment] Next, a model display system according to a fifth embodiment will be described. Note that the configuration of the model display system according to the fifth embodiment is the same as that of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0120] The fifth embodiment differs from the first embodiment in that, when multiple allocation areas overlap each other, the other overlapping allocation areas are also referenced to identify the model to which person tracking information is assigned.

[0121] When overlapping occurs between set allocation areas, the setting unit 102 of the model display device 100 of the fifth embodiment adds information about the other overlapping allocation areas to the overlapping allocation area.

[0122] When a new moving object is recognized within an assigned area and part of the assigned area overlaps with another assigned area, the update unit 106 identifies whether the new moving object is a model representing a moving object that has already been tracked within the assigned area or a model representing a moving object that has already been tracked in another assigned area, and updates the information of the identified model.

[0123] For example, as shown in Fig. 21, by detecting overlap between assignment areas 160 and 360 and adding information about the overlapping assignment area to each assignment area, it is possible to appropriately assign a model within the assignment area when a new person is recognized in either assignment area. As shown in Fig. 22, when a person moves from dynamic assignment area 360 to static assignment area 160 and is recognized as a new person within assignment area 160, it is possible to assign not only assignment area 160 but also the closest model among models representing moving objects that have already been tracked in assignment area 360.

[0124] Incidentally, detection of overlapping of assigned areas may be performed, for example, by generating objects for the assigned areas in the virtual space and using collision detection between these objects as a trigger.

[0125] 23 is a flowchart showing the flow of overlapping area addition processing by the model display device 100. The CPU 11 reads out a model display program from the ROM 12 or storage 14, expands it in the RAM 13, and executes it to perform overlapping area addition processing. The overlapping area addition processing is an example of a model display method. When transmission of person tracking information from the recognition device 22 has started, the model display device 100 repeatedly performs the following processing.

[0126] In step S190, the CPU 11, functioning as the setting unit 102, determines whether or not overlapping of the assigned areas has occurred. If overlapping of the assigned areas has occurred, the process proceeds to step S192, whereas if overlapping of the assigned areas has not occurred, the overlapping area addition process ends.

[0127] In step S192, the CPU 11 as the setting unit 102 adds information about other overlapping allocation areas to each overlapping allocation area as information to be referenced during allocation, and ends the overlapping area addition process.

[0128] As in the first embodiment, the model display device 100 executes the model display processing shown in Fig. 9. Here, step S118 in the model display processing is realized by the processing routine shown in Fig. 24.

[0129] In step S200, the CPU 11 functions as the update unit 106 and acquires the received person tracking information.

[0130] In step S202, the CPU 11, functioning as the update unit 106, determines whether there is information about other overlapping assigned areas that was added as information to be referenced at the time of allocation. If there is information about other overlapping assigned areas that was added as information to be referenced at the time of allocation, the process proceeds to step S204. On the other hand, if there is no information about other overlapping assigned areas that was added as information to be referenced at the time of allocation, the process proceeds to step S206.

[0131] In step S204, the CPU 11 functions as the update unit 106 and acquires data for identifying a model representing a person who has been tracked within another overlapping allocation area, which data was added as information to be referenced at the time of allocation.

[0132] In step S206, the CPU 11, as the update unit 106, identifies the closest model from the models representing the tracked person within the assigned area 160 or the other overlapping assigned areas, based on the posture data of the new person represented by the acquired person tracking information and data for identifying a model representing the tracked person within the assigned area 160 or data for identifying a model representing the tracked person within the other overlapping assigned areas.

[0133] In step S208, the CPU 11, functioning as the update unit 106, assigns the received person tracking information to the identified closest model, and ends the processing routine.

[0134] The other configurations and operations of the model display system 10 according to the fifth embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0135] As described above, when multiple assigned areas overlap, the model display device of the fifth embodiment identifies a model to which information about a newly recognized person is assigned from a model representing a person who has already been tracked within the assigned area and a model representing a person who has already been tracked in other overlapping assigned areas, thereby making it possible to appropriately track a moving object in real space and display it in virtual space even when the moving object moves across multiple assigned areas.

[0136] [Sixth embodiment] Next, a model display system according to a sixth embodiment will be described. Note that the configuration of the model display system according to the sixth embodiment is the same as that of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0137] The sixth embodiment differs from the first embodiment in that an allocation area is set based on an overlapping region of the detection ranges of a plurality of sensors.

[0138] In the sixth embodiment, as shown in Fig. 25, a plurality of sensors 20 have different monitoring ranges as their detection ranges. In Fig. 25, the detection range of one sensor 20 is set to monitor area 1, and the detection range of the other sensor 20 is set to monitor area 2.

[0139] The setting unit 102 of the model display device 100 sets an allocation area based on the overlapping region of the detection ranges of the multiple sensors 20.

[0140] Specifically, an overlapping area 652 of the detection ranges is identified based on the detection ranges of the plurality of sensors 20 that have been determined in advance, and an area including the overlapping area 652 is set as the assigned area 660.

[0141] For example, if one sensor 20 has insufficient detection range or accuracy due to a person's movement, and the other sensor 20 detects the person, handover based on the assigned area becomes possible. Also, since the model is shared and becomes one, it is possible to deal with cases where duplicate detections occur. In other words, even if the recognizing sensor 20 is changed, the ID can be passed on, and by avoiding duplicate displays, it is possible to grasp the number of people in the area.

[0142] As described above, when the detection ranges of multiple sensors overlap, the model display device of the sixth embodiment sets an allocation area based on the overlapping area, so that even if a moving object moves across the detection ranges of multiple sensors, the moving object in the real space can be appropriately tracked and displayed in the virtual space.

[0143] <Modification> The setting unit may set the assigned area based on a predetermined detection range of the sensor. For example, the setting unit may set the assigned area at the boundary (switching unit) of the detection range of the sensor. This allows the moving object in the real space to be properly tracked and displayed in the virtual space even if the moving object moves outside the detection range of the sensor. Furthermore, the methods for setting allocation areas described in each embodiment may be combined.

[0144] Furthermore, various processes executed by the CPU after reading software (programs) in the above-described embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) whose circuit configuration can be changed after fabrication, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. Furthermore, various processes may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0145] In the above embodiment, the program is pre-stored (installed) in a ROM, but the present invention is not limited to this. The program may be provided in a form recorded on a non-transitory tangible storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0146] The controller and methods described herein may be implemented by a special-purpose computer having a processor programmed to perform one or more functions embodied in a computer program. Alternatively, the apparatus and methods described herein may be implemented by a special-purpose computer having a processor configured with dedicated hardware logic circuitry. Alternatively, the apparatus and methods described herein may be implemented by one or more special-purpose computers configured by a combination of a processor executing a computer program and one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0147] <Additional Notes> The features of the present invention are as follows.

[0148] (Appendix 1) a setting unit that sets an allocation area within a display target range in which the sensor is installed; an acquisition unit that acquires a result of tracking a moving object recognized by the sensor; updating information of a model representing the tracked moving object based on a result of tracking the tracked moving object; an update unit that, when a new moving object is recognized within the assigned area, updates information of a model representing a moving object that has already been tracked within the assigned area based on information of the new moving object; an output unit that displays a model representing the moving object in a virtual space that is predetermined for the display target range; A model display device including:

[0149] (Appendix 2) 2. The model display device according to claim 1, wherein the setting unit sets the allocation area based on a result of movement prediction performed on the recognized moving object.

[0150] (Appendix 3) 3. The model display device according to claim 1, wherein the setting unit sets the allocation area based on a group of moving objects consisting of the recognized moving objects.

[0151] (Appendix 4) 4. The model display device according to claim 1, wherein the setting unit sets the allocation area based on a stationary object determined in advance in the display target range.

[0152] (Appendix 5) 5. The model display device according to any one of Supplementary Note 1 to Supplementary Note 4, wherein the setting unit sets the allocation area based on the recognized moving object.

[0153] (Appendix 6) 6. The model display device according to any one of Supplementary Note 1 to Supplementary Note 5, wherein the setting unit sets the assigned area based on a detection range of the sensor that is determined in advance.

[0154] (Appendix 7) a learning unit that learns data for identifying the model based on information about a moving object that represents the model; The update unit A model display device according to any one of appendices 1 to 6, wherein when a new moving object is recognized within the assigned area, the device identifies which of the models representing moving objects already tracked within the assigned area the new moving object is based on information about the new moving object and data for identifying the model, and updates information about the identified model.

[0155] (Appendix 8) A model display device according to any one of Supplementary Note 1 to Supplementary Note 7, wherein when a new moving object is recognized within the assigned area and part of the assigned area overlaps with another of the assigned areas, the update unit identifies whether the new moving object is a model representing a moving object that has already been tracked within the assigned area or a model representing a moving object that has already been tracked in another of the assigned areas, and updates information about the identified model.

[0156] (Appendix 9) The model display device according to any one of Supplementary Note 1 to Supplementary Note 8, wherein the update unit updates information of each model representing a tracked moving body based on the results of tracking the tracked moving body or the results of movement prediction for the tracked moving body.

[0157] (Appendix 10) The sensor is a plurality of sensors, 10. The model display device according to any one of Supplementary Note 1 to Supplementary Note 9, wherein the setting unit sets the assigned area based on an overlapping region of detection ranges of the plurality of sensors.

[0158] (Appendix 11) 11. The model display device according to any one of Supplementary Note 1 to Supplementary Note 10, wherein the sensor is a fixed sensor.

[0159] (Appendix 12) Set the assigned area within the display range where the sensor is installed, Acquire a result of tracking a moving object recognized by the sensor; updating information of a model representing the tracked moving object based on a result of tracking the tracked moving object; When a new moving object is recognized within the assigned area, information of a model representing the moving object that has been tracked within the assigned area is updated based on information of the new moving object; A model representing the moving body is displayed in a virtual space predetermined for the display target range. That is how the computer performs the model display.

[0160] (Appendix 13) At least one processor has Set an allocation area within the display range where the sensor is installed, acquiring a result of tracking a moving object recognized by the sensor; updating information of a model representing the tracked moving object based on a result of tracking the tracked moving object; When a new moving object is recognized within the assigned area, information of a model representing the moving object that has been tracked within the assigned area is updated based on information of the new moving object; displaying a model representing the moving object in a virtual space predetermined for the display target range; Model display program. [Explanation of symbols]

[0161] 10 Model Display System 11 CPU 20 sensors 22 Recognition device 40 servers 100 Model Display Device 102 Setting section 104 Acquisition Department 106 Update section 108 Learning Department 110 Output section

Claims

1. a setting unit (102) that sets an allocation area within a display target range in which a sensor is installed; an acquisition unit (104) that acquires a result of tracking a moving object recognized by the sensor; updating information of a model representing the tracked moving object based on a result of tracking the tracked moving object; an update unit (106) that, when a new moving object is recognized within the assigned area, updates information of a model representing a moving object that has already been tracked within the assigned area based on information of the new moving object; an output unit (110) that displays a model representing the moving body in a virtual space that is predetermined for the display target range; A model display device (100) including:

2. The model display device according to claim 1 , wherein the setting unit sets the assigned area based on a result of movement prediction performed on the recognized moving object.

3. 2. The model display device according to claim 1, wherein the setting unit sets the assigned area based on a group of moving objects made up of the recognized moving objects.

4. 2. The model display device according to claim 1, wherein the setting unit sets the assigned area based on a stationary object determined in advance within the display target range.

5. The model display device according to claim 1 , wherein the setting unit sets the assigned area based on the recognized moving object.

6. 2. The model display device according to claim 1, wherein the setting unit sets the assigned area based on a detection range of the sensor that is determined in advance.

7. The method further includes a learning unit (108) that learns data for identifying the model based on information about a moving object that represents the model, The update unit A model display device as described in claim 1, wherein when a new moving object is recognized within the assigned area, the device identifies which of the models representing moving objects already tracked within the assigned area the new moving object is based on information about the new moving object and data for identifying the model, and updates information about the identified model.

8. 2. A model display device according to claim 1, wherein, when a new moving object is recognized within the assigned area and part of the assigned area overlaps with another assigned area, the update unit identifies whether the new moving object is a model representing a moving object that has already been tracked within the assigned area or a model representing a moving object that has already been tracked in another assigned area, and updates information about the identified model.

9. The model display device according to claim 1 , wherein the update unit updates information of each model representing a tracked moving object based on a result of tracking the tracked moving object or a result of movement prediction for the tracked moving object.

10. The sensor is a plurality of sensors, The model display device according to claim 1 , wherein the setting unit sets the assigned area based on an overlapping area of ​​the detection ranges of the plurality of sensors.

11. 2. A model display device according to claim 1, wherein the sensor is a fixed sensor.

12. Set the assigned area within the display range where the sensor is installed, Acquire a result of tracking a moving object recognized by the sensor; updating information of a model representing the tracked moving object based on a result of tracking the tracked moving object; When a new moving object is recognized within the assigned area, information of a model representing the moving object that has been tracked within the assigned area is updated based on information of the new moving object; A model representing the moving body is displayed in a virtual space predetermined for the display target range. That is how the computer performs the model display.

13. At least one processor (11) Set an allocation area within the display range where the sensor is installed, acquiring a result of tracking a moving object recognized by the sensor; updating information of a model representing the tracked moving object based on a result of tracking the tracked moving object; When a new moving object is recognized within the assigned area, information of a model representing the moving object that has been tracked within the assigned area is updated based on information of the new moving object; displaying a model representing the moving object in a virtual space predetermined for the display target range; Model display program.

Images