Information processing system and three-dimensional data generation method

Abstract

To generate three-dimensional data which is suitable for analyzing a cause of anomaly of a subject, by focusing on the subject, to reproduce a motion of the subject.SOLUTION: An information processing system generates abnormal state three-dimensional data to reproduce a situation where something abnormal has occurred in the subject. The information processing system includes an imaging apparatus, a state estimation apparatus, and an information processing apparatus. The information processing apparatus includes a first generation unit, a second generation unit, and an integration unit. When the state estimation apparatus for estimating a state of the subject detects that something abnormal has occurred in the subject, the first generation unit generates environment three-dimensional data representing surrounding environment of the subject on the basis of an image captured by the imaging apparatus, and the second generation unit generates motion three-dimensional data representing motion of the subject on the basis of the image captured by the imaging apparatus. The integration unit generates abnormal state three-dimensional data by integrating the environment three-dimensional data with the motion three-dimensional data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for generating three-dimensional data showing the situation when an abnormality such as a fall occurs in a subject such as a worker working at a construction site or factory, or a person requiring care living in a care facility. [Background technology]

[0002] When an abnormality occurs, such as an injury or accident, the cause is usually determined by a human being directly observing the situation at the site where the abnormality occurred.

[0003] Even if a surveillance camera is installed, a person must only view the partial video recording captured by the camera and make a judgment.

[0004] However, the cause of an abnormality occurring in a subject is often not simple, but is often caused by a combination of various factors. Therefore, it is often difficult to guess or determine the cause of the abnormality just by looking at the situation at the scene.

[0005] In addition, the situation at the site where the abnormality occurred is often not preserved, making detailed analysis difficult in some cases.

[0006] In addition, depending on where the surveillance cameras are installed, they may not always be able to record the exact moment an accident occurs, and often only a partial record is recorded.

[0007] As a related technique, there is one disclosed in Patent Document 1 that generates a video of an abnormality occurring situation as seen from an arbitrary viewpoint. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2007-226515 A Summary of the Invention [Problem to be solved by the invention]

[0009] The object of the present invention is to solve the problem of difficulty in investigating the cause of an abnormality occurring in a subject. [Means for solving the problem]

[0010] One aspect of the present invention is an information processing system that generates abnormal situation three-dimensional data that reproduces a situation when an abnormality occurs in a subject. The information processing system includes an imaging device, a state estimation device, and an information processing device, and the information processing device includes a first generation unit, a second generation unit, and an integration unit. When an abnormality occurs in the subject is detected by the state estimation device that estimates the state of the subject, the first generation unit generates environment three-dimensional data that shows the environment around the subject based on an image captured by the imaging device, and the second generation unit generates action three-dimensional data that shows the action of the subject based on the image captured by the imaging device. The integration unit integrates the environment three-dimensional data and the action three-dimensional data to generate the abnormal situation three-dimensional data. Effect of the Invention

[0011] According to the present invention, it is possible to generate and reproduce three-dimensional data suitable for subject-centered analysis of the cause of an abnormality occurring in a subject. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram of an information processing system according to an embodiment. [Diagram 2] 3A to 3D are diagrams illustrating an example of a state estimation device, an imaging device, environmental three-dimensional data, action three-dimensional data, and abnormal situation three-dimensional data. [Diagram 3] 4 is a flowchart showing an operation of the state estimation device. [Figure 4] 4 is a flowchart showing the operation of the imaging apparatus. [Diagram 5] 4 is a flowchart showing an operation of the information processing device. [Figure 6]FIG. 1 is a diagram illustrating an example of augmented reality data and virtual reality data. [Figure 7] FIG. 1 illustrates an example of a computer for implementing an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0014] 1 is a configuration diagram of an information processing system according to an embodiment. The information processing system shown in FIG. 1 includes a state estimating device 1, an imaging device 2, an information processing device 3, and a display device 4.

[0015] The state estimation device 1 is attached to the body of a subject, such as a worker working at a construction site or a factory, or a person requiring care living in a care facility. For example, as shown in Fig. 2(a), the state estimation device 1 is mounted on a helmet worn by the subject. The attachment position of the state estimation device 1 is not particularly limited as long as it is possible to properly acquire sensor values, which will be described later.

[0016] 1 includes a sensor 11, such as a gyro sensor or an acceleration sensor, that is attached to a subject and outputs a sensor value that changes according to the subject's movement, and a memory 12 (first memory). The sensor value output from the sensor 11 can be periodically stored in the memory 12.

[0017] Furthermore, the state estimation device 1 estimates the state of the subject based on the sensor value output from the sensor 11. For example, when the sensor value output from the sensor 11 is equal to or less than a threshold value, the state estimation device 1 determines that the subject is in a normal state (such as a state in which the subject is walking). When the sensor value output from the sensor 11 is greater than the threshold value, the state estimation device 1 determines that the subject is in an abnormal state (such as a state in which the subject has fallen). Note that the sensor value output from the sensor 11 is not only used when detecting that an abnormality has occurred in the subject, but may also be used when estimating the motion (posture) of the subject, as described below.

[0018] Furthermore, when the state estimation device 1 detects that an abnormality has occurred in the subject, it transmits an abnormality detection time (detection time) indicating the time when the abnormality occurred in the subject to the imaging device 2 via a wired or wireless connection. For example, when the state estimation device 1 detects that an abnormality has occurred in the subject at 12:00:00 on March 31, 2021, it transmits "12:00:00 on March 31, 2021" as the abnormality detection time to the imaging device 2. Note that, when the state estimation device 1 detects that an abnormality has occurred in the subject, it may be configured to read out from the memory 12 the sensor value output from the sensor 11 in a predetermined period T based on the abnormality detection time, and transmit the read sensor value to the information processing device 3. For example, when the state estimation device 1 detects that an abnormality has occurred in a subject at 12:00:00 on March 31, 2021, the state estimation device 1 reads from the memory 12 a plurality of sensor values ​​output from the sensor 11 during one minute from 11:59:30 on March 31, 2021 to 12:00:30 on March 31, 2021, and transmits each of the read sensor values ​​to the information processing device 3. In this way, the predetermined period T may be any period before or after the abnormality detection time, any period after the abnormality detection time, or any period before the abnormality detection time.

[0019] Here, Fig. 3 is a flowchart showing an example of the operation of the state estimating device 1. It is assumed that the series of operations shown in the flowchart of Fig. 3 is periodically executed by the state estimating device 1.

[0020] First, the state estimation device 1 acquires a sensor value output from the sensor 11 (step S11).

[0021] Next, when the state estimation device 1 determines that the subject is in a normal state based on the sensor value acquired in step S11 (step S12: No), it ends the current state estimation process.

[0022] On the other hand, when the state estimation device 1 detects that an abnormality has occurred in the subject based on the sensor value acquired in step S11 (step S12: Yes), the state estimation device 1 transmits the abnormality detection time to the imaging device 2 (step S13).

[0023] The imaging device 2 shown in FIG. 1 is attached to the subject's body. For example, the imaging device 2 is attached to a helmet worn by the subject, as shown in FIG. 2(a). The mounting position of the imaging device 2 is not particularly limited as long as it is possible to properly capture an image described later. In addition, at least one of the functions of the state estimation device 1, the imaging device 2, and the display device 4 may be configured to be included in the information processing device 3. For example, the state estimation device 1, the imaging device 2, and the information processing device 3 may be integrated to form an information processing device of the embodiment. In this case, the integrated information processing device is attached to the subject's body. In addition, by including the imaging device 2 in the state estimation device 1, the subject does not need to wear multiple devices, and the burden on the subject can be reduced.

[0024] 1 includes a camera 21 and a memory 22 (second memory). The camera 21 may be a monocular camera or a 360-degree camera, and periodically captures images (still images or moving images) in the field of view of the subject or images (still images or moving images) of the subject's surroundings, and stores the captured images in the memory 22.

[0025] Furthermore, when the imaging device 2 receives the abnormality detection time from the state estimation device 1, it reads out from the memory 22 images captured by the camera 21 in a predetermined period T based on the abnormality detection time, and transmits the read images to the information processing device 3. For example, when the imaging device 2 receives "12:00:00 on March 31, 2021" as the abnormality detection time, it reads out from the memory 22 images captured by the camera 21 in one minute from 11:59:30 on March 31, 2021 to 12:00:30 on March 31, 2021 (multiple still images in one minute or moving images in one minute), and transmits the read images to the information processing device 3. Note that the images captured by the camera 21 may be transferred from the imaging device 2 to the information processing device 3 via a recording medium such as a memory card.

[0026] 4 is a flowchart showing an example of the operation of the imaging device 2. It is assumed that the imaging device 2 repeatedly captures images using the camera 21 and stores the images in the memory 22.

[0027] First, when the imaging device 2 determines that it has received the abnormality detection time (step S21: Yes) and that it has stored images (multiple still images or video images) for at least a predetermined period T in the memory 22 (step S22: Yes), it reads out the images for the predetermined period T from the memory 22 (step S23).

[0028] Next, the imaging device 2 transmits the image read out in step S23 to the information processing device 3 (step S24).

[0029] The information processing device 3 shown in FIG. 1 includes a first generating unit 31, a second generating unit 32, an integrating unit 33, and a converting unit .

[0030] The first generating unit 31 generates environmental three-dimensional data indicating the environment around the subject when an abnormality occurs in the subject, based on images from the imaging device 2. For example, when the subject falls, the first generating unit 31 generates environmental three-dimensional data corresponding to the "floor", "pillar", "stairs", and "pipe" that exist around the subject from a series of images, as shown in FIG. 2(b), in an arbitrary three-dimensional space. The generated environmental three-dimensional data can be generated in chronological order in a predetermined period T.

[0031] The second generating unit 32 shown in FIG. 1 generates three-dimensional motion data showing the motion (posture) of the subject when an abnormality occurs in the subject, based on the image from the imaging device 2. For example, when the subject falls, the second generating unit 32 generates three-dimensional motion data showing the skeleton of the subject and the trajectory of the skeleton when the subject falls in any three-dimensional space for each time series in a predetermined period T, as shown in FIG. 2(c). When generating three-dimensional motion data showing the skeleton of the subject and the trajectory of the skeleton, the three-dimensional motion data can be generated more accurately by using the sensor value detected by the state estimation device. For example, by using the sensor value detected by a gyro sensor, it is possible to determine whether the subject simply fell or whether he / she turned over while rotating to avoid something. Furthermore, by adding an image, the movement of the eyes when the subject falls can be detected. If the subject is in a normal state, he / she will try to take a defensive posture, but if the subject is overworked or in poor health, the overall movement will be slow, and the change in the eyes to take a defensive posture will be reduced.

[0032] The integrating unit 33 shown in Fig. 1 integrates the three-dimensional environment data and the three-dimensional action data to generate three-dimensional abnormal situation data. For example, the integrating unit 33 integrates the three-dimensional environment data showing the "floor", "pillar", "stairs", and "pipe" shown in Fig. 2(b) with the three-dimensional action data showing the skeleton and trajectory of the subject shown in Fig. 2(c) to generate three-dimensional abnormal situation data that reproduces the situation up until the subject falls in an arbitrary three-dimensional space in which the "floor", "pillar", "stairs", and "pipe" are present around the subject, as shown in Fig. 2(d).

[0033] The conversion unit 34 shown in FIG. 1 converts the abnormal situation three-dimensional data into augmented reality data for displaying on the display device 4 using AR (Augmented Reality) technology or virtual reality data for displaying on the display device 4 using VR (Virtual Reality) technology.

[0034] FIG. 5 is a flowchart showing an example of the operation of the information processing device 3.

[0035] First, the first generating unit 31 of the information processing device 3 generates environmental three-dimensional data based on an image from the imaging device 2 (step S31). For example, the first generating unit 31 performs SfM (Structure from Motion) analysis and MVS (Multi-View Stereo) analysis on each image corresponding to each time series in a predetermined period T to generate environmental three-dimensional data. That is, first, the first generating unit 31 calculates the position of the imaging device 2 based on feature points in the image as the SfM analysis, and generates low-density point cloud data in an arbitrary three-dimensional space from the position and feature points of the imaging device 2 using a triangulation technique. Next, the first generating unit 31 generates high-density point cloud data from the low-density point cloud data as the MVS analysis, and then generates a textured polygon model (environment three-dimensional data) from the high-density point cloud data. Note that the method of generating environmental three-dimensional data is not limited to the method using SfM analysis and MVS analysis as long as it is possible to generate environmental three-dimensional data from an image. In addition, the first generation unit 31 may be configured to generate environmental three-dimensional data by further using images captured by an imaging device other than the imaging device 2, such as a fixed surveillance camera installed at a construction site, factory, nursing home, etc.

[0036] Next, the second generating unit 32 of the information processing device 3 generates three-dimensional motion data based on the images from the imaging device 2 (step S32). For example, the second generating unit 32 performs posture estimation by deep learning for each image corresponding to each time series in the predetermined period T, and generates three-dimensional motion data.

[0037] That is, the second generating unit 32 estimates the skeleton and skeletal trajectory of the subject based on the learning result of the correlation between the movement of a part of the subject's body in the image, the change in scenery other than the subject in the image, the skeleton of the subject, and the trajectory of the skeleton of the subject, as posture estimation by deep learning. The second generating unit 32 may further estimate the skeleton and skeletal trajectory of the subject by using the sensor value output from the sensor 11 of the state estimation device 1. Alternatively, it can be said that the second generating unit 32 estimates the skeleton and skeletal trajectory of the subject based on the learning result of the correlation between the movement of a part of the subject's body in the image, the change in scenery other than the subject in the image, the change in the sensor value, the skeleton of the subject, and the trajectory of the skeleton of the subject, as posture estimation by deep learning.

[0038] By estimating the subject's skeleton and the trajectory of the skeleton in this way, not only a simple fall but also "the subject was avoiding an obstacle such as a reinforcement before falling," "the subject dropped something he was holding before falling," "the subject fell backwards," "the subject fell while rotating while trying to take a defensive position," etc. can be reflected in the three-dimensional motion data. Note that the method of generating the three-dimensional motion data is not limited to a method using deep learning, as long as it is possible to generate the three-dimensional motion data from an image or a sensor value. Also, the order of steps S31 and S32 may be reversed.

[0039] Next, the integration unit 33 of the information processing device 3 integrates the three-dimensional environment data generated in step S31 and the three-dimensional action data generated in step S32 to generate three-dimensional abnormal situation data (step S33).

[0040] Then, the conversion unit 34 of the information processing device 3 converts the abnormal situation three-dimensional data into augmented reality data or virtual reality data (step S34).

[0041] The display device 4 shown in FIG. 1 is configured with a mobile device such as a smartphone or a tablet terminal, or a personal computer, or a head-mounted display, and displays augmented reality data or virtual reality data. For example, when augmented reality data is displayed on the display of a tablet terminal as the display device 4, the display device 4 displays an image (an image showing the situation at the site after the abnormality occurs) captured by a camera (not shown) mounted on the display device 4 and augmented reality data (an image showing the situation at the site when the abnormality occurs) superimposed on the display as shown in FIG. 6(a). For example, unlike FIG. 6(a), the image showing the situation at the site after the abnormality occurs does not show a "pipe". In addition, the image reproducing the situation at the site when the abnormality occurs shows a "pipe". This makes it possible to reproduce that "the subject did not fall by tripping over the gap between the stairs and the floor, but fell by stepping on a pipe that had fallen on the floor". In addition, it is possible to analyze and verify the cause of the abnormality from multiple angles or in a complex manner, centering on the subject, such as "the fall could have been prevented if there had been a handrail on the stairs at the time the abnormality occurred".

[0042] When the display of a personal computer is used as the display device 4 for displaying virtual reality data, the display device 4 can display BIM (Building Information Modeling) data (an image showing the situation at the site before the abnormality occurs) and virtual reality data (an image showing the situation at the site when the abnormality occurs) superimposed on each other, as shown in FIG. 6(b). In the example shown in FIG. 6(b), the image showing the situation at the site before the abnormality shows a "handrail" and does not show a "reinforcement material". Also, the image showing the situation at the site when the abnormality occurs shows a "reinforcement material". This allows the subject to analyze and verify the cause of the abnormality from multiple angles or in a complex manner, such as "the reinforcement material not included in the BIM data was present at the time of the abnormality, and the reinforcement material may have made the work flow dangerous" or "the fall may have been prevented if the handrail had been present". The display device 4 is not particularly limited as long as it is capable of displaying augmented reality data or virtual reality data.

[0043] According to the information processing device 3 of the embodiment, the environment around the subject and the subject's movements when an abnormality occurs can be reproduced by the abnormal situation three-dimensional data. As a result, when analyzing the cause of an abnormality by checking the abnormal situation three-dimensional data, it is possible to analyze in more detail what kind of abnormality occurred in what environment, what caused it, etc., compared to analyzing the cause of the abnormality by checking the scene after the abnormality occurred or a recorded video at the time of the abnormality. In other words, according to the information processing system of the embodiment, it is possible to generate abnormal situation three-dimensional data suitable for multifaceted analysis of the cause of an abnormality occurring to a subject, with the subject at the center.

[0044] Furthermore, according to the information processing device 3 of the embodiment, the environment around the subject at the time of the occurrence of the abnormality can be reproduced by using the three-dimensional abnormality situation data, so that even if the on-site environment after the occurrence of the abnormality is not preserved, the situation at the time of the abnormality can be analyzed in detail.

[0045] Furthermore, according to the information processing device 3 of the embodiment, the situation at the time of occurrence of an abnormality can be reproduced by using three-dimensional abnormal situation data generated based on an image captured by the imaging device 2 attached to the subject's body. Therefore, compared to the case where the situation at the time of occurrence of an abnormality is confirmed by using an image captured by a fixed camera such as a surveillance camera that is prone to blind spots, it is possible to avoid analyzing the cause of the abnormality by using a partial recorded image and to analyze the situation at the time of occurrence of an abnormality in detail.

[0046] FIG. 7 is a diagram illustrating an example of a computer for implementing the information processing device 3. As shown in FIG.

[0047] 7 includes a processor 71, a memory 72, a storage device 73, a reader 74, a communication interface 75, and an input / output interface 76. The processor 71, the memory 72, the storage device 73, the reader 74, the communication interface 75, and the input / output interface 76 are connected to each other via a bus 77.

[0048] The processor 71 is configured with a single processor, a multiprocessor, a multicore, etc. The processor 71 also reads out a program stored in a storage device 73, and uses a memory 72 to execute a program in which the procedure of the flowchart shown in Fig. 5 is described, thereby providing some or all of the functions of the first generating unit 31, the second generating unit 32, the integrating unit 33, and the converting unit 34.

[0049] The memory 72 is a semiconductor memory or the like, and may include a RAM (Random Access Memory) area and a ROM (Read Only Memory) area.

[0050] The storage device 73 is a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0051] The reader 74 accesses the removable storage medium 78 in accordance with instructions from the processor 71. The removable storage medium 78 is realized by a semiconductor device, a medium where information is input and output by magnetic action, a medium where information is input and output by optical action, or the like. The semiconductor device is a Universal Serial Bus (USB) memory or the like. The medium where information is input and output by magnetic action is a magnetic disk or the like. The medium where information is input and output by optical action is a Compact Disc (CD)-ROM, a Digital Versatile Disc (DVD), a Blu-ray Disc (Blu-ray is a registered trademark), or the like.

[0052] The communication interface 75 communicates with other devices in accordance with instructions from the processor 71. For example, the information processing device 3 collects information from the state estimation device 1 and the imaging device 2 via the communication interface 75.

[0053] The input / output interface 76 is, for example, an interface between an input device and an output device. The input device is a device such as a keyboard, a mouse, or a touch panel that receives instructions from a subject or an administrator who checks the augmented reality data or virtual reality data. The output device is a display device such as a display, or an audio device such as a speaker.

[0054] Each program according to the embodiment is provided to the information processing device 3 in the following form, for example. (1) Pre-installed in storage device 73. (2) Provided by a removable storage medium 78. (3) Provided from a server such as a program server.

[0055] 7 is an example of the hardware configuration of the computer 70 for realizing the information processing device 3, and the embodiment is not limited thereto. For example, a part of the above-mentioned configuration may be deleted, or a new configuration may be added. In another embodiment, for example, a part or all of the functions of the above-mentioned information processing device 3 may be implemented as hardware such as an FPGA, a System-on-a-Chip (SoC), an Application Specific Integrated Circuit (ASIC), or a PLD.

[0056] The above describes the embodiments. However, the embodiments are not limited to the above embodiments, and should be understood to include various modified and alternative forms of the above embodiments. For example, it will be understood that the various embodiments can be embodied by modifying the components without departing from the spirit and scope of the embodiments. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-mentioned embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments. [Explanation of symbols]

[0057] 1 State Estimation Device 2. Imaging device 3. Information processing equipment 4 Display device 11 Sensors 12. Memory 21 Camera 22 Memory 31 First Generation Section 32 Second Generation Section 33 Integration Department 34 Conversion section

Claims

1. The present invention includes an imaging device, a state estimation device, and an information processing device, The information processing device includes a first generation unit, a second generation unit, and an integration unit, When the state estimation device that estimates the state of a subject detects that an abnormality has occurred in the subject, The first generation unit generates three-dimensional environmental data indicating an environment around the subject based on an image captured by the imaging device, The second generation unit generates three-dimensional motion data indicating a motion of the subject based on an image captured by the imaging device, The integration unit integrates the three-dimensional environment data and the three-dimensional action data to generate three-dimensional abnormal situation data. Information processing system.

2. The information processing system according to claim 1 , wherein the state estimating device and the imaging device are worn by the subject.

3. In claim 1 or 2, The state estimator includes a sensor and a first memory. periodically storing a sensor value detected by the sensor in the first memory; When the state estimation device that estimates the state of the subject detects that an abnormality has occurred in the subject, The state estimation device notifies the imaging device of a detection time indicating the time when an abnormality occurred in the subject, and transmits the sensor value for a set period from the detection time to the information processing device.

4. In any one of claims 1 to 3, The imaging device includes a camera and a second memory. periodically storing images captured by the camera in the second memory; When the state estimation device that estimates the state of the subject detects that an abnormality has occurred in the subject, The imaging device transmits to the information processing device the images for a set period from a detection time indicating the time when an abnormality occurred in the subject, which was notified by the state estimation device.

5. In any one of claims 1 to 4, The information processing system includes a display device, The information processing device further includes a conversion unit, The conversion unit converts the abnormal situation three-dimensional data into augmented reality data or virtual reality data, The display device displays the transformed augmented reality data or virtual reality data. Information processing system.

6. A three-dimensional data generation method for generating abnormal situation three-dimensional data that reproduces a situation when an abnormality occurs in a subject in an information processing device, comprising: The information processing device includes: When a state estimation device that estimates a state of the subject detects that an abnormality has occurred in the subject, generating three-dimensional environmental data indicating an environment surrounding the subject based on an image captured by an imaging device; generating three-dimensional motion data indicating a motion of the subject based on an image captured by the imaging device; generating the abnormal situation three-dimensional data by integrating the environmental three-dimensional data and the action three-dimensional data; A three-dimensional data generating method comprising the steps of:

Images