Data analysis device and data analysis method

The data analysis device addresses the challenge of low analysis accuracy by using spatial information and analysis models to group and tag data, resulting in improved analysis results.

WO2024236957A1PCT designated stage expired Publication Date: 2024-11-21HITACHI LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/014535
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-18
Filing Date
2024-04-10
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing data analysis technologies face challenges in achieving accurate analysis due to unreliable measurement data, leading to low accuracy in analysis results, especially in varying measurement environments.

Method used

A data analysis device that includes a processor and memory, which stores and processes spatial information associated with data, groups similar data based on predetermined conditions, and attaches tags to analysis results for improved accuracy, using spatial information and analysis models to enhance data analysis.

Benefits of technology

This approach improves the accuracy of data analysis by effectively grouping and tagging data, leveraging spatial information and analysis models to enhance the reliability of analysis results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014535_21112024_PF_FP_ABST
    Figure JP2024014535_21112024_PF_FP_ABST
Patent Text Reader

Abstract

This data analysis device retains a plurality of analysis target data, analysis results corresponding to each of the plurality of analysis target data, and spatial information indicating a position and / or object related to each of the plurality of analysis target data, generates groups of analysis target data that have similar spatial information on the basis of a prescribed first condition, and for each group, matches the analysis results in the group on the basis of a prescribed rule, assigns a tag indicating the matched analysis results to the analysis target data included in the group, and generates information indicating the plurality of analysis target data, the tag, and the spatial information.
Need to check novelty before this filing date? Find Prior Art

Description

Data analysis device and data analysis method Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-082073, filed on May 18, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to a data analysis device and a data analysis method.

[0003] Japanese Patent Laid-Open Publication No. 2017-167889 (Patent Document 1) describes a background technology in this technical field as follows: "A data management device includes a knowledge graph data generation unit that generates knowledge graph data in which spatial data related to the space within a building in which maintenance equipment is installed, equipment data related to the maintenance equipment, and measurement data measuring the operating status of the maintenance equipment are associated in accordance with ontology conversion rules; a search information input unit that inputs a search keyword including a character string representing an incident; a data search unit that searches for knowledge graph data based on the search keyword; and a search result output unit that associates and outputs spatial data, equipment data, and measurement data that match the search keyword based on the search results of the data search unit" (see Abstract).

[0004] JP 2017-167889 A

[0005] The technology described in Patent Document 1 generates knowledge graph data that associates spatial data, equipment data, and measurement data, but there is a risk that accurate measurement data may not be obtained depending on the measurement environment, etc., resulting in a decrease in the accuracy of analysis of the measurement data. Therefore, one aspect of the present invention improves the accuracy of analysis of data to be analyzed.

[0006] In order to solve the above problem, one aspect of the present invention employs the following configuration: A data analysis device includes a processor and a memory, wherein the memory holds a plurality of pieces of analysis target data, analysis results corresponding to each of the plurality of pieces of analysis target data, and spatial information indicating positions and / or objects associated with each of the plurality of pieces of analysis target data, and the processor generates groups of analysis target data having similar spatial information based on a predetermined first condition, aligns the analysis results within each group based on a predetermined rule, assigns a tag indicating the aligned analysis result to the analysis target data included in the group, and stores knowledge information indicating the plurality of pieces of analysis target data, the tag, and the spatial information in the memory.

[0007] According to one aspect of the present invention, it is possible to improve the accuracy of analysis of analysis target data.

[0008] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0009] FIG. 1 is a block diagram showing an example of a functional configuration of a data analysis device according to a first embodiment; FIG. 2 is a block diagram showing an example of a hardware configuration of a data analysis device according to a first embodiment; FIG. 3 is a diagram showing an example of a data configuration of sensor data according to a first embodiment; FIG. 4 is a diagram showing an example of a data configuration of spatial information assignment method data according to a first embodiment; FIG. 5 is a diagram showing an example of a data configuration of positioning data according to a first embodiment; FIG. 6 is a diagram showing an example of a data configuration of an analysis model DB (DataBase) according to a first embodiment; FIG. 7 is a diagram showing an example of a data configuration of a knowledge DB according to a first embodiment; FIG. 8 is a flowchart showing an example of a data analysis process according to a first embodiment; FIG. 9 is a block diagram showing an example of a functional configuration of a data analysis device according to a second embodiment; FIG. 10 is a diagram showing an example of a screen configuration of a search screen according to a second embodiment;

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted. It should be noted that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention.

[0011] In this embodiment, an example will be described in which the data analysis device 100 analyzes data relating to the environment of a facility such as a large-scale factory or plant, data relating to the equipment of the facility, and data relating to the status or behavior of people in the facility, but the data to be analyzed by the data analysis device 100 is not limited to these.

[0012] 1 is a block diagram showing an example of the functional configuration of a data analysis device 100. The data analysis device 100 includes, for example, a spatial information assignment unit 102 and a data analysis unit 103, which are both functional units. The data analysis device 100 holds various types of information including sensor data 101, spatial information assignment method data 104, positioning data 105, an analysis model DB (DataBase) 106, and a knowledge DB 107.

[0013] The sensor data 101 includes data measured by sensors. The sensors include sensors that measure values ​​related to the environment of a facility, sensors that measure values ​​related to equipment in the facility, and sensors that measure values ​​related to the status or behavior of people in the facility (e.g., workers who use the equipment or workers who maintain it). Specifically, the sensors include, for example, cameras, microphones, smartphones, smartwatches, wearable sensors, electroencephalographs, eye-tracking glasses, thermometers, pressure gauges, and ammeters.

[0014] The types of data stored in the sensor data 101 include, for example, images, videos, voice, audio, motion, acceleration, gyro, geomagnetism, heart rate, pulse rate, electrical skin resistance, sweat rate, body temperature, brain waves, point of gaze, air temperature, humidity, pressure (voltage, oil pressure, air pressure, etc.), and current.

[0015] Furthermore, the sensor data 101 may store data other than data measured by sensors, as long as the data is information about the environment of the facility, information about equipment in the facility, and information about the status or behavior of people in the facility. Specifically, for example, the sensor data 101 may include work procedures, equipment manuals, work reports, memos, and the like (all of which are documents). Both the data measured by sensors and data other than data measured by sensors, such as work procedures, included in the sensor data 101 are examples of data to be analyzed.

[0016] The spatial information assigning unit 102 assigns spatial information to the sensor data 101 by referring to the spatial information assigning method data 104 and the positioning data 105. The spatial information includes location information (which may be two-dimensional planar coordinate values, three-dimensional coordinate values ​​including height, the name of a room or section within a facility, or a mathematical formula (e.g., an inequality) indicating an area such as a room or section) where the data included in the sensor data 101 was collected, and information on objects related to the data (hereinafter also referred to as related matters). Objects that are the measurement targets of the data and objects described in the data are both examples of related matters of the data.

[0017] For example, an image included in the sensor data 101 may be assigned spatial information such as location information of a point (latitude X, longitude Y) (i.e., indicating that the image was acquired at (latitude X, longitude Y)), or may be assigned information about a related object such as transformer A (i.e., indicating that the image is an image of transformer A).

[0018] As the location information in the spatial information, for example, if outdoors, location information identified by GPS (Global Positioning System) can be used, and if indoors, location information identified by PDR (Pedestrian Dead Reckoning), SLAM (Simultaneous Localization and Mapping), infrared beacons, surveillance cameras, RFID, or a combination of these can be used.

[0019] As information on related objects in spatial information, for example, the name or ID of the related object may be specified by manual user input, or an object automatically identified by image recognition technology such as general object detection may be specified.

[0020] The spatial information adding method data 104 indicates a method by which the spatial information adding unit 102 adds spatial information to the sensor data 101. Furthermore, for example, the spatial information adding method data 104 indicates a positioning device that measures the position to be added to the sensor data 101, a model that detects related matters to be added to the sensor data 101, etc. The positioning data 105 holds positioning information obtained by a positioning device that measures the position as spatial information to be added to the sensor data 101.

[0021] The data analysis unit 103 analyzes the sensor data 101 using the analysis model indicated by the analysis model DB 106 , updates the analysis result based on the spatial information, and stores the updated analysis result in the knowledge DB 107 .

[0022] The analysis model DB 106 holds an analysis model that outputs an analysis result when data included in the sensor data 101 is input. The knowledge DB 107 holds the analysis results obtained by the data analysis unit 103.

[0023] 2 is a block diagram showing an example of the hardware configuration of the data analysis device 100. The data analysis device 100 is configured by a computer having, for example, a CPU (Central Processing Unit) 110, a memory 120, an auxiliary storage device 130, an input device 140, an output device 150, and a communication device 160.

[0024] The CPU 110 is an example of a processor and executes programs stored in the memory 120. The memory 120 includes a non-volatile storage element, Read Only Memory (ROM), and a volatile storage element, Random Access Memory (RAM). The ROM stores unchanging programs (e.g., a Basic Input / Output System (BIOS)). The RAM is a high-speed, volatile storage element such as Dynamic Random Access Memory (DRAM), and temporarily stores programs executed by the CPU 110 and data used when the programs are executed.

[0025] The auxiliary storage device 130 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores programs to be executed by the CPU 110 and data to be used when the programs are executed. In other words, the programs are read from the auxiliary storage device 130, loaded into the memory 120, and executed by the CPU 110.

[0026] The input device 140 is a device that receives input from an operator, such as a keyboard, a mouse, a VR (Virtual Reality) controller, or a game controller. The output device 150 is a device that outputs the results of program execution in a format that can be viewed by the operator, such as a display device or a printer.

[0027] The communication device 160 is a network interface device that controls communication with other devices in accordance with a predetermined protocol, and includes a serial interface such as a USB (Universal Serial Bus).

[0028] Some or all of the programs executed by the CPU 110 may be provided to the data analysis device 100 from a removable medium (such as a CD-ROM or flash memory) that is a non-transitory storage medium, or from an external computer equipped with a non-transitory storage device via a network, and may be stored in the non-volatile auxiliary storage device 130 that is a non-transitory storage medium. For this reason, the data analysis device 100 may preferably have an interface for reading data from removable media.

[0029] The data analysis device 100 is a computer system configured on a single physical computer or on multiple logically or physically configured computers, and may operate in separate threads on the same computer, or may operate on a virtual computer constructed on multiple physical computer resources.

[0030] The CPU 110 includes each functional unit included in the data analysis device 100. For example, the CPU 110 functions as a spatial information assignment unit 102 by operating in accordance with a spatial information assignment program loaded into the memory 120, and functions as a data analysis unit 103 by operating in accordance with a data analysis program loaded into the memory 120. The relationship between the programs and the functional units is similar in other examples described below.

[0031] Note that some or all of the functions of the functional units included in the CPU 110 may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0032] The various types of information held by the data analysis device 100 are stored, for example, in the auxiliary storage device 130. Note that some or all of the information stored in the auxiliary storage device 130 may be stored in the memory 120 or in a database connected to the data analysis device 100.

[0033] In this embodiment, the information used by the data analysis device 100 does not depend on the data structure and may be expressed in any data structure. In this embodiment, the information is expressed in a table format, but the information can be stored in a data structure appropriately selected from, for example, a list, a database, or a queue.

[0034] 3 is a diagram showing an example of the data configuration of the sensor data 101. The sensor data 101 includes, for example, a sensor ID column 1011, a data type column 1012, a data column 1013, and a time column 1014.

[0035] The sensor ID column 1011 holds a sensor ID for identifying a sensor. The data type column 1012 holds information indicating the type of data measured by the sensor. The data column 1013 holds data measured by the sensor. The time column 1014 holds the time when the data was measured by the sensor. The time column 1014 may also hold a time period.

[0036] As described above, data other than data measured by a sensor may be stored in the sensor data 101. For example, a record of the data type "document" in the sensor data 101 in Fig. 3 is not data measured by a sensor, but is registered document data such as a work procedure manual.

[0037] A sensor ID is also assigned to data that is not measured by a sensor, such as document data. As the time corresponding to data that is not measured by a sensor, such as document data, the time at which the data is registered may be registered in the sensor data 101, or a time period indicating the validity period of the data (for example, the validity period of a work procedure manual) may be manually registered.

[0038] 4 is a diagram showing an example of the data configuration of the spatial information adding method data 104. The spatial information adding method data 104 holds, for example, a sensor ID column 1041, a position column 1042, and a related object column 1043.

[0039] The sensor ID column 1041 holds a sensor ID. The position column 1042 indicates a method for assigning a position as spatial information and notes on the method for assigning the position. The method for assigning a position includes manual (i.e., input by a user via the input device 140) and the aforementioned GPS and PDR. The notes in the position column 1042 store, for example, a positioning device ID (information identifying a positioning device) when a position is assigned by a method other than manual, such as GPS or PDR. If the method for assigning a position is manual, the notes in the position column 1042 may be a null value.

[0040] The related thing column 1043 indicates the method of assigning the related thing as spatial information and notes on the method of assigning the related thing. The method of assigning the related thing includes manual (i.e., input by the user via the input device 140) and the object detection described above. The notes in the related thing column 1043 store, for example, an object detection model in cases where the position is assigned by a method other than manual, such as object detection. If the method of assigning the related thing is manual, the notes in the related thing column 1043 may be a null value. Note that it is sufficient that at least one of the assignment methods in the position column 1042 and the related thing column 1043 is registered for each sensor ID in the spatial information assignment method data 104.

[0041] 5 is a diagram showing an example of the data configuration of the positioning data 105. The positioning data 105 holds, for example, a positioning device ID 1051, a position column 1052, and a time column 1053. The positioning device ID column 1051 holds a positioning device ID. The position column 1052 holds information indicating the position measured by the positioning device. The time column 1053 holds information indicating the time when the position was measured by the positioning device.

[0042] 6 is a diagram showing an example of the data configuration of the analytical model DB 106. In the analytical model DB 106, analytical models corresponding to data types and analytical models corresponding to data types and spatial information (positions and / or related matters) are defined. The analytical model DB 106 includes, for example, a model ID column 1061, a data type column 1062, a position column 1063, a related matter column 1064, and an analytical model column 1065.

[0043] The model ID column 1061 holds a model ID that identifies an analytical model. The data type column 1062 holds information indicating a data type. The position column 1063 holds information indicating a position. The related matter column 1064 holds information indicating a related matter.

[0044] The analytical model column 1065 holds information indicating trained analytical models. Specifically, for example, the analytical model column 1065 stores a method for extracting feature quantities as explanatory variables from the data in the corresponding data type column 1062, and an analytical model that outputs information indicating an analysis result when the feature quantities are input as explanatory variables. Note that the trained analytical models stored in the analytical model DB 106 are, for example, models that have been trained in advance by machine learning using training data of the corresponding data type.

[0045] 7 is a diagram showing an example of the data configuration of the knowledge DB 107. The knowledge DB 107 includes, for example, a sensor ID column 1071, a data type column 1072, a data column 1073, a tag column 1074, a feature column 1075, a position column 1076, a related object column 1077, and a time column 1078.

[0046] The sensor ID column 1071 holds a sensor ID. The data type column 1072 holds information indicating a data type. The data column 1073 holds the sensor ID and the data type.

[0047] The tag field 1074 holds tags based on the analysis results output from the analysis model. The names of objects in an image are examples of tags assigned to image data, keywords contained in audio data are examples of tags assigned to audio data, and document types are examples of tags assigned to document data. Tags do not need to be highly readable; for example, they may be features of high-dimensional vectors obtained by feature extraction processing.

[0048] The feature amount column 1075 holds feature amounts extracted by the data analysis unit 103. The position column 1076 holds information indicating the position assigned by the spatial information assignment unit 102. The related object column 1077 holds information indicating the related object assigned by the spatial information assignment unit 102. The time column 1078 holds information indicating the time corresponding to the data in the data column 1073.

[0049] 8 is a flowchart showing an example of the data analysis process. For example, before the process of FIG. 8 starts, it is assumed that information in the sensor ID column 1011 and the data type column 1012 of the sensor data 101, the spatial information assignment method data 104, and the positioning device ID column 1051 of the positioning data 105 are registered in advance.

[0050] The spatial information assigning unit 102 acquires a sensor ID, data, and time (for example, a timestamp of a sensor value) transmitted from a sensor connected to the data analysis device 100 or an external device connected to the data analysis device 100, or input to the input device 140, and stores the acquired data and time in the sensor data 101 (S801). Note that the spatial information assigning unit 102 may store the time at which it acquired data from the sensor, the external device, or the input device (for example, the time at which sensor information was uploaded) in the time column 1014.

[0051] The spatial information assigning unit 102 assigns spatial information to the sensor data 101 with reference to the spatial information assigning method data 104 and the positioning data 105 (S802). Specifically, for example, the spatial information assigning unit 102 acquires a sensor ID for each record of the sensor data 101, identifies an assignment method and notes for the position and related matter corresponding to the sensor ID from the spatial information assigning method data 104, and assigns information on the position and / or related matter to the record in accordance with the identified assignment method and notes.

[0052] In the examples of Figures 3 and 4, the spatial information assignment unit 102 assigns information on the position and related matters that are "manually" input via the input device 140 to a record in the sensor data 101 whose sensor ID is "001".

[0053] Furthermore, for example, for a record having a sensor ID of "002" in the sensor data 101, the spatial information assigning unit 102 acquires and assigns information on a position measured by a positioning device having a positioning device ID of "AAA" from the positioning data 105. In this case, it is desirable that the spatial information assigning unit 102 assigns information on a position measured by a positioning device having a positioning device ID of "AAA" indicated in the positioning data 105, the value of which in the time column 1053 is closest to the time column 1014 having the sensor ID of "002".

[0054] Furthermore, for example, for a record in the sensor data 101 whose sensor ID is "003", the spatial information assignment unit 102 assigns position information that is "manually" input via the input device 140, and assigns information on related things (for example, the name of the detected related thing) that is output by "object detection" using the data of the record and "object recognition model X".

[0055] The data analysis unit 103 selects, for each record of the sensor data 101 to which spatial information has been assigned in step S102, an analysis model corresponding to the data type and the assigned space type from the analysis model DB 106 (S103). Note that, in the example of the analysis model DB 106 in Fig. 6, an analysis model corresponding to a combination of the data type and the space type is defined, but an analysis model corresponding to only the data type may be defined. In this case, for each record of the sensor data 101 to which spatial information has been assigned in step S102, the data analysis unit 103 selects, from the analysis model DB 106, an analysis model corresponding to the data type.

[0056] For each record of the sensor data 101 to which spatial information was assigned in step S102, the data analysis unit 103 extracts features as explanatory variables indicated by the analysis model selected in step S103 from the data indicated by the record, and inputs the extracted features into the analysis model to obtain the analysis results (S104).

[0057] The data analysis unit 103 groups the sensor data 101 having similar spatial information assigned in step S102 based on predetermined conditions (S105). Specifically, for example, the data analysis unit 103 groups records in the sensor data 101 having similar assigned positions (for example, records whose assigned position coordinates are within a circle of a predetermined radius, or records whose assigned positions indicate the same room or section), or groups records whose assigned related matters are the same or similar (for example, which related matters are similar are determined in advance).

[0058] The data analysis unit 103 may group only records in the sensor data 101 whose times are close based on a predetermined condition (for example, records whose time difference is within a predetermined value) based on spatial information. In this way, the data analysis unit 103 can group data more accurately by utilizing time information, thereby improving the accuracy of tagging, which will be described later. Furthermore, records that were not grouped in step S105 are treated as a group consisting of only a single record in step S106, which will be described later.

[0059] The data analysis unit 103 generates a tag for each group created in step S105 based on the analysis result obtained in step S104, and assigns the tag to each record included in the group (S106). Specifically, for example, the data analysis unit 103 assigns the analysis result obtained in step S104 as a tag to a group consisting of only a single record.

[0060] The data analysis unit 103 aligns the analysis results within a group containing multiple records based on a predetermined rule and assigns a tag indicating the aligned analysis result. An example of the predetermined rule for assigning tags will be described below. It should be noted that the rule for assigning tags is predefined in accordance with the data type of the records included in the group, the combination of the data type of the records included in the group and the analysis model used to analyze the records, the combination of the data type of the records included in the group and spatial information, etc. For a group for which no corresponding rule exists, the data analysis unit 103 assigns the analysis result obtained in step S104 as a tag to each of the multiple records included in the group.

[0061] For example, there is a rule that the data analysis unit 103 assigns tags to a group containing multiple records based on predetermined statistics for the analysis results corresponding to each of the multiple records obtained in step S104.

[0062] An example of the first rule based on a predetermined statistical quantity will be described. For example, suppose a first group contains three records, and A is the analysis result obtained for the first of the three records, B is the analysis result obtained for the second record, and A is the analysis result obtained for the third record. In this case, A, which is the analysis result that appears most frequently in the first group, is assigned as a tag to each of the three records.

[0063] An example of the second rule based on a predetermined statistical quantity will be described. For example, suppose that the second group includes three records, all of which are images (hereinafter referred to as image 1, image 2, and image 3), and all three images are input to an object recognition model, which is an analytical model. Furthermore, suppose that an image is input to the object recognition model, and the probability that the object included in the image is object A, the probability that it is object B, and the probability that it is object C are output as analysis results.

[0064] Furthermore, suppose that the analysis results for image 1 by the object recognition model are output as follows: probability 0.4 that the object contained in image 1 is object A, probability 0.5 that it is object B, and probability 0.1 that it is object C; the analysis results for image 2 by the object recognition model are output as follows: probability 0.6 that the object contained in image 2 is object A, probability 0.3 that it is object B, and probability 0.1 that it is object C; and the analysis results for image 3 by the object recognition model are output as follows: probability 0.5 that the object contained in image 3 is object A, probability 0.4 that it is object B, and probability 0.1 that it is object C.

[0065] In this case, the data analysis unit 103 averages the probabilities indicated by the analysis results for image 1, image 2, and image 3, for example. That is, the data analysis unit 103 obtains an average probability of 0.5 for the object being object A, an average probability of 0.4 for the object being object B, and an average probability of 0.1 for the object being object C. The data analysis unit 103 assigns object B, which has the highest average probability, as a tag to each of the three records in the second group. Furthermore, the data analysis unit 103 may assign an average probability of 0.5 for the object being object A, an average probability of 0.4 for the object being object B, and an average probability of 0.1 for the object being object C as tags to each of the three records in the second group.

[0066] In the second example, although the object contained in image 1 is most likely to be object B, when information on images 2 and 3, which are images of the same or a similar space as image 1, is also taken into account, the analysis results can be corrected to indicate that the object contained in image 1 is object A, and a tag can then be assigned.

[0067] An example of the third rule will be described. For example, suppose the third group includes two records, and the data type of the first record of the two records is an image and the related object is device A, and the data type of the second record is audio and the related object is device A. Furthermore, suppose the image in the first record is input to an object recognition model, which is an analysis model, and the probability that an abnormality has occurred in device A and the probability that an abnormality has not occurred are output as analysis results. Furthermore, suppose the audio in the second record is input to an abnormal sound detection model, which is an analysis model, and the probability that an abnormality has occurred in device A and the probability that an abnormality has not occurred are output as analysis results.

[0068] In this case, the data analysis unit 103 assigns, for example, the analysis result corresponding to the higher of the average value of the probability that an abnormality has occurred in device A and the average value of the probability that an abnormality has not occurred in the analysis results of the first record and the second record (i.e., either an abnormality has occurred or no abnormality has occurred) as a tag to the records of the third group.

[0069] For example, if the space corresponding to the third group becomes dark or if the camera capturing the image has a blind spot, the analysis accuracy for the image will decrease but the analysis accuracy for the audio will not decrease, and if the amount of sound output from sources other than device A in the space increases, the analysis accuracy for the audio will decrease but the analysis accuracy for the image will not decrease. In this way, in the third example, by using different types of data such as images and audio (i.e., multimodal data), tags can be assigned more accurately based on complementary information.

[0070] In addition to the rule based on a predetermined statistical quantity, there is also a rule that the data analysis unit 103 assigns a tag to a group containing multiple records after prioritizing the analysis result corresponding to one of the multiple records acquired in step S104 and aligning the analysis results.

[0071] An example of such a rule, the fourth rule, will now be described. Assume that the fourth group includes two records, the data type of the first record being (the worker's) acceleration, and the data type of the second record being a document (work procedure manual). Furthermore, assume that the acceleration in the first record is input to a behavior recognition model, which is an analysis model, and the probability that the work being performed by the worker is work A, work B, or work C is output as an analysis result. Furthermore, assume that the document in the second record is input to a document analysis model, which is an analysis model, and the work (the name of the work) described in the document is output as an analysis result.

[0072] Furthermore, suppose that the analysis results for the acceleration of the first record output a probability of 0.4 that the work being performed by the worker is work A, a probability of 0.5 that it is work B, and a probability of 0.1 that it is work C, and that the analysis results for the document of the second record output work A and work C.

[0073] In this case, the analysis result for the acceleration of the first record indicates a high probability that task B is being performed, but the data analysis unit 103 prioritizes the analysis result for the document of the second record and excludes task B from the analysis result for the first record. That is, for example, the data analysis unit 103 compares the probability that the task being performed by the worker is task A with the probability that it is task C, and assigns task A, which has the higher probability, as a tag to the first record. Furthermore, the data analysis unit 103 may assign task C, which was included in the analysis result, as a tag to the second record in addition to task A.

[0074] In this way, also in the fourth example, by using different types of data (i.e., multimodal data) such as acceleration and documents, tags can be assigned more accurately based on complementary information.

[0075] Next, the data analysis unit 103 registers various pieces of information in the knowledge DB 107 (S107), and ends the data analysis process of FIG.

[0076] In this embodiment, an example has been described in which the data analysis device 100 automatically assigns tags to each group in step S106. However, for example, the data analysis device 100 may assign tags by displaying the grouped sensor data 101 to which spatial information has been assigned and the analysis results corresponding to the records of each group on the output device 150, and allowing the user to input tags into the input device 140 after checking the display.

[0077] In this embodiment, an example has been described in which the sensor data 101 and the positioning data 105 are assigned with time stamps, but these data do not necessarily have to be assigned with time stamps. In this case, time stamps are not taken into consideration when grouping the data in step S105.

[0078] Furthermore, the data analysis unit 103 may re-learn the network parameters of the neural network using contrastive learning or the like, so that, among the data analyzed using the same neural network (an example of an analytical model), the similarity between the feature quantities extracted from the data of records classified into the same group in step S105 is high, and the similarity between the feature quantities extracted from the data of records classified into different groups is low.

[0079] The data analysis unit 103 may generate a group ID for identifying the group generated in step S105 and store the group ID in the knowledge DB 107. This allows the user to easily understand which data has been grouped based on the spatial information.

[0080] As described above, the data analysis device 100 of this embodiment assigns spatial information to the sensor data 101, analyzes each piece of data included in the sensor data 101, groups the data included in the sensor data 101 based on the spatial information, and assigns a tag indicating the analysis result for the data to each group, thereby improving the accuracy of tagging. The data analysis device 100 can further improve the accuracy of tagging by additionally using time information in grouping the data. Furthermore, by grouping multimodal data and assigning tags, the data analysis device 100 can assign tags more accurately based on complementary information.

[0081] Hereinafter, the description of the contents common to the first embodiment will be omitted as appropriate, and the differences from the first embodiment will be mainly described.

[0082] 9 is a block diagram showing an example of the functional configuration of the data analysis device 100. The data analysis device 100 of this embodiment further includes a search unit 201, a screen data generation unit 202, an area calculation unit 204, and an intention estimation unit 205, which are all functional units. The various information held by the data analysis device 100 further includes 3D model data 203.

[0083] The search unit 201 searches the knowledge DB 107 according to information input to the input device 140. The screen data generation unit 202 generates screen data for a search screen, a search result display screen, and a 3D model, and displays them on the output device 150.

[0084] The 3D model data 203 is a 3D model that represents the interior of a facility that is the target of data collection. If the facility that is the target of data collection is a factory, a CAD (Computer Aided Design) model of the factory is an example of the 3D model data 203. If the facility that is the target of data collection is a construction site, a 3D model created by point cloud measurement of the site is an example of the 3D model data 203.

[0085] The 3D model data 203 may be data in any of various forms, such as CAD data converted into 3D CG (Computer Graphics), point cloud data, or photogrammetry. Instead of the 3D model data 203, 2D data, such as an image acquired by a 360-degree camera, may be used.

[0086] The 3D model data 203 indicates spatial information consistent with the spatial information assigned by the spatial information assigning unit 102. Specifically, for example, the 3D model data 203 has the same coordinate system as the position of the positioning data 105 and a position that can be input by the user, and includes names and IDs of related matters that can be automatically assigned by the spatial information assigning unit 102 and related matters that can be input by the user. This allows the screen data generating unit 202 to display the position of the data assigned with spatial information on the 3D model, the related matter to which the data is linked, and so on.

[0087] The area calculation unit 204 calculates a display range of the 3D model data 203 to be displayed on the output device 150 in accordance with a command from the user input to the input device 140, and outputs the calculation result to the screen data generation unit 202. The area calculation unit 204 may also calculate the display range by performing voice recognition and gesture recognition based on voice input, gesture input, or the like to the input device 140. The area calculation unit 204 may also estimate the user's head motion and gaze point position, and calculate the display range based on the estimated information.

[0088] When a search condition using an abstract expression is specified, the intention estimation unit 205 performs a search that reflects the user's intention estimated from the expression.

[0089] 10 is a diagram showing an example of the screen configuration of a search screen. The search screen 1000 includes, for example, a search keyword input area 1010 and a search condition setting area 1020. When the search unit 201 receives a search request in which a search keyword is input in the search keyword input area 1010 and / or a search condition is set in the search condition setting area 1020, the search unit 201 searches the knowledge DB 107 according to the search keyword and / or the search condition.

[0090] The search condition setting area 1020 includes, for example, a search type setting area 1021 , a time setting area 1022 , a data type setting area 1023 , a location setting area 1024 , a related matter setting area 1025 , and a free input area 1026 .

[0091] The search type setting area 1021 is an area for setting a search range based on, for example, a search keyword. When "content" is selected in the search type setting area 1021, data containing the search keyword in the content (e.g., sentences in a document file, file names of each file, etc.) is searched for in the knowledge DB 107.

[0092] When "tag" is selected in the search type setting area 1021, for example, data whose tags contain the search keyword is searched for in the knowledge DB 107. When "full text" is selected in the search type setting area 1021, for example, data whose contents and / or tags contain the search keyword is searched for in the knowledge DB 107.

[0093] The time setting area 1022 is an area for setting search conditions relating to the time of data. The data type setting area 1023 is an area for setting search conditions relating to the data type.

[0094] The position setting area 1024 is an area for setting search conditions related to the position of data. In the example of Fig. 10, a center and a radius can be set in the position setting area 1024, and data indicating a position within the circle defined by the set center and radius can be searched for in the knowledge DB 107. Note that the position setting area 1024 may be set in an area of ​​a shape other than a circle, and the name of a room or section, etc. may also be set.

[0095] The related matter setting area 1025 is an area for setting search conditions for related matters of data. In the example of Fig. 10, the related matter setting area 1025 displays boxes in which related matters included in the knowledge DB 107 can be selected.

[0096] In the above example, search conditions for spatial information including positions and related objects are set in the position setting area 1024 and the related object setting area 1025, but it may also be possible to set search conditions for spatial information, for example, by having the area calculation unit 204 calculate a 3D model of the area specified by the user, and having the screen data generation unit 202 display the 3D model of the area on the output device 150, and having the user select positions and related objects on the 3D model displayed on the output device 150 through input.

[0097] The free input area 1026 is an area where the user can freely input text indicating search conditions. The intention estimation unit 205 performs a search that reflects the user's intention in the text input in the free input area 1026. For example, when the text "image data taken under a cooling device" is input in the free input area 1026, the intention estimation unit 205 searches the knowledge DB 107 for image data whose (x, y) coordinates are within a predetermined distance from the (x, y) coordinates of the cooling device and whose z coordinate is smaller than the z coordinate of the cooling device.

[0098] Furthermore, for example, when the text "report created after task A" is entered in the free input field 1026, the intention estimation unit 205 searches the knowledge DB 107 for only document data that is later in time than the data corresponding to the tag "task A."

[0099] The intention estimation unit 205 may estimate the intention from the text based on rules created in advance, or may estimate the intention based on a language processing algorithm such as Question Answering.

[0100] Furthermore, when a search result by the intention estimation unit 205 is selected by the user on a search result display screen described later, the search result may be regarded as the data desired by the user (correct data), and information regarding the pair of the text (query) entered in the free input field 1026 and the correct data may be accumulated, and data that should be returned in response to a specific query may be learned using machine learning technology.

[0101] Furthermore, the intention estimation unit 205 may output a question asking about the intention when it determines that the intention input in the free input area 1026 is unclear. For example, when the text "image data near the cooling device" is input in the free input area 1026, the intention estimation unit 205 may output a question asking about the intention such as "Does "near the cooling device" mean within a 5-meter radius of the cooling device?"

[0102] 11 is a diagram showing an example of the screen configuration of a search result display screen. The search result display screen 1100 includes, for example, a 3D model display area 1110 and a search result display area 1120. The 3D model display area 1110 displays a 3D model (extracted from the 3D model data 203) of an area including positions and related matters corresponding to data included in the search results. The positions and related matters corresponding to the data included in the search results are also shown on the 3D model displayed in the 3D model display area 1110 (the positions and related matters are indicated by the numbers "1" and "2" in the 3D model display area 1110 in FIG. 11).

[0103] The search result display area 1120 displays the search results based on the search keywords and search conditions specified on the search screen 1000. In the example of Fig. 11, the search result display area 1120 displays data, data type, time, tag, location, and related items as search results, but any item included in the knowledge DB 1107 may also be displayed. Also, for example, a group ID may be displayed in the search result display area 1120.

[0104] Additionally, items extracted from the knowledge DB 1107 such as data, data type, time, tag, location, and related items may also be displayed superimposed on the 3D model in the 3D model display area 1110 .

[0105] When data displayed in the search result display area 1120 is selected, the selected data is displayed.

[0106] Furthermore, for example, the data may be arranged in chronological order in the search result display area 1120, or the data may be sorted by data type and then arranged in chronological order. For example, the search unit 201 may arrange the image data obtained as the search results in chronological order and apply an image analysis method such as change detection or change captioning, thereby detecting or describing major changes that have occurred at points specified as search conditions and presenting them to the user.

[0107] Furthermore, for example, data may be sorted by related matters and then arranged in chronological order in the search result display area 1120. For example, the search unit 201 may sort document data obtained as search results by related matters and then arrange them in chronological order, and apply a document analysis method such as timeline summarization, thereby making it possible to present to the user changes that have occurred in related matters specified as search conditions.

[0108] In addition to the search results based on the search keywords and search conditions specified on the search screen 1000, the search unit 201 may also display on the search result display screen 1100 data included in the search results and data that was determined to belong to the same group and / or has the same tag in step S105.

[0109] As a result, for example, if a user wants to check from image data whether an accident has occurred in a cooling device, but the image was taken in a dark space and the user has difficulty understanding the situation of the accident by looking at the image data, audio data belonging to the same group as the image data can also be displayed, allowing the user to understand the situation of the accident from the audio data, etc., it is possible to present information that is useful to the user but that the user is not aware of. In particular, by displaying data that is different from the data type specified by the user but that is determined to belong to the same group and / or has the same tag on the search result display screen 1100, it is possible to present complementary multimodal data to the user.

[0110] Furthermore, for example, the search unit 201 may display, on the search result display screen 1100, data that would be included in the search result if the range of the position set in the position setting area 1024 is expanded by a predetermined distance, or data that would be included in the search result if the range of the time set in the time setting area 1022 is expanded by a predetermined time (for example, by moving the start time forward by a predetermined time and / or moving the end time back by a predetermined time), in addition to the search results based on the search keywords and search conditions specified on the search screen 1000. This makes it possible to present information that is useful to the user but that the user is not aware of.

[0111] Note that multiple data analysis devices 100 may be connected via a network such as the Internet, and in this case, the multiple data analysis devices 100 may share and display search results with each other. Specifically, for example, the screen data generation unit 202 of each of the multiple data analysis devices 100 displays, on its own search result display screen 1100, search results from other data analysis devices 100 that are displaying search results for which the areas of 3D models displayed in the 3D model display area 1110 are within a predetermined distance (the spatial information (areas including positions and / or related objects) of the data displayed in the search result display area 1120 are close based on predetermined conditions).

[0112] This allows users of the multiple data analysis devices 100 to share what information (tags, related things, etc.) of a nearby area on the 3D model other users are focusing on in that area. Furthermore, if discussions can be held between users of the multiple data analysis devices 100 via chat, phone calls, etc., the users can share search results and learn about the information that each user values, which in turn facilitates consensus building in discussions.

[0113] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0114] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0115] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A data analysis device comprising a processor and a memory, wherein the memory holds a plurality of pieces of analysis target data, analysis results corresponding to each of the plurality of pieces of analysis target data, and spatial information indicating positions and / or objects related to each of the plurality of pieces of analysis target data, wherein the processor generates groups of analysis target data having similar spatial information based on a predetermined first condition, aligns the analysis results within each group based on a predetermined rule, and assigns a tag indicating the aligned analysis result to the analysis target data included in the group, and stores knowledge information indicating the plurality of analysis target data, the tag, and the spatial information in the memory.

2. A data analysis device as claimed in claim 1, wherein the processor aligns the analysis results within a first group, which is included in the generated group and consists of analysis target data of the same data type, based on a predetermined statistical quantity for the analysis results corresponding to the analysis target data.

3. A data analysis device as described in claim 2, wherein the first group consists of analysis target data whose data type is an image, and the analysis results corresponding to the images indicate the probability that each object is included in the image, and the processor calculates, as the predetermined statistical quantity, an average value of the probability that each object is included in the images included in the first group, determines the objects included in the images included in the first group based on the calculated average value, and assigns a tag indicating the determined object to the analysis target data included in the first group.

4. A data analysis device as described in claim 1, wherein the processor is adapted to align the analysis results within a second group, which is included in the generated group and contains analysis target data of a predetermined data type included in the plurality of data types, by giving priority to analysis results corresponding to analysis target data of a predetermined data type included in the plurality of data types.

5. A data analysis device as described in claim 4, wherein the second group includes analysis target data whose data type is a document and analysis target data whose data type is acceleration, the predetermined data type is a document, the analysis result corresponding to the acceleration indicates the probability that each task was performed by the worker whose acceleration was measured, and the analysis result corresponding to the document indicates the task described in the document, and the processor assigns a tag to the analysis target data included in the second group that indicates the task among the tasks described in the document that has the highest probability in the analysis result corresponding to the acceleration.

6. A data analysis device as described in claim 1, wherein the memory holds time information corresponding to each of the plurality of analysis target data, and the processor generates groups of analysis target data in which the spatial information is similar based on the predetermined first condition and the time information is close based on a predetermined second condition.

7. A data analysis device as claimed in claim 1, connected to an input device and a display device, wherein the processor receives a search request for searching for analysis target data contained in the knowledge information via an input to the input device, and displays, as search results, analysis target data that matches the search request and analysis target data that belongs to the same group as the analysis target data on the display device.

8. A data analysis device as described in claim 7, further comprising: a 3D model of a location including the position and / or object indicated by the spatial information; the processor refers to the 3D model to calculate an area including the position and / or object corresponding to the analysis target data included in the search results; displays a 3D model of the calculated area on the display device; and displays the search results on the 3D model of the calculated area.

9. A data analysis device as described in claim 7, which is connected to another data analysis device, and the processor displays on the display device search results from the other data analysis device, where the search results include an area containing a position and / or object related to the analysis target data included in the search results, and the area is determined to be close to an area containing a position and / or object related to the analysis target data included in the search results from the processor based on a predetermined third condition.

10. A data analysis method using a data analysis device, wherein the data analysis device has a processor and a memory, and the memory holds: a plurality of pieces of analysis target data, analysis results corresponding to each of the plurality of pieces of analysis target data, and spatial information indicating positions and / or objects related to each of the plurality of pieces of analysis target data, and the data analysis method comprises the steps of: generating groups of analysis target data having similar spatial information based on a predetermined first condition; for each group, the processor aligns the analysis results within the group based on a predetermined rule, and assigns a tag indicating the aligned analysis result to the analysis target data included in the group; and storing knowledge information indicating the plurality of analysis target data, the tag, and the spatial information in the memory.

Citation Information

Patent Citations

  • Data management device, data management system, and data management method

    JP2017167889A

  • Computer and learning method of model

    JP2021076913A