Presentation information generation apparatus and presentation information generation method

Abstract

To efficiently collect training data.SOLUTION: A presentation information generation apparatus includes: a data acquisition unit that acquires data from a sensor; a determination unit that determines an acquisition status of acquisition target data based on a comprehensive condition and the data acquired, the comprehensive condition being a condition related to the acquisition target data; and an information generation unit that generates presentation information corresponding to insufficient data among pieces of the acquisition target data for which the acquisition status is inadequate.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a presentation information generation device and a presentation information generation method.

Background Art

[0002] In recent years, various techniques have been proposed for collecting teacher data for machine learning on a computer. For example, when a camera sequentially performs shooting and uses the obtained images as teacher data, there is a technique for instructing so as to obtain an image that satisfies a predetermined condition (for example, Patent Documents 1 to 3).

[0003] For shooting teacher data, detailed instructions are given for each scene and shooting is performed multiple times, so it takes time for shooting. Although the shooting time is shortened by increasing the number of cameras, it is costly to increase the number of cameras.

[0004] Also, a technique has been proposed in which, based on the occurrence frequency of past data, the time until the collection of insufficient data is completed is displayed and the data collection is continued (for example, Patent Document 4).

[0005] Also, a technique has been proposed in which an area where the accumulated survey data is insufficient is extracted and a survey request for the extracted insufficient data is transmitted to the member's terminal (for example, Patent Document 5).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Summary of the Invention

Problems to be Solved by the Invention

[0007] For machine learning of a computer, a large amount of data corresponding to various situations is collected. For this purpose, efficient collection of teacher data has been studied.

[0008] This disclosure contributes to providing a presentation information generation device and a presentation information generation method capable of efficiently collecting teacher data.

Means for Solving the Problems

[0009] A presentation information generation device according to an embodiment of the present disclosure includes: a data acquisition unit that acquires data from a sensor; a determination unit that determines an acquisition status of the acquisition target data based on an exhaustive condition that is a condition regarding the acquisition target data and the acquired data; and an information generation unit that generates presentation information corresponding to insufficient data for which the acquisition status is insufficient among the acquisition target data.

[0010] Also, a presentation information generation method according to an embodiment of the present disclosure acquires data from a sensor, determines insufficient data that does not satisfy the exhaustive condition among the acquisition target data based on the exhaustive condition set for the acquisition target data and the acquired data, and generates presentation information corresponding to the insufficient data.

Effects of the Invention

[0011] According to the present disclosure, teacher data can be efficiently collected.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the embodiments described below are examples, and the present disclosure is not limited to the following embodiments.

[0014] However, a more detailed description may be omitted as necessary. For example, a detailed description of well-known matters or a redundant description of substantially the same configuration may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art.

[0015] <Embodiment 1> Hereinafter, Embodiment 1 of the present disclosure will be described.

[0016] FIG. 1 is a functional block diagram of a presentation information generation device according to Embodiment 1. This presentation information generation device is a device that collects teacher data used for machine learning. For example, this presentation information generation device collects image data used for machine learning for estimating the work content of workers as teacher data in order to improve the work efficiency of workers in a factory.

[0017] The prompt information generation device includes a sensor 10, an acquired data storage unit 20, an exhaustive condition storage unit 30, a data collection processing unit 40, and a presentation unit 50. The data collection processing unit 40 includes a data acquisition unit 41, a data acquisition status determination unit 42, a similarity calculation unit 43, and a prompt information generation unit 44.

[0018] The data collection processing unit 40 is composed of a processor such as a CPU. The data acquisition unit 41, the data acquisition status determination unit 42, the similarity calculation unit 43, and the prompt information generation unit 44 exist as functions of the CPU.

[0019] The sensor 10, the acquired data storage unit 20, the exhaustive condition storage unit 30, and the presentation unit 50 may be a processing device integrated with the data collection processing unit 40, or may be connected to the data collection processing unit 40 by communication means.

[0020] The sensor 10 acquires various data. The sensor 10 is, for example, a camera, a microphone, a laser, a lidar, an ultrasonic sensor, or a sensor that acquires temperature, humidity, wind speed, concentration of substances, etc. Hereinafter, the case where the sensor 10 is a camera and the data to be acquired is image data will be described.

[0021] This sensor 10 may capture images at predetermined intervals or may capture images continuously. The sensor 10 outputs the image data of the captured images to the data acquisition unit 41 and the prompt information generation unit 44.

[0022] The acquired data storage unit 20 stores the image data acquired from the sensor 10 up to now as acquired data. In addition, the acquired data storage unit 20 stores the meta information generated based on the acquired data in association with the acquired data. The meta information will be described in detail later.

[0023] The exhaustive condition storage unit 30 stores the exhaustive conditions in which the classification of the image data to be acquired is set. In the exhaustive conditions, as teacher data, it is determined what kind of classified image data needs to be collected, and the acquisition target number of the image data in each classification is set.

[0024] In addition, in order to use the image data acquired by the sensor 10 as teacher data, the data acquisition unit 41 performs image analysis on the acquired image data and generates meta information including information on feature amounts corresponding to the classification of the image data.

[0025] The classification of the image data is, for example, in a factory, a classification obtained by dividing the orientation and inclination of the body of the worker included in the image data into a plurality of ranges. The data acquisition unit 41 generates meta information including information on the feature amounts corresponding to the classification such as the skeleton information of the worker included in the image data acquired by the sensor 10, the orientation and inclination of the body.

[0026] Note that the data acquisition unit 41 may extract the image data at predetermined time intervals from the image data continuously captured by the sensor 10 and generate meta information, or generate meta information for some or all of the image data acquired by the sensor 10 at predetermined time intervals.

[0027] Based on the comprehensive conditions set for the classification of the image data to be acquired and the information on the feature amounts of the image data acquired from the sensor 10, the data acquisition status determination unit 42 determines the classification of the image data that is lacking (the classification with an insufficient acquisition status).

[0028] For example, the data acquisition status determination unit 42 refers to the comprehensive conditions stored in the comprehensive condition storage unit 30 and determines whether the number for each classification of the image data stored in the acquired data has reached the collection target number of teacher data for each classification set in the comprehensive conditions.

[0029] Then, the data acquisition status determination unit 42 determines the image data corresponding to the classification that has not reached the collection target number of teacher data (the classification with an insufficient acquisition status) as the lacking image data (lacking data).

[0030] The similarity calculation unit 43 determines whether the feature amount of the image data (acquired data) acquired from the sensor 10 is similar to the feature amount of the image data included in the classification of insufficient data (classification with insufficient acquisition status), and outputs the determination result to the presentation information generation unit 44. Note that when the similarity calculation unit 43 determines that the feature amount of the image data currently acquired by the sensor 10 is not similar to the classification of "insufficient image data (insufficient data)" which is a new acquisition target, it may omit outputting the determination result.

[0031] For example, when the orientation of the body and the inclination of the body of the person in the image data acquired from the sensor 10 are within a predetermined range from the orientation of the body and the inclination of the body of the insufficient data, the similarity calculation unit 43 determines that the feature amount of the image data acquired from the sensor 10 is similar to the classification of the insufficient data.

[0032] Also, when the image data acquired from the sensor 10 includes a person, the similarity calculation unit 43 may determine that the feature amount of the image data acquired from the sensor 10 is similar to the classification of the insufficient data, and when the image data acquired from the sensor 10 does not include a person, it may determine that they are not similar.

[0033] When the presentation information generation unit 44 determines that the feature amount of the image data acquired from the sensor 10 is similar to the classification of the insufficient data by the similarity calculation unit 43, it generates presentation information corresponding to the classification of the insufficient data, and outputs the generated presentation information to the presentation unit 50.

[0034] For example, when the insufficient data is image data of a person facing 90 degrees to the right, the presentation information generation unit 44 generates, as presentation information, image data in which character information "Please face 90 degrees to the right" is superimposed on the image data acquired by the sensor 10. By presenting this presentation information to the user 60, the user 60 can be made to face 90 degrees to the right, and acquisition of the insufficient data becomes possible.

[0035] Note that the information superimposed on the image data is not limited to character information, and may be pictograms or animations, as long as it is information that can be visually recognized by the user 60.

[0036] When the feature amount of the image data acquired from the sensor 10 by the similarity calculation unit 43 and the classification of the shortage data are not determined to be similar, the presentation information generation unit 44 may not generate presentation information, or may generate image data in which pre-stored character information such as "There is no instruction" is superimposed.

[0037] The presentation unit 50 presents the presentation information generated by the presentation information generation unit 44 to the user 60. The presentation unit 50 is, for example, an output unit such as a display or a speaker. Then, while the user 60 confirms the information presented by the presentation unit 50, the user adjusts the body orientation or the like so as to be able to acquire the shortage data.

[0038] Note that the presentation unit 50 is not limited to one, and there may be a plurality of them. Also, the presentation unit 50 is not limited to a display, and may be a device such as AR (Augmented Reality) glasses that enables three-dimensional visualization.

[0039] FIG. 2 is a diagram showing an example of exhaustive conditions. Four classifications of body orientations, namely 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and three classifications of body inclinations, namely 0 degrees, 45 degrees, and 90 degrees, are set as conditions for the data to be acquired.

[0040] When the body inclination is 0 degrees and 45 degrees, the target number of image data collections is set to 100 regardless of the body orientation. When the body inclination is 90 degrees, the target number of image data collections is set to 50 regardless of the body orientation. Since the frequency of the state where the body inclination is 90 degrees is not very high, the number of teacher data is set to be small.

[0041] FIG. 3 is a diagram showing the determination result of the division of insufficient data (the division with insufficient acquisition status). In the example of FIG. 3, it is shown that the image data with the body orientation of 0 degrees and the body inclination of 0 degrees, the image data with the body orientation of 0 degrees and the body inclination of 45 degrees, and the image data with the body orientation of 90 degrees and the body inclination of 0 degrees are insufficient as teacher data and the shooting has not been completed.

[0042] In the example of FIG. 3, the data acquisition status determination unit 42 determines whether it is "shooting completed" or "shooting not completed". Instead of "shooting not completed", it may be determined how much image data is insufficient, for example, as "32 more pieces are needed". Note that "shooting completed" may be called sufficient data, and "shooting not completed" may be called insufficient data.

[0043] FIG. 4 is a diagram showing an example of the image data captured by the camera. FIG. 5 is a diagram showing an example of the meta information generated from the image data. The data acquisition unit 41 analyzes the person in the image data of FIG. 4 and generates, as meta information, skeleton information (dotted line in FIG. 5), information on the body orientation of 0 degrees, and information on the body inclination of 0 degrees. Here, it is assumed that the meta information includes skeleton information, information on the body orientation, and information on the body inclination, but other information may also be included.

[0044] Then, the image data shown in FIG. 4 and the meta information generated by the data acquisition unit 41 are stored in the acquired data storage unit 20.

[0045] FIG. 6 is a diagram showing the determination result of the division of insufficient data after new image data is acquired. FIG. 6 shows the situation where, in the state shown in FIG. 3, the image data shown in FIG. 4 is captured and registered together with the meta information, and as a result, the shooting status has changed to "shooting completed" because the image data with the body orientation of 0 degrees and the body inclination of 0 degrees has satisfied the collection target number defined in the coverage condition.

[0046] After that, when the orientation of the person's body and the inclination of the body included in the image data acquired by the sensor 10 are within a predetermined range from 90 degrees and 0 degrees respectively, it is determined that the image data in the category where the body orientation is 90 degrees and the body inclination is 0 degrees is "insufficient data" that is newly the acquisition target.

[0047] Then, presentation information such as "Please face 90 degrees to the right" or "Please assume a posture where the body orientation is 90 degrees and the body inclination is 0 degrees" is generated.

[0048] FIG. 7 is a diagram showing an example of the presentation information presented by the presentation unit 50. The presentation unit 50 presents the presentation information "Please face 90 degrees to the right" generated by the presentation information generation unit 44.

[0049] FIG. 8 is a diagram showing an example of the image data in which the meta information is generated. FIG. 8 shows the image data in which the subject (user 60) changed the body orientation and the body orientation was 45 degrees to the right from the state of FIG. 4. The data acquisition unit 41, the similarity calculation unit 43, and the presentation information generation unit 44 perform the same processing as when the image data of FIG. 4 was taken, for example, generate presentation information such as "Please face 45 degrees to the right".

[0050] FIG. 9 is a diagram showing an example of the presentation information presented by the presentation unit 50. The presentation information includes character information "Please face 45 degrees to the right". By the subject changing the body orientation by 45 degrees further according to this presentation information, it is possible to acquire the image data where the body orientation is 90 degrees and the body inclination is 0 degrees, which is insufficient as teacher data.

[0051] Here, the case where the image data used for learning to estimate the work content of the worker is collected as teacher data has been described. However, as long as it is the collection of teacher data used for learning, the technology of the present disclosure can be applied to other types of image data or teacher data that is not image data.

[0052] Hereinafter, as an example of other teacher data, a case of collecting image data used for machine learning to estimate the presence of a person in order for a vehicle to avoid a collision with a person will be described.

[0053] FIG. 10 is a diagram showing another example of coverage conditions. In these coverage conditions, a classification of the distance to a person is set. Specifically, in these coverage conditions, four classifications of distances in the X-axis direction, namely 0 cm, 100 cm, 200 cm, and 300 cm, and three classifications of distances in the Z-axis direction, namely 100 cm, 200 cm, and 300 cm, are set.

[0054] And for each combination of the classification of the distance in the X direction and the classification of the distance in the Z direction, the target number of image data to be collected is set to 100.

[0055] FIG. 11 is a diagram showing the determination result of the classification of insufficient data. In the example of FIG. 11, it is shown that the image data with a distance in the X-axis direction of 0 cm and a distance in the Z-axis direction of 100 cm, the image data with a distance in the X-axis direction of 100 cm and a distance in the Z-axis direction of 100 cm, and the image data with a distance in the X-axis direction of 0 cm and a distance in the Z-axis direction of 300 cm are insufficient as teacher data and the shooting has not been completed.

[0056] FIG. 12 is a diagram showing an example of image data captured by a camera. FIG. 13 is a diagram showing an example of meta information generated from the image data. The data acquisition unit 41 analyzes the person in the image data of FIG. 12 and generates, as meta information, information on the distance in the X-axis direction of 0 cm and the distance in the Z-axis direction of 300 cm as feature amounts of the image data.

[0057] Note that the meta information may include other information. Also, instead of the meta information generated by the data acquisition unit 41, the distance data measured by a distance sensor may be set as the meta information of the image data.

[0058] The image data shown in FIG. 13 and the meta information generated by the data acquisition unit 41 are stored in the acquired data storage unit 20.

[0059] Figure 14 is a diagram showing the determination result of the classification of insufficient data after new image data is acquired. Figure 14 shows a situation where, in the state shown in Figure 11, the image data shown in Figure 12 is taken, and as a result of registering the image data together with the meta information, the image data at a distance of 0 cm in the X-axis direction and 300 cm in the Z-axis direction satisfies the target number of collections defined by the coverage condition, and the shooting status has changed to "shooting completed".

[0060] After that, when the distances in the X-axis direction and the Z-axis direction to the person included in the image data acquired by the sensor 10 are within a predetermined range from 0 cm and 100 cm respectively, the image data in the section where the distance in the X-axis direction is 0 cm and the distance in the Z-axis direction is 100 cm is determined to be "insufficient data". Then, the prompt information "Please move 200 cm forward" is generated.

[0061] Figure 15 is a diagram showing an example of the prompt information presented by the presentation unit 50. The prompt information includes the character information "Please move 200 cm forward".

[0062] Figure 16 is a diagram showing an example of the image data for which meta information is generated. Figure 16 shows a state where the data acquisition unit 41 generates meta information for the image data in which the state where the photographed person (user 60) has moved forward according to the prompt information presented by the presentation unit 50 is photographed by the camera.

[0063] Specifically, Figure 16 shows the image data in which the photographed person has moved 100 cm forward from the state of Figure 15 and the distance in the Z-axis direction has become 200 cm. In this case, the data acquisition unit 41, the similarity calculation unit 43, and the prompt information generation unit 44 perform the same processing as when the image data of Figure 12 is taken, and generate, for example, the prompt information "Please move 100 cm forward".

[0064] FIG. 17 is a diagram showing an example of the presentation information presented by the presentation unit 50. By moving forward according to the presentation information presented by the presentation unit 50 to the photographed person (user 60), it is possible to acquire image data with a distance of 0 cm in the X-axis direction and a distance of 100 cm in the Z-axis direction, which is insufficient as teacher data.

[0065] FIG. 18 is a diagram showing a flowchart of the presentation information generation process.

[0066] The data collection processing unit 40 receives the setting of the coverage condition by an input unit (not shown) and stores the coverage condition in the coverage condition storage unit 30 (step S101). In the coverage condition, it is set how much image data belonging to each section is to be acquired, and the coverage condition is set by the designer of machine learning or the like.

[0067] Subsequently, when the coverage condition is set, the data acquisition unit 41 acquires data from the sensor 10 (step S102). For example, when the sensor 10 is a camera, the data acquisition unit 41 acquires image data.

[0068] Thereafter, the data acquisition unit 41 generates meta information for the image data acquired from the sensor 10, associates the image data and the meta information, and stores them in the acquired data storage unit 20 (step S103).

[0069] Note that the data acquired by a sensor other than the camera may be used as meta information. For example, the distance acquired by the distance sensor may be used as the meta information of the image data.

[0070] Then, the data acquisition status determination unit 42 refers to the coverage condition and determines whether the number of image data for each section stored in the acquired data is insufficient (step S104). The result of this determination may be indicated as "completed" or "incomplete", or may be indicated by the number of insufficient image data.

[0071] Furthermore, the data acquisition status determination unit 42 determines whether to end the shooting (step S105). When the data acquisition status determination unit 42 determines that the required number of image data has been acquired for all sections, it causes the sensor 10 to end the shooting (step S105, Yes).

[0072] When the data acquisition status determination unit 42 determines that the required number of image data has not been acquired for all sections, the process proceeds to step S106 (step S105, No).

[0073] Next, the similarity calculation unit 43 determines whether the feature amount of the image data acquired from the sensor 10 is similar to the section of the missing data (step S106).

[0074] When it is determined that the feature amount of the image data is not similar to the section of the missing data (step S106, No), the process returns to step S102, and the data acquisition unit 41 acquires the next image data from the sensor 10.

[0075] When it is determined that the feature amount of the image data is similar to the section of the missing data (step S106, Yes), presentation information corresponding to the section of the missing data is generated, and the generated presentation information is output to the presentation unit 50 (step S107).

[0076] The presentation information generated by the presentation information generation unit 44 is, for example, character information such as "Please turn 45 degrees to the right".

[0077] Then, the presentation unit 50 presents the presentation information generated by the presentation information generation unit 44 to the user 60 (step S108).

[0078] Thereafter, the process returns to step S102, and the data acquisition unit 41 acquires the next image data from the sensor 10. The data acquisition unit 41 may acquire image data from the sensor 10 after a predetermined time has elapsed since the presentation information was presented in step S108.

[0079] The image data acquired in step S102 is the image data after the presentation information is presented to the user 60 in step S108 and the user 60 moves their body according to the presentation information. Therefore, it is expected that the insufficient data determined by the data acquisition status determination unit 42 will be acquired.

[0080] As described above, the presentation information generation device according to Embodiment 1 can efficiently collect image data with a low occurrence frequency and difficult to collect.

[0081] <Embodiment 2> FIG. 19 is a functional block diagram of a presentation information generation device according to Embodiment 2.

[0082] This presentation information generation device has an acquisition status absolute amount determination unit 45 instead of the data acquisition status determination unit 42 in Embodiment 1. The functions of the other functional units are the same as those in Embodiment 1.

[0083] The acquisition status absolute amount determination unit 45 determines the classification of insufficient data based on the comprehensive conditions in which the classification of the image data to be acquired is set and the feature amount of the image data acquired from the sensor 10, similar to the data acquisition status determination unit 42.

[0084] However, the acquisition status absolute amount determination unit 45 determines the image data that is the target of priority acquisition among the insufficient data. Then, the presentation information generation unit 44 generates presentation information from the image data that is the target of priority acquisition.

[0085] For example, the acquisition status absolute amount determination unit 45 records the occurrence frequency of the image data in each classification as log information, and preferentially selects, as the classification for acquiring new image data, the classification with a lower occurrence frequency of the image data in each classification of the image data.

[0086] When the similarity calculation unit 43 determines that the feature amount of the image data currently acquired by the sensor 10 is similar to the category of the new acquisition target image data, the presentation information generation unit 44 generates presentation information corresponding to the category of the insufficient data and outputs the generated presentation information to the presentation unit 50.

[0087] By presenting such presentation information to the user 60 and prompting the user to take action, it is possible to efficiently collect the set number of image data in each category.

[0088] Note that the acquisition status absolute amount determination unit 45 may preferentially select a category with a higher occurrence frequency of image data as the category for acquiring new image data. Further, the acquisition status absolute amount determination unit 45 may refer to the coverage condition and select, as the image data to be preferentially acquired, the image data with a large number of collection target numbers.

[0089] Further, the acquisition status absolute amount determination unit 45 may, based on the collection target number of each category set in the coverage condition and the feature amount of the image data acquired from the sensor 10, select, as the image data to be preferentially acquired, the image data with a larger shortage number.

[0090] Further, the acquisition status absolute amount determination unit 45 may consider the time when the image data acquired from the sensor 10 was taken. For example, if the category of the image data includes the category of time and the image data in the night category is insufficient, the image data in the night may be preferentially selected as the image data to be acquired.

[0091] In this case, the presentation information generation unit 44 generates presentation information such as "Please set the environment to night" and "The environment is now night. Please move 200 cm forward."

[0092] Alternatively, you may manually define in advance the categories of image data to be preferentially collected, and select the data of those categories as the image data to be preferentially acquired. Note that the categories of image data to be preferentially collected may be appropriately defined by a person in consideration of, based on empirical rules, those for which a larger absolute amount is better or those for which a smaller absolute amount is acceptable.

[0093] <Embodiment 3> FIG. 20 is a functional block diagram of the presentation information generation device according to Embodiment 3.

[0094] Embodiment 3 has a relative information generation unit 46 instead of the presentation information generation unit 44 in Embodiment 1. The functions of the other functional units are the same as those in Embodiment 1.

[0095] Similar to the presentation information generation unit 44, the relative information generation unit 46 generates presentation information corresponding to the categories of insufficient data from the sensor 10.

[0096] The relative information generation unit 46 generates, as presentation information, information indicating an action (state difference) for transitioning from the current state to the state of insufficient data. Therefore, the person being photographed (user 60) can easily recognize what to do with respect to the current situation.

[0097] For example, the relative information generation unit 46 generates presentation information such as "Please turn 90 degrees to the right" and "Please move 100 cm forward" as instructions based on the information indicating the action (difference).

[0098] <Embodiment 4> FIG. 21 is a functional block diagram of the presentation information generation device according to Embodiment 4.

[0099] Embodiment 4 has an animation generation unit 47 instead of the presentation information generation unit 44 in Embodiment 1. The functions of the other functional units are the same as those in Embodiment 1.

[0100] Similar to the presentation information generation unit 44, the animation generation unit 47 generates presentation information corresponding to the classification of the missing data from the sensor 10.

[0101] However, the animation generation unit 47 generates a video (animation) or a still image showing the transition from the current state to the state of the missing data as the presentation information to be generated.

[0102] FIG. 22 is a diagram showing an example of the animation generated by the animation generation unit 47. For example, in the state of FIG. 4, as presentation information for acquiring image data in which the body orientation is 90 degrees and the body tilt is 0 degrees, the animation generation unit 47 generates a video in which the body orientation rotates 90 degrees as shown in FIG. 22. By performing an operation according to the animation presented to the user by the presentation unit 50, it becomes possible to acquire the missing data.

[0103] In addition, when a still image showing the transition from the current state to the state of the missing data is generated as the presentation information, the transition is represented by, for example, an arrow or the like.

[0104] <Embodiment 5> FIG. 23 is a functional block diagram of the presentation information generation device according to Embodiment 5.

[0105] In Embodiment 5, the data acquisition unit 41 in Embodiment 1 outputs meta information to the presentation information generation unit 44. Then, the presentation information generation unit 44 generates information including the meta information as the presentation information to be presented to the user 60. Then, the presentation unit 50 presents the presentation information including the meta information. The functions of the other functional units are the same as those in Embodiment 1.

[0106] FIG. 24 is a diagram showing an example of the presentation information including the meta information. The presentation information in FIG. 24 includes character information "Please face 90 degrees to the right" and skeleton information which is a kind of meta information. Thereby, the user 60 can easily confirm the current state, and the presentation information generation device can efficiently acquire the missing data.

[0107] <Embodiment 6> FIG. 25 is a functional block diagram of a presentation information generation apparatus according to Embodiment 6.

[0108] In Embodiment 6, the data acquisition status determination unit 42 determines the classification of acquired image data (sufficient data) and the classification of insufficient data (deficient data) based on the feature amount of the image data stored in the acquired data storage unit 20 and the coverage condition stored in the coverage condition storage unit 30, and outputs the result to the presentation information generation unit 44.

[0109] For example, the data acquisition status determination unit 42 outputs information indicating a determination result as shown in FIG. 3 to the presentation information generation unit 44.

[0110] Then, the presentation information generation unit 44 generates information including the determination results of the classification of the acquired image data and the classification of the insufficient data as presentation information. Then, the presentation unit 50 presents the presentation information. The functions of the other functional units are the same as those in Embodiment 1.

[0111] For example, when presenting presentation information including a determination result as shown in FIG. 3, since the determination result can represent two states (variables) in a table format, the user can recognize what states are required.

[0112] Also, when the classification of the image data is represented by three or more states (variables), the two states used for the information displayed in the table format may be selected from the three or more states, and the selected states may be the states of interest or representative states.

[0113] FIG. 26 is a diagram showing an example of an image in which table format information indicating the unacquired / acquired classification is superimposed on the upper right part of the screen.

[0114] In this way, by presenting which category is insufficient, the user 60 can recognize the category for which image data is to be acquired, so the user 60 can quickly act to make the data in a state of insufficient data.

[0115] <Embodiment 7> FIG. 27 is a functional block diagram of the presentation information generation device according to Embodiment 7.

[0116] Embodiment 7 has a display 51 and a speaker 52 as a presentation unit. Note that the display 51 may not be provided. The presentation information generation unit 44 generates audio information as well as display information as presentation information. The audio information generated by the presentation information generation unit 44 is output to the user 60 by the speaker 52. The functions of the other functional units are the same as those in Embodiment 1.

[0117] For example, when the presentation information generation unit 44 generates presentation information as an image as shown in FIG. 7, it also generates audio information output by the speaker 52 such as "Please turn 90 degrees to the right".

[0118] In the case of the presentation information "Please turn around", when the user performs an operation based on the presentation information, it is difficult for the user 60 to check the presentation information displayed on the display 51. However, by outputting the presentation information as audio from the speaker 52, it becomes easier for the user 60 to continue to recognize the presentation information.

[0119] <Embodiment 8> FIG. 28 is a functional block diagram of the presentation information generation device according to Embodiment 8.

[0120] In Embodiment 8, the exhaustive condition dynamic control unit 48 changes the exhaustive condition based on the data stored in the acquired data storage unit 20. When changing the exhaustive condition, the exhaustive condition dynamic control unit 48 may consider the data being acquired by the sensor 10. The functions of the other functional units are the same as those in Embodiment 1.

[0121] The coverage condition dynamic control unit 48 changes the coverage condition every time a predetermined number of image data are newly stored in the acquired data storage unit 20. For example, when the coverage condition dynamic control unit 48 determines that the number of stored image data of a specific category is small as the acquired data, it changes the category of the image data with the small number of stored data.

[0122] For example, assume that the body orientation is set in four categories of 0 degrees, 90 degrees, 180 degrees, and 270 degrees.

[0123] In this situation, as a result of analyzing the meta information of the image data stored in the acquired data storage unit 20, assume that the coverage condition dynamic control unit 48 determines that among the image data from -45 degrees to 45 degrees included in the 0-degree coverage condition category, the image data from 0 degrees to 45 degrees are stored in a large amount, but the image data from -45 degrees to 0 degrees are not stored very much. And due to this bias, the accuracy of inference using the learned model may not increase.

[0124] In this case, the coverage condition dynamic control unit 48 sets, as the coverage condition, a -22.5-degree category corresponding to the image data from -45 degrees to 0 degrees and a 22.5-degree category corresponding to the image data from 0 degrees to 45 degrees.

[0125] Then, since there is not much stored image data in the -22.5-degree category, it becomes "shooting not completed", and since there is a large amount of stored image data in the 22.5-degree category, it becomes "shooting completed".

[0126] In this way, by changing the category of the coverage condition, the category can be set more appropriately, and the image data belonging to the category can be efficiently collected.

[0127] Note that the coverage condition dynamic control unit 48 may exist outside the data collection processing unit 40.

[0128] <Embodiment 9> FIG. 29 is a functional block diagram of the presentation information generation device according to Embodiment 9.

[0129] Embodiment 9 has a variable instruction unit 49 instead of the presentation information generation unit 44 in Embodiment 1. The functions of the other functional units are the same as those in Embodiment 1.

[0130] Similar to the presentation information generation unit 44, the variable instruction unit 49 generates presentation information corresponding to the classification of the missing data from the sensor 10.

[0131] However, the variable instruction unit 49 generates information with different presentation modes according to the content to be presented as the presentation information.

[0132] For example, when the variable instruction unit 49 generates presentation information such as "Please turn 45 degrees to the right" and when it generates presentation information such as "Please turn 135 degrees to the right", the variable instruction unit 49 changes the presentation mode of the presentation information.

[0133] Specifically, when the variable instruction unit 49 causes the display 51 to display presentation information such as "Please turn 45 degrees to the right", it generates presentation information that causes the display to be performed with fast blinking or to be displayed in red characters. When the presentation information is output from the speaker 52, it generates presentation information that shortens the output interval of the voice or outputs a sound like "pip pip pip" with a short interval and a high pitch.

[0134] On the other hand, when the variable instruction unit 49 causes the display 51 to display presentation information such as "Please turn 135 degrees to the right", for example, it generates presentation information that causes the display to be performed with slow blinking or to be displayed in blue characters. When the presentation information is output from the speaker 52, it generates presentation information that lengthens the output interval of the voice or outputs a sound like "pi, pi, pi" with a long interval and a low pitch.

[0135] In addition, the variable instruction unit 49 may change the presentation form generated according to, for example, the similarity output by the similarity calculation unit 43.

[0136] For example, when the similarity calculation unit 43 determines that the feature amount of the image data acquired from the sensor 10 is similar to the classification of the missing data, the variable instruction unit 49 generates presentation information that is displayed in red characters and also generates presentation information that outputs a sound at short intervals (or a high sound).

[0137] On the other hand, when the similarity calculation unit 43 determines that the feature amount of the image data acquired from the sensor 10 is not similar to the classification of the missing data, the variable instruction unit 49 generates presentation information that is displayed in blue characters and also generates presentation information that outputs a sound at long intervals (or a low sound).

[0138] <Combination of Embodiments> Note that the above-described multiple embodiments can be combined.

[0139] FIG. 30 is a functional block diagram of a presentation information generation apparatus that combines a plurality of embodiments. In this example, the presentation information generation unit 44 includes a relative information generation unit 46, an animation generation unit 47, and a variable instruction unit 49.

[0140] In this case, for example, the relative information generation unit 46 generates information indicating the difference between the state of the missing data and the current state as presentation information. Then, based on the information indicating the difference between the two states, the animation generation unit 47 generates a moving image (animation) or a still image indicating the transition from the current state to the state of the missing data.

[0141] Furthermore, the variable instruction unit 49 generates presentation information with different presentation modes according to the classification of the image data. Then, the display 51 or the speaker 52 presents the presentation information to the user 60 in different presentation modes according to the classification of the missing data from the sensor 10. The functions of the other functional units are the same as those in the first embodiment.

[0142] For example, when the animation generation unit 47 and the variable instruction unit 49 generate presentation information that prompts the user 60 to face right, they may generate an animation of a person with a low voice (male voice) wearing blue clothing, and when generating presentation information that prompts the user to face left, they may generate an animation of a person with a high voice (female voice) wearing red clothing.

[0143] Furthermore, in addition to the animation, the presentation information generation unit 44 may generate presentation information that displays, in table format, information including the classification of the acquired image data and the determination result of the classification of the missing data, and the exhaustive condition dynamic control unit 48 may further perform dynamic modification of the exhaustive condition.

[0144] As described above, the embodiments have been described with reference to the drawings, but the present disclosure is not limited to such examples. It is obvious that those skilled in the art can conceive of various modification examples or correction examples within the scope described in the claims. Such modification examples or correction examples are also understood to belong to the technical scope of the present disclosure. Also, within the scope not departing from the gist of the present disclosure, the components in the embodiments may be arbitrarily combined.

[0145] (1) The presentation information generation device according to an embodiment of the present disclosure includes a data acquisition unit that acquires data from a sensor, a determination unit that determines the acquisition status of the acquisition target data based on an exhaustive condition that is a condition regarding the acquisition target data and the acquired data, and an information generation unit that generates presentation information corresponding to missing data for which the acquisition status is insufficient among the acquisition target data.

[0146] (2) The presentation information generation device according to an embodiment of the present disclosure further includes, in the presentation information generation device of (1), a similarity calculation unit that determines whether the acquired data and the missing data are similar, and the information generation unit generates the presentation information when the acquired data and the missing data are similar.

[0147] (3) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the data acquisition unit generates meta-information including information on the feature amount of the acquired data and associates it with the acquired data.

[0148] (4) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the exhaustive condition includes a category for classifying the data to be acquired and the acquisition target number for each category, and the determination unit determines a category that has not reached the acquisition target number as the insufficient data.

[0149] (5) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the determination unit determines priority data that is the target of priority acquisition among the insufficient data, and the information generation unit preferentially generates the presentation information of the priority data.

[0150] (6) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the information generation unit generates information indicating actions from the acquired data to the insufficient data as the presentation information.

[0151] (7) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the acquired data is an image, and the information generation unit generates a moving image or a still image indicating the transition from the acquired data to the insufficient data as the presentation information.

[0152] (8) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (3), the information generation unit generates information including the meta-information as the presentation information.

[0153] (9) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the information generation unit further generates the presentation information corresponding to the sufficient data for which the acquisition status is sufficient among the data to be acquired.

[0154] (10) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the information generation unit generates information including voice information as the presentation information.

[0155] (11) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), further includes a control unit that changes the coverage condition based on the acquired data.

[0156] (12) In an embodiment of the present disclosure, the presentation information generation device, in the presentation information generation device of (1), the information generation unit generates information with different presentation modes according to the content to be presented as the presentation information.

[0157] (13) In an embodiment of the present disclosure, the presentation information generation method acquires data from a sensor, determines missing data that does not meet the coverage condition among the data based on the coverage condition set for the data to be acquired and the acquired data, and generates presentation information corresponding to the missing data.

[0158] (14) In an embodiment of the present disclosure, the presentation information generation method, in the presentation information generation method of (13), further generates the presentation information when the acquired data and the missing data are similar.

[0159] (15) In an embodiment of the present disclosure, the presentation information generation method, in the presentation information generation method of (13), further generates meta information including information on the feature amount of the data based on the acquired data, and stores it in a storage unit in association with the data.

[0160] (16) In an embodiment of the present disclosure, the presentation information generation method, in the presentation information generation method of (13), the number of the data to be acquired is set for each category of the data in the coverage condition, and the determination determines the data in the category where the number of the data is insufficient.

[0161] (17) In an embodiment of the present disclosure, the method for generating presentation information is, in the method for generating presentation information in (13), the determination determines data that is preferentially an acquisition target among the lacking data, and the generation of the presentation information generates the presentation information from the data that is preferentially an acquisition target.

[0162] (18) In an embodiment of the present disclosure, the method for generating presentation information is, in the method for generating presentation information in (13), as the presentation information, generates information indicating a difference in actions when transitioning from the current state to the state of the lacking data.

[0163] (19) In an embodiment of the present disclosure, the method for generating presentation information is, in the method for generating presentation information in (13), as the presentation information, generates a moving image or a still image indicating the transition from the current state to the state of the lacking data.

[0164] (20) In an embodiment of the present disclosure, the method for generating presentation information is, in the method for generating presentation information in (15), as the presentation information, generates information including the meta information.

Industrial Applicability

[0165] The present disclosure can be used in a presentation information generation device and a presentation information generation method.

Explanation of Signs

[0166] 10 Sensor 20 Acquired Data Storage Unit 30 Coverage Condition Storage Unit 40 Data Collection Processing Unit 41 Data Acquisition Unit 42 Data Acquisition Status Judgment Unit 43 Similarity Calculation Unit 44 Presentation Information Generation Unit 45 Acquisition Status Absolute Amount Judgment Unit 46 Relative Information Generation Unit 47 Animation Generation Unit 48 Coverage Condition Dynamic Control Unit 49 Variable Instruction Unit 50 Presentation Unit 51 Display 52 Speaker 60 User

Claims

1. A data acquisition unit that acquires data from a sensor, A determination unit that determines the acquisition status of the data to be acquired based on the exhaustive condition, which is a condition regarding the data to be acquired, and the acquired data, An information generation unit that generates presentation information corresponding to insufficient data for which the acquisition status is insufficient among the data to be acquired, A presentation information generation device comprising:

2. Further comprising a similarity calculation unit that determines whether the acquired data and the insufficient data are similar, The information generation unit generates the presentation information when the acquired data and the insufficient data are similar, The presentation information generation device according to Claim 1.

3. The data acquisition unit generates meta information including information on the feature amount of the acquired data and associates it with the acquired data, The presentation information generation device according to Claim 1.

4. The exhaustive condition includes a category for classifying the data to be acquired and a target acquisition number for each category, The determination unit determines a category that has not reached the target acquisition number as the insufficient data, The presentation information generation device according to Claim 1.

5. The determination unit determines priority data that is the target of priority acquisition among the insufficient data, The information generation unit preferentially generates the presentation information for the priority data, The presentation information generation device according to Claim 1.

6. The information generation unit generates, as the presentation information, information indicating actions until transitioning from the acquired data to the insufficient data, The presentation information generation device according to Claim 1.

7. The acquired data is an image, The information generation unit generates, as the presentation information, a moving image or a still image indicating the transition from the acquired data to the insufficient data, The presentation information generation device according to Claim 1.

8. The information generation unit generates, as the presentation information, information including the meta information, The presentation information generation device according to Claim 3.

9. The information generation unit further generates the presentation information corresponding to sufficient data for which the acquisition status is sufficient among the data to be acquired, The presentation information generation device according to Claim 1.

10. The information generation unit generates, as the presentation information, information including audio information, The presentation information generation device according to Claim 1.

11. Further comprising a control unit that changes the exhaustive condition based on the acquired data, The presentation information generation device according to Claim 1.

12. The information generation unit generates information with different presentation modes according to the content to be presented as the presentation information. The presentation information generation device according to claim 1.

13. Obtain data from a sensor, Based on the comprehensive conditions for the acquisition target data set and the acquired data, determine the insufficient data among the acquisition target data that does not meet the comprehensive conditions, Generate presentation information corresponding to the insufficient data. Presentation information generation method.

14. Furthermore, when the acquired data and the insufficient data are similar, generate the presentation information. The presentation information generation method according to claim 13.

15. Furthermore, based on the acquired data, generate meta information including information on the feature amount of the data, and store it in the storage unit in association with the data. The presentation information generation method according to claim 13.

16. In the comprehensive conditions, the number of the data to be acquired for each category of the data is set, and the determination is to determine the data of the category with insufficient number of the data. The presentation information generation method according to claim 13.

17. The determination is to determine the data that is preferentially the acquisition target among the insufficient data, and the generation of the presentation information is to generate the presentation information from the data that is preferentially the acquisition target. The presentation information generation method according to claim 13.

18. As the presentation information, generate information indicating the difference in actions when transitioning from the current state to the state of the insufficient data. The presentation information generation method according to claim 13.

19. As the presentation information, generate a video or a still image indicating the transition from the current state to the state of the insufficient data. The presentation information generation method according to claim 13.

20. As the presentation information, generate information including the meta information. The presentation information generation method according to claim 15.

Images