Data generation device and data generation method

Abstract

The purpose of the present invention is to efficiently and accurately create learning data necessary for creating a moving image analysis model. A data generation device (1) is provided with: an information display control unit (101) that associates and displays job content information to each of a plurality of moving image data in which production jobs are captured; a playback display control unit (102) for playing back the moving image data; an input receiving unit (103) that receives the input of the content information of the homework to be handled by the user; and a data output control unit (104) that outputs, as output data, information relating to the moving image data and the job content information. The playback display control unit (102) changes the playback display form of the moving image data in accordance with the type of job content information associated with the moving image data being played back and displayed.

Description

Technical Field

[0001] The present invention relates to a data generation device and the like that generates learning data required for creating a dynamic image analysis model. Background Art

[0002] Conventional methods involve capturing the progress of work processes at production sites such as factories as moving images and analyzing the moving image data to improve productivity and quality. Currently, artificial intelligence (AI) is used to create moving image analysis models and automatically determine the work content.

[0003] In order to create a moving image analysis model, a large amount of moving image data must be prepared as learning data (teaching data). In order to create the learning data, it is necessary to associate the work content with the annotation of each moving image data.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 4221011

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2019-159884

[0008] Patent Document 3: Japanese Patent Application Laid-Open No. 2004-363831 Summary of the Invention

[0009] Problems to be solved by the invention

[0010] As mentioned above, since annotation of a large amount of dynamic image data as learning data is necessary, this task is extremely time-consuming. Furthermore, if there are any errors in the association of the work content, the accuracy of the learning data will be reduced, and even the precision of the dynamic image analysis model will be reduced.

[0011] An object of one embodiment of the present invention is to provide a data generation device capable of efficiently and accurately generating learning data required for generating a dynamic image analysis model.

[0012] Technical means to solve the problem

[0013] In order to solve the above-mentioned problem, a data generating device according to one aspect of the present invention includes: an information display control unit for performing the following control, i.e., associating the operation content information with each of a plurality of dynamic image data captured with the production operation and displaying the same; a replay display control unit for replaying and displaying the dynamic image data; an input acceptance unit for accepting a user's input of the operation content information associated with the dynamic image data; and a data output control unit for outputting information as output data on the association between the dynamic image data input accepted by the input acceptance unit and the operation content information, wherein the replay display control unit changes the replay display form of the dynamic image data according to the type of the operation content information associated with the dynamic image data being replayed and displayed.

[0014] In order to solve the above-mentioned problem, a data generation method according to one aspect of the present invention includes: an information display control step for performing the following control, i.e., associating the operation content information with each of a plurality of dynamic image data captured with the production operation and displaying the same; a playback display control step for replaying and displaying the dynamic image data; an input acceptance step for accepting the user's input of the operation content information associated with the dynamic image data; and a data output control step for outputting information as output data on the association between the dynamic image data and the operation content information accepted by the input acceptance step, wherein the playback display control step changes the playback display form of the dynamic image data according to the type of the operation content information associated with the dynamic image data being played back and displayed.

[0015] Effects of the Invention

[0016] According to one embodiment of the present invention, it is possible to efficiently and accurately create learning data required for creating a dynamic image analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] [ Figure 1 ] is an explanatory diagram schematically showing the structure of a data collection system 50 as an example of a data generation device to which an embodiment of the present disclosure is applied.

[0018] [ Figure 2 ] is a flowchart showing the process until AI model #2 is applied.

[0019] [ Figure 3 ] is represented by an image diagram Figure 2 A diagram illustrating the process.

[0020] [ Figure 4 ] is a block diagram showing an example of the main part structure of a data generating device.

[0021] [ Figure 5] is a diagram showing an example of the contents of a task list table stored in the storage unit of the data generating device.

[0022] [ Figure 6 ] is a diagram showing an example of a screen display for manually correcting annotation data #2, which is displayed on the display unit of the data generating device.

[0023] [ Figure 7 ] is a diagram showing another example of making the playback display form different in the playback display of dynamic image data.

[0024] [ Figure 8 ] is a diagram showing an enlarged portion of a time graph image.

[0025] [ Figure 9 ] is a diagram showing an enlarged portion of another time chart image with a different display form.

[0026] [ Figure 10 ] is a diagram showing an example of the contents of a playback speed table stored in the storage unit of the data generating device. DETAILED DESCRIPTION

[0027] Hereinafter, an embodiment of one aspect of the present disclosure (hereinafter also referred to as “the present embodiment”) will be described with reference to the drawings.

[0028] §1 Application

[0029] [Overview of the data collection system including the data generation device]

[0030] First, use Figure 1 To illustrate an example of a scenario in which the present invention is applicable, Figure 1 This is an explanatory diagram schematically illustrating the configuration of a data collection system 50, an example of a data generation device 1 according to an embodiment of the present disclosure. The data collection system 50 collects dynamic image data capturing production operations and, based on the collected dynamic image data, uses a dynamic image analysis model to collect data related to human operations (human motions) being performed.

[0031] like Figure 1 As shown, the data collection system 50 includes a data generating device 1 , a cloud platform 2 , a photographing device 31 and a terminal device 32 installed at a site 3 such as a manufacturing site.

[0032] Camera 31 captures production operations at site 3 and transmits the captured dynamic image data to cloud platform 2. Camera 31 is, for example, a webcam. To improve the accuracy of dynamic image analysis AI (artificial intelligence), multiple cameras 31 can be deployed within cloud server 21 within cloud platform 2 to transmit dynamic image data capturing the same operations from different viewing angles. Terminal device 32 displays the analysis results received from cloud platform 2, generated by the dynamic image analysis AI, to inform managers at site 3.

[0033] The cloud platform 2 includes at least the aforementioned cloud server 21 and cloud storage 22. Cloud storage 22 stores dynamic image data sent from the site 3 that captures production operations. The cloud server 21 includes dynamic image analysis AI, which uses a dynamic image analysis model to perform dynamic image analysis on the dynamic image data captured from the site 3. The cloud server 21 transmits the analysis results to a terminal device 32 at the site 3.

[0034] Therefore, as long as the site 3 is equipped with a shooting device 31 to shoot the production operation and send the captured dynamic image data to the cloud platform 2, data collection related to human operations in the production operation can be achieved to continuously improve productivity and quality.

[0035] The data generation device 1 generates the learning data required to create a dynamic image analysis model used by the dynamic image analysis AI of the cloud server 21. As previously mentioned, creating a dynamic image analysis model requires a large amount of dynamic image data as learning data. Creating this learning data requires annotating each piece of dynamic image data in order to associate the work content with the data.

[0036] The data generation device 1 automates the annotation process by using the dynamic image analysis AI of the cloud server 21, thereby improving the accuracy of the automatic annotation performed by the dynamic image analysis AI. The dynamic image analysis AI performs annotation using a dynamic image analysis model for annotation.

[0037] Hereinafter, the dynamic image analysis model used by the dynamic image analysis AI for annotation will be referred to as "AI Model #1." Furthermore, the dynamic image analysis model used to analyze dynamic image data, which was generated by the dynamic image analysis AI using learning data annotated using "AI Model #1" and was transmitted from Site 3 and captured during production work, will be referred to as "AI Model #2." Furthermore, the operation of annotating dynamic image data using the dynamic image analysis AI is sometimes referred to as "automatic annotation."

[0038] §2 Structure Example

[0039] [Process until AI Model #2 is applied on-site]

[0040] use Figure 2 and Figure 3 The process until AI model #2 is applied to site 3 is explained. Figure 2 This is a flowchart showing the process until AI model #2 is applied. Figure 3 It is represented by an image diagram Figure 2 A diagram illustrating the process.

[0041] Figure 2 P1: The user uses the data generation device 1 to collect dynamic images (video data) of a series of operations repeated for 20 cycles, for example, from the cloud storage 22 (refer to Figure 3 1.) It is also possible to collect 20 cycles of moving images of a series of operations and create one file of video data. The number of cycles collected is not limited to 20.

[0042] Figure 2 P2: The user uses the data generation device 1 to manually annotate one cycle of the collected 20 cycles of dynamic images to create annotation data #1 (see Figure 3 2.).

[0043] Specifically, the user views a single loop of the moving image while segmenting it into individual tasks. For example, by pressing a predetermined button to split the moving image, the time up to the point of pressing the button is defined as the end point of one task, and the time of pressing the button is defined as the start point of the next task. Furthermore, the user assigns task information (task content information) indicating the content of each task to each segmented moving image.

[0044] Figure 2 P3: The user uses the data generation device 1 to use the annotation data #1 as learning data to make the dynamic image analysis AI learn (train) and generate the AI ​​model #1 (refer to Figure 3 3.).

[0045] Figure 2 P4: The user uses the data generation device 1 to make the dynamic image analysis AI use the AI ​​model #1 to automatically annotate the remaining 19 cycles of dynamic images and create annotation data #2 (refer to Figure 3 4.).

[0046] Figure 2 P5: The user uses the data generation device 1 to manually correct the annotation data #2 created by the automatic annotation in P4 (see Figure 3 4.).

[0047] Figure 2 P6: The user uses the data generating device 1 to create annotation data #3 (see Figure 3 5.).

[0048] Figure 2 P7: The user uses the data generation device 1 to use the annotation data #3 created in P6 as learning data to make the dynamic image analysis AI learn and generate AI model #2 (refer to Figure 3 6.).

[0049] Figure 2 P8: The user uses the data generation device 1 to apply the AI ​​model #2 generated in P7 to the scene 3 (refer to Figure 3 7.).

[0050] [Structure of data generation device]

[0051] Next, based on Figure 4 The structure of the data generating device 1 will be described. Figure 4 1 is a block diagram showing an example of the main structure of the data generating device 1. Figure 4 As shown, data generation device 1 includes an input unit 11 for accepting input of various data; a display unit 12 for displaying various information; and a communication unit 13 for communicating between data generation device 1 and other devices. Alternatively, input unit 11 may be, for example, a touch screen, in which case the touch screen serves as both input unit 11 and display unit 12. Furthermore, data generation device 1 includes a control unit 10 for comprehensively controlling the various components of data generation device 1; and a storage unit 14 for storing various data used by data generation device 1.

[0052] The control unit 10 includes an information display control unit 101 , a playback display control unit 102 , an input accepting unit 103 , a data output control unit 104 , and a data acquiring unit 105 .

[0053] The information display control unit 101 controls the display of task information associated with each of the plurality of dynamic image data capturing production tasks. The "plural dynamic image data capturing production tasks" herein refers to image data segmented into dynamic images for each task. Hereinafter, to distinguish between dynamic image data in a broader sense, dynamic image data (in a narrow sense) segmented into dynamic images for each task will be referred to as dynamic image data D.

[0054] The playback display control unit 102 plays back and displays a plurality of moving image data D in which production work is captured.

[0055] The input accepting unit 103 accepts input of task information associated with the moving image data D from a user (user).

[0056] The data output control unit 104 outputs as output data the information on the association between the moving image data D and the task information received by the input receiving unit 103. The "information on the association between the moving image data D and the task information" referred to here is annotation data.

[0057] The data acquisition unit 105 acquires various data via the communication unit 13 .

[0058] Hereinafter, the information display control unit 101 , the playback display control unit 102 , the input accepting unit 103 , the data output control unit 104 , and the data acquiring unit 105 will be described in detail.

[0059] <Information Display Control Unit>

[0060] The information display control unit 101 may also perform the following control, namely, displaying the association between the dynamic image data D and the task information whose input has been accepted by the input acceptance unit 103 in a display format different from the display of the association between the dynamic image data D and the task information whose input has not been accepted by the input acceptance unit 103.

[0061] Specifically, for example, the information display control unit 101 may also perform control to display information identifying the dynamic image data D and the associated task information in a list format (see FIG. Figure 6 As information for identifying the dynamic image data D, for example, the start time and end time of the dynamic image data D may be used. The start time and end time are times counted from the beginning of the recorded data constituting one file.

[0062] Furthermore, in this case, the information display control unit 101 may also perform control such that the display of the association between the dynamic image data D and the task information, for which the confirmation acceptance unit 1032 of the input acceptance unit 103 has accepted the input of the instruction indicating the confirmation, is displayed in at least one of a background color, a character color, and a font format different from the display of the association between the dynamic image data D and the task information for which the confirmation acceptance unit 1032 has not accepted the input of the instruction indicating the confirmation (see FIG. Figure 6 The confirmed list column 611A and the unconfirmed list column 611B in the list image 61 are shown.

[0063] Alternatively, the information display control unit 101 may control the display in a time graph format in which a color corresponding to the type of task information associated with the moving image data D is displayed in a time region indicating the time when the moving image data D was captured (see Figure 6In this case, the information display control unit 101 may also perform the following control, that is, the color of the time area in which the confirmation acceptance unit 1032 of the input acceptance unit 103 receives the instruction input of the confirmation intention is different from the color of the time area in which the confirmation acceptance unit 1032 does not receive the instruction input of the confirmation intention (refer to Figure 9 The confirmed time area 68A and the unconfirmed time area 68B in the time map image 68' are shown in FIG.

[0064] The information display control unit 101 may perform both control of displaying in a list format and control of displaying in a time graph format, or may perform only control of displaying in a list format or only control of displaying in a time graph format.

[0065] <Playback Display Control Unit>

[0066] The playback display control unit 102 may change the playback display mode of the dynamic image data D during playback display of the dynamic image data D according to the type of task information associated with the dynamic image data D being played back and displayed.

[0067] Specifically, for example, the playback display control unit 102 may display an identification image indicating the type of the associated task information in at least one of at least a portion of the playback area of ​​the playback display dynamic image data D and at least a portion of the area surrounding the playback area (see FIG. Figure 6 Recognition image 65, Figure 8 Identification images 65A to 65C).

[0068] Furthermore, at this time, the playback display control unit 102 may display a frame image surrounding the playback area as an identification image, and set the color of the frame image to a color corresponding to the type of the associated task information (see Figure 6 66).

[0069] Furthermore, the playback display control unit 102 may also change the playback speed of the moving image data D according to the type of the associated task information (see Figure 5 The playback display control unit 102 may also change the playback speed based only on the type of task information associated with the moving image data D being played back and displayed.

[0070] Furthermore, the playback display control unit 102 may reduce the playback speed during at least a portion of the period from a predetermined time before to a predetermined time after the switching timing of the playback display of the dynamic image data D (see Figure 10 playback speed table).

[0071] <Input Acceptance Department>

[0072] The input receiving unit 103 receives an instruction input indicating that the association between the dynamic image data D and the task information has been confirmed. The input receiving unit 103 includes an edit receiving unit 1031 for receiving input and modification of the task information associated with the dynamic image data D, and a confirmation receiving unit 1032 for receiving an instruction input indicating that the association between the dynamic image data D and the task information has been confirmed.

[0073] At this time, the information display control unit 101 performs the following control, that is, the display of the association between the dynamic image data D and the task information, for which the confirmation acceptance unit 1032 has accepted the input of the instruction of the confirmed intention, is displayed in a display format different from the display of the association between the dynamic image data D and the task information, for which the confirmation acceptance unit 1032 has not yet accepted the input of the instruction of the confirmed intention.

[0074] The confirmation accepting unit 1032 may accept the confirmation instruction input by, for example, performing the following operation, that is, the list image 61 (see FIG. 6 ) attached to the list image 61 (see FIG. 6 ) which displays the information specifying the dynamic image data D and the associated task information in a list format. Figure 6 ) in the check box 613. Furthermore, the confirmation receiving unit 1032 may also confirm that the list image 61 (refer to Figure 6 ) in the task information field 612, it is determined that the confirmation instruction input has been accepted. That is, the confirmation acceptance unit 1032 may also regard the confirmation as having been accepted by the editing acceptance unit 1031 when the input / change is accepted.

[0075] <Data output control unit>

[0076] The data output control unit 104 may output as output data (i.e., annotation data) only information relating to the dynamic image data D and the task information for which the confirmation receiving unit 1032 has received an instruction indicating confirmation. The data output control unit 104 may not output (save) the annotation data unless all of the read dynamic image data D has been confirmed.

[0077] <Data Acquisition Department>

[0078] The data acquisition unit 105 may also acquire annotation data #2 (first output data) output by AI model #1 (first dynamic image analysis model), which associates task information with each of the plurality of dynamic image data D. In this case, the information display control unit 101 may also perform display control based on annotation data #2.

[0079] [Operation of the data generating device]

[0080] use Figures 5 to 10 Here, an example of the operation (data generation method) of the data generation device 1 is described. Figures 5 to 10 To illustrate the above-mentioned implementation of a series of operations for forming a workpiece including multiple cycles as video data (file) of a production operation. Figure 2 The status of P5.

[0081] Figure 5 1 is a diagram showing an example of the content of the task list table stored in the storage unit 14 . Figure 6 This is a diagram showing an example of a screen display for manually correcting annotation data # 2 displayed on the display unit 12 of the data generating device 1 . Figure 7 This is a diagram showing another example of making the playback display mode of the dynamic image data D different. Figure 8 This is a diagram showing a portion of the time chart image 68 in an enlarged manner. Figure 9 This is a diagram showing an enlarged portion of another time chart image 68 having a different display form. Figure 10 1 is a diagram showing an example of the content of the playback speed table stored in the storage unit 14 .

[0082] like Figure 5 As shown, task information includes "Gripping During Processing," "Workpiece Reset," "Workpiece Setup," "Forming Start," "Appearance Work," "Other Work in Progress," and "Abnormal Movement." Each segmented piece of dynamic image data D is labeled with the corresponding item. Furthermore, each piece of task information is assigned a color and playback speed. The color assigned to each piece of task information is unique, allowing the task information to be uniquely identified. For example, "Gripping During Processing" is assigned "First Color" and "Playback Speed ​​1x."

[0083] The playback speed can also be set as a multiple of the normal playback speed. Figure 5 In this example, a playback speed faster than the normal playback speed is set for "Other Work in Progress," "Workpiece Reset," and "Workpiece Setup," which have relatively long display times for the moving image data D. A playback speed slower than the normal playback speed is set for "Forming Start," which has a short display time for the moving image data D. A playback speed of 1 times the normal playback speed is set for moving image data D that falls between those with relatively long and short display times.

[0084] like Figure 6 As shown, the file name of the read recorded data is displayed in area 121. The read recorded data is automatically annotated and includes annotation data #2.

[0085] In area 122, a list image 61 of the read video data is displayed by the information display control unit 101. The list image 61 displays information that specifies the dynamic image data D and the associated task information in a list format. In the list image 61, the horizontally extending list column 611 corresponds to the display of the association between the dynamic image data D and the task information. The start time and end time are information that specifies the dynamic image data D, and the content of the task information column 612 is the task information. Figure 6 In the example of , information on the display time of the dynamic image data D is also displayed.

[0086] For example, the task information of the moving image data D in the uppermost list box 611 in the list image 61 , which is specified by the start time “00:00:13,913” and the end time “00:00:16,113” and the display time “00:00:02,200”, is “gripping in process”.

[0087] Furthermore, in the list image 61 , the display formats of the confirmed list field 611A for which the confirmation accepting unit 1032 has accepted the confirmation instruction input and the unconfirmed list field 611B for which the confirmation accepting unit 1032 has not accepted the confirmation instruction input are different.

[0088] Figure 6 In the example shown in FIG. 6 , a check box 613 is provided in the list image 61. A check mark indicating unconfirmed is added to the check box 613 of the unconfirmed list column 611B for the unconfirmed moving image data D among the moving image data D created using automatic annotation. This check mark is added to all moving image data D created using automatic annotation.

[0089] Furthermore, the background color of the unchecked list column 611B is displayed differently from the background color of the unchecked checked list column 611A. As mentioned above, in addition to the background color, the character color and font format may also be different.

[0090] The confirmation acceptance unit 1032 can input a confirmation instruction by, for example, clicking a check mark in the checkbox 613. If the task information associated with the moving image data D in the unconfirmed list column 611B is appropriate, the user unchecks the check mark in the checkbox 613. This causes the background color to switch to the background color of the unchecked confirmed list column 611A.

[0091] On the other hand, if the task information associated with the moving image data D in the unchecked list column 611B is inappropriate, the user uses the edit accepting unit 1031 to input or change the task information in the task information column 612 in the list image 61. As a result, the check mark in the check box 613 is removed, and the background color switches to the background color of the unchecked checked list column 611A.

[0092] The user replays the moving image data D and observes the replayed moving image 62 to confirm whether the moving image data D and the associated task information are appropriate.

[0093] In an area (reproduction area) 123 adjacent to the area (reproduction area) 122 , the dynamic image 62 of the dynamic image data D in the column 611 selected in the list image 61 is displayed by the reproduction display control unit 102 . Figure 6 In the example shown in FIG. 6 , the dynamic image data D of the task information "Other work in progress" specified by the start time "00:00:22,512" and the end time "00:00:39,412" in the selected list column 611C in the list image 61 is being played back. The user selects column 611 of the dynamic image data D to be confirmed in the list image 61 and clicks the playback button B1, thereby starting the playback display of the dynamic image 62.

[0094] The playback display form of the moving image 62 to be played back and displayed differs depending on the type of task information associated with the moving image data D. Figure 6 In the example shown in FIG. 1 , a frame image 66 in the color set for the task information is displayed in area 124 surrounding area 123 as an identification image 65 corresponding to the type of task information. Frame image 66 is displayed so as to surround area 123. Since the task information associated with the dynamic image data D currently being played back is "Other work in progress," the color of frame image 66 is the sixth color.

[0095] As described above, the identification image 65 may be displayed on at least a portion of the region 124 surrounding the region 123 , may be displayed on at least a portion of the region 123 , or may be displayed on both the region 123 and the region 124 .

[0096] For example, you can also Figure 7 As shown in the symbol 701, the identification image 65A is displayed by overlapping the playback display at the four corners of the area 123. Figure 7 As shown in reference numeral 702, a linear identification image 65B is displayed in the left and right end regions of the region 123 where no image is displayed. The display colors of the identification images 65A and 65B are the colors set in the task information.

[0097] Moreover, it is also possible to Figure 7 As shown in reference numeral 701, an identification image 65C including text of task information is displayed in a light color that does not impair the visibility of the dynamic image being played back in area 123. In this case, the display color of the identification image 65C may be set to the color set for the task information.

[0098] As described above, the playback speed of the moving image 62 being played back and displayed may also vary depending on the type of task information associated with the moving image data D. Figure 6 In the example of , the task information associated with the dynamic image data D being played back and displayed is "other work in progress", so the playback speed is 2x speed.

[0099] Furthermore, in an area 125 above the area 123 , task information associated with the moving image 62 being reproduced and displayed is displayed as a text image 63 .

[0100] Furthermore, a slide bar 64 indicating the playback position of the moving image 62 being played back and displayed in the area 123 is displayed in the area 126 below the area 123. By sliding the slide bar 64 left and right, the moving image 62 being played back and displayed can be moved forward or backward.

[0101] Furthermore, in the area 127 below the area 126, the time chart image 68 of the read video data is displayed by the information display control unit 101. Figure 8 As shown, the time map image 68 displays a color corresponding to the type of task information associated with the dynamic image data D in the time area representing the time when the dynamic image data D was captured. Figure 6 As shown, in the time chart image 68 , the task information of 19 cycles of recorded data is displayed in color.

[0102] Moreover, it can also replace Figure 8 The time diagram shown in Figure 68 will be Figure 9 The time diagram image 68' is shown in area 126. Figure 9 As shown, in the time map image 68', the display color of the confirmed time area 68A, for which the confirmation receiving unit 1032 has received an instruction indicating confirmation, is different from the display color of the unconfirmed time area 68B, for which no confirmation instruction has been received. In this case, the colors of the confirmed time area 68A and the unconfirmed time area 68B are different based on the color set in the task information, for example, by changing the saturation or brightness.

[0103] like Figure 6As shown, the time chart image 68 may also include a time axis 68c at the bottom. By sliding the marker 68d on the time axis 68c left or right, the dynamic image data D to be played back and displayed in the area 123 can be selected at the stop position. Similar to the case of selecting the bar 611 in the list image 61, the playback display control unit 102 plays back and displays the dynamic image data D at the time corresponding to the stop position of the marker 68b.

[0104] When all list boxes 611 in list image 61 have been marked as confirmed boxes 611A, the user clicks Save button B2 to save the manually corrected annotation data #2. The data output control unit 104 may gray out Save button B2 and disable it until all list boxes 611 in list image 61 have been marked as confirmed boxes 611A.

[0105] Alternatively, when the save button B2 is clicked, the data output control unit 104 may output only the information relating the moving image data D and the task information received by the confirmation receiving unit 1032 as output data (ie, annotation data).

[0106] Furthermore, as described above, when the playback speed is controlled to be reduced at the switching timing of the playback display of the dynamic image data D, the storage unit 14 can be configured to include a separate Figure 10 The playback speed table shown is sufficient.

[0107] The playback display control unit 102 sets the playback speed when playing back the moving image based on the playback speed table. Figure 10 In the playback speed table, the specified period for switching playback display is set to a playback speed slower than the normal playback speed, for example, 0.5 times. Furthermore, for dynamic image data D whose display time t is longer than the specified value T, the playback speed is set to be faster than the normal playback speed, for example, twice. For dynamic image data D whose display time t is shorter than the specified value T, the playback speed is set to be slower than the normal playback speed, for example, 0.5 times. For dynamic image data D whose display time t is between T1 and T2, the playback speed is set to be 1 times the normal playback speed.

[0108] Furthermore, the above description describes a case where the user data generation device 1 manually corrects 19 cycles of annotation data #2 generated by the motion image analysis AI of the cloud server 21. However, the manually corrected annotation data is not limited to that generated by the motion image analysis AI; it may also include annotation data with low-precision annotations.

[0109] Furthermore, although the data collection system 50 including the cloud platform 2 is employed, a data collection system may be configured without the cloud platform 2. That is, a dynamic image analysis AI included in a local cloud server (not shown) may be used instead of the cloud server 21.

[0110] (Effect)

[0111] According to the above structure, the association between dynamic image data D and task information can be displayed, and the user can input task information. Thus, the user can replay the dynamic image data to confirm the content and input task information. Furthermore, the dynamic image data D is displayed in a replay display format corresponding to the type of task information associated with it, allowing the user to identify the type of associated task information simply by observing the dynamic image data D. In other words, the user can focus on viewing the replay display of the dynamic image data D while confirming the appropriateness of the task information. This allows for efficient and accurate generation of the learning data required for creating a dynamic image analysis model.

[0112] Furthermore, the data acquisition unit 105 may acquire annotation data #2 (first output data) output by the AI ​​model #1, and the information display control unit 101 may perform display control based on the annotation data #2.

[0113] With this configuration, the user can confirm the appropriateness of the association data between the moving image data D output by AI model #1 and the task information, and create learning data by modifying the association as needed. This allows for more efficient generation of learning data.

[0114] Furthermore, the playback display control unit 102 may display an identification image 65 indicating the type of associated task information in at least one of at least a portion of the area 123 where the moving image data D is played back and displayed and at least a portion of an area surrounding the playback area.

[0115] With this configuration, the user can observe the identification image 65 and grasp the task information associated with the moving image data D without looking away from the moving image 62 reproducing the moving image data D. Therefore, the user can efficiently confirm the appropriateness of the associated data between the moving image data D and the task information.

[0116] Furthermore, the playback display control unit 102 may display the frame image 66 surrounding the moving image 62 as the identification image 65 in a color corresponding to the type of task information.

[0117] According to the above configuration, the user can more easily confirm the appropriateness of the associated data between the moving image data D and the task information by confirming the color of the frame image 66 displayed around the moving image 62 .

[0118] Furthermore, the playback display control unit 102 may change the playback speed of the moving image data D according to the type of the associated task information.

[0119] According to the above configuration, for example, in the case of task information that requires attention or ends quickly, the playback speed can be slowed down. Therefore, the user can more accurately confirm the appropriateness of the associated data between the moving image data D and the task information.

[0120] The playback display control unit 102 may reduce the playback speed during at least a portion of a period from a predetermined time before to a predetermined time after the timing of switching the playback display of the moving image data D.

[0121] At the timing of switching the playback display of the dynamic image data D, the job content changes, so the likelihood of obtaining important information is particularly high. Here, according to the above configuration, the playback speed is reduced near the timing of switching the playback display of the dynamic image data D, allowing the status of the job content changes to be confirmed in more detail. As a result, the user can more accurately confirm the appropriateness of the association data between the dynamic image data D and the task information.

[0122] [Example of implementation using software]

[0123] The functions of the data generating device 1 (hereinafter referred to as "device") can be realized by a program for causing a computer to function as the device and a program for causing a computer to function as each control block of the device (especially each unit included in the control unit 10).

[0124] In this case, the apparatus includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The program is executed by the control device and the storage device, thereby realizing the functions described in the various embodiments.

[0125] The program may also be recorded on one or more non-transitory computer-readable recording media. The recording medium may or may not be included in the apparatus. In the latter case, the program may be supplied to the apparatus via any transmission medium, whether wired or wireless.

[0126] Furthermore, some or all of the functions of each control block may be implemented using logic circuits. For example, integrated circuits incorporating logic circuits that function as each control block are also within the scope of the present invention. Furthermore, the functions of each control block may be implemented using, for example, a quantum computer.

[0127] 〔Summarize〕

[0128] The data generating device of form 1 of the present invention includes: an information display control unit, which performs the following control, i.e., associates the operation content information with each of a plurality of dynamic image data captured with the production operation and displays it; a replay display control unit, which replays and displays the dynamic image data; an input acceptance unit, which accepts the user's input of the operation content information associated with the dynamic image data; and a data output control unit, which outputs information on the association between the dynamic image data input accepted by the input acceptance unit and the operation content information as output data, wherein the replay display control unit changes the replay display form of the dynamic image data according to the type of the operation content information associated with the dynamic image data being replayed and displayed.

[0129] According to the above structure, the association between dynamic image data and work content information can be displayed, and the user can input the work content information. Therefore, the user can replay the dynamic image data to confirm the content and input the work content information. Moreover, the dynamic image data is displayed in a replay display format corresponding to the type of work content information associated with it, so the user can identify the type of work content information associated with it by simply observing the dynamic image data. In other words, the user can focus on viewing the replay display of the dynamic image data while confirming the appropriateness of the work content information. As a result, the learning data required for creating a dynamic image analysis model can be generated efficiently and accurately.

[0130] The data generating device of form 2 of the present invention is based on the data generating device of form 1, and further includes: a data acquisition unit, which acquires first output data output by a first dynamic image analysis model, wherein the first dynamic image analysis model associates the operation content information with each of the multiple dynamic image data, and the information display control unit performs display control based on the first output data.

[0131] With this configuration, the user can confirm the appropriateness of the association data between the moving image data output by the first moving image analysis model and the work content information, and create learning data by modifying the associated work as needed. This allows for more efficient generation of learning data.

[0132] The data generating device of form 3 of the present invention is based on the data generating device of form 1 or form 2, wherein the playback display control unit displays an identification image representing the type of the associated work content information in at least a part of the playback area for replaying and displaying the dynamic image data, and in at least one of at least a part of the area around the playback area.

[0133] According to the above configuration, the user can observe the identification image displayed on at least one of the playback area of ​​the moving image data and the area surrounding the playback area, thereby efficiently confirming the appropriateness of the association data between the moving image data and the work content information.

[0134] The data generating device of form 4 of the present invention is based on the data generating device of form 3, wherein the playback display control unit displays a frame image surrounding the playback area as the identification image, and sets the color of the frame image to a color corresponding to the type of the associated job content information.

[0135] According to the above configuration, the user can more easily check the appropriateness of the associated data between the moving image data and the work content information by checking the color of the frame area displayed around the playback area of ​​the moving image data.

[0136] A data generating device according to aspect 5 of the present invention is the data generating device according to any one of aspects 1 to 4, wherein the playback display control unit changes the playback speed of the moving image data according to the type of the associated work content information.

[0137] With this configuration, the playback speed is changed according to the type of work content information. For example, for work content information that requires attention or is completed quickly, the playback speed can be slowed down. This allows users to more accurately confirm the appropriateness of the associated data between the moving image data and the work content information.

[0138] The data generating device of form 6 of the present invention is a data generating device according to any one of forms 1 to 5, wherein the playback display control unit reduces the playback speed during at least a portion of the period from a specified time before to a specified time after the switching timing of the playback display of the dynamic image data.

[0139] Since the work content changes at the time when the replay display of the moving image data switches, the likelihood of obtaining important information is particularly high. According to the above configuration, the playback speed is reduced near the time when the replay display of the moving image data switches, allowing the status of the work content changes to be confirmed in more detail. Consequently, the user can more accurately confirm the appropriateness of the association data between the moving image data and the work content information.

[0140] The data generation method of form 7 of the present invention includes: an information display control step, which performs the following control, namely, associating the operation content information with each of a plurality of dynamic image data captured with the production operation and displaying it; a playback display control step, which plays back and displays the dynamic image data; an input acceptance step, which accepts the user's input of the operation content information associated with the dynamic image data; and a data output control step, which outputs information on the association between the dynamic image data accepted by the input acceptance step and the operation content information as output data, wherein the playback display control step changes the playback display form of the dynamic image data according to the type of the operation content information associated with the dynamic image data being played back and displayed.

[0141] <Additional Notes>

[0142] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining technical components disclosed in different embodiments are also included in the technical scope of the present invention.

[0143] Explanation of Figure Numbers

[0144] 1: Data generation device

[0145] 2: Cloud Platform

[0146] 10: Control Department

[0147] 11: Input section

[0148] 12: Display

[0149] 14: Storage

[0150] 62: Motion Graphics

[0151] 65, 65A, 65B, 65C: Identify images

[0152] 66: Frame Image

[0153] 101: Information display control unit

[0154] 102: playback display control unit

[0155] 103: Input acceptance department

[0156] 104: Data output control unit

[0157] 105: Data Acquisition Department

[0158] 123: Area (playback area)

[0159] 124: Area (area surrounding the playback area)

Claims

1. A data generating device, comprising: an information display control unit for performing control so as to display the work content information in association with each of a plurality of moving image data capturing the production work; a playback and display control unit configured to playback and display the dynamic image data; an input receiving unit for receiving a user's input of the work content information associated with the dynamic image data; as well as The data output control unit outputs information on the association between the moving image data and the work content information, the input of which is accepted by the input acceptance unit, as output data. The playback display control unit changes the playback display mode of the moving image data according to the type of the work content information associated with the moving image data being played back and displayed.

2. The data generating device according to claim 1, further comprising: a data acquisition unit that acquires first output data output by a first dynamic image analysis model, wherein the first dynamic image analysis model associates the work content information with each of the plurality of dynamic image data; The information display control unit performs display control based on the first output data.

3. The data generating device according to claim 1, wherein the playback display control unit displays an identification image indicating the type of the associated work content information in at least one of at least a portion of a playback area in which the dynamic image data is played back and displayed and at least a portion of an area surrounding the playback area.

4. The data generating device according to claim 3, wherein the playback display control unit displays a frame image surrounding the playback area as the identification image, and sets a color of the frame image to a color corresponding to a type of the associated work content information. 5 . The data generating device according to claim 1 , wherein the playback display control unit changes a playback speed of the moving image data according to a type of the associated work content information.

6. The data generating device according to claim 1, wherein the playback display control unit reduces the playback speed during at least a portion of a period from a predetermined time before to a predetermined time after a switching timing of playback display of the moving image data.

7. A data generation method comprising: an information display control step for performing control such that the work content information is associated with each of a plurality of moving image data capturing the production work and displayed; a playback and display control step of playing back and displaying the dynamic image data; an input accepting step of accepting a user's input of the work content information associated with the dynamic image data; as well as a data output control step of outputting information on association between the dynamic image data inputted by the input acceptance step and the work content information as output data; The playback display control step changes the playback display mode of the moving image data according to the type of the work content information associated with the moving image data being played back and displayed.

Images