Information processing method in information processing device, information processing device, and information processing program

Abstract

In the information processing method, object data and a tag list including a plurality of tags for classifying the object data are displayed on a worker terminal, a selection from any one of the plurality of tags is received, and the object data is classified on the basis of the selection result of the tag. And a control unit that updates an input history in which the number of selections for each of the plurality of tags is recorded, and updates the display order of the plurality of tags included in the tag list in accordance with the number of selections recorded in the input history every time the display of the object data is switched to the next object data.

Description

Technical Field

[0001] This disclosure relates to techniques for assisting in setting annotation data for object data. Background Technology

[0002] Conventionally, a system disclosed in Patent Document 1 is known as a technique for simplifying the annotation process of setting annotation data for object data. This system acquires an image of a product with a barcode, i.e., a product image, and the time at which the product image was captured. It then determines the barcode scanned by the POS terminal at the time closest to the capture time and annotates the product identification information corresponding to that barcode onto the product image. Thus, it is possible to set the product identification information, which serves as annotation data, onto the product image, which is object data.

[0003] Thus, the system described in Patent Document 1 is a technology based on the premise that the object data contains barcodes. Therefore, the system does not envision annotation operations for object data that does not contain barcodes, and further improvements are needed to simplify the annotation process.

[0004] Prior art literature

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-033494 Summary of the Invention

[0007] This disclosure was made to address such a problem, and its purpose is to provide a technique for further simplifying the annotation process.

[0008] The information processing method in one aspect of this disclosure is an information processing method in an information processing device. In this information processing method, object data and a tag list containing multiple tags for classifying the object data are displayed on an operator's terminal. The operator receives a selection from any one of the multiple tags. Based on the selection result of the tag, the operator updates the input history that records the number of selections for each of the multiple tags. Whenever the display of the object data is switched to the next object data, the operator updates the display order of the multiple tags contained in the tag list according to the number of selections recorded in the input history.

[0009] According to this disclosure, annotation work can be further simplified. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example of an annotation work assistance system according to Embodiment 1 of this disclosure.

[0011] Figure 2This is a diagram showing an example of the input history stored in the input history storage section.

[0012] Figure 3 This is a flowchart illustrating an example of the overall processing of the annotation task assistance system according to Embodiment 1.

[0013] Figure 4 This diagram illustrates the processing of the image display order update unit according to Embodiment 1.

[0014] Figure 5 This is a diagram showing a display device displaying the first display image and label list according to Embodiment 1.

[0015] Figure 6 This is a diagram showing the first display image with an annotation area set.

[0016] Figure 7 This diagram illustrates the processing of the input history update unit when the number of times a label is selected is increased.

[0017] Figure 8 This is a diagram showing a display device displaying the second display image and label list according to Embodiment 1.

[0018] Figure 9 This diagram illustrates the processing of the input history update unit when the number of times a label is selected is increased.

[0019] Figure 10 This is a diagram showing a display device displaying the third display image and label list according to Embodiment 1.

[0020] Figure 11 This diagram illustrates the processing of the input history update unit when the number of times a label is selected is reset.

[0021] Figure 12 This is a block diagram illustrating an example of the annotation work assistance system according to Embodiment 2.

[0022] Figure 13 This is a flowchart illustrating an example of the overall processing of the annotation task assistance system according to Embodiment 2.

[0023] Figure 14 This diagram illustrates the processing of the input history update unit when the number of selections for similar tags is increased.

[0024] Figure 15 This is a block diagram illustrating an example of the annotation work assistance system according to Embodiment 3.

[0025] Figure 16This is a flowchart illustrating an example of the overall processing of the annotation task assistance system according to Embodiment 3.

[0026] Figure 17 This is a diagram showing a display device displaying the first display image and label list according to Embodiment 3.

[0027] Figure 18 This is a diagram showing a display device displaying the second display image and label list according to Embodiment 3.

[0028] Figure 19 This is a diagram used to illustrate the processing of the region similarity evaluation unit.

[0029] Figure 20 This is a diagram illustrating the processing of the input history update unit when the number of times a tag is selected for the first annotation region increases based on the number of the third annotation region.

[0030] Figure 21 This is a diagram showing a display device displaying the third display image and label list according to Embodiment 3.

[0031] Figure 22 This is a diagram showing a display device displaying the first display image and label list according to Embodiment 4. Detailed Implementation

[0032] (The process of realizing this disclosure)

[0033] Previously, annotation was sometimes performed to generate learning data used when enabling machine learning models. Annotation was performed, for example, as follows: First, object data such as image data and a list of tags containing labels for classifying the content of that object data were displayed on the display device of the terminal operated by the operator. Next, the operator selected the appropriate tag from the tag list. For example, the operator used a given input device such as a mouse to select the appropriate tag from the tag list. Then, the selected tag was assigned to the image data. The annotation process was repeated below.

[0034] However, in recent years, the number of labels in the label list has become enormous, sometimes forcing operators to select the appropriate label from, say, hundreds of labels. In such cases, finding the right label from the label list becomes a very tedious task.

[0035] This disclosure was made to address such a problem.

[0036] (1) The information processing method in one aspect of this disclosure is an information processing method in an information processing device. In the information processing method, object data and a tag list containing multiple tags for classifying the object data are displayed on the operator's terminal. The selection of any one of the multiple tags is received. Based on the selection result of the tag, the input history that records the number of selections of each of the multiple tags is updated. Whenever the display of the object data is switched to the next object data, the display order of the multiple tags contained in the tag list is updated according to the number of selections recorded in the input history.

[0037] This structure allows for updating the display order of multiple labels, making it easier for operators to select labels that are frequently chosen—those with a higher probability of being selected. This simplifies the task of finding the appropriate label from the label list, and consequently, simplifies annotation.

[0038] (2) In the information processing method described in (1) above, the object data may also be image data. Further, an annotation instruction for the image data is received, and the label is selected in order to classify the target contained in the annotation area corresponding to the annotation instruction.

[0039] Based on this structure, labels can be assigned to targets contained in the annotation regions of image data. This enables the generation of more accurate learning data.

[0040] (3) In the information processing method described in (2) above, the object data may also include: first image data and second image data displayed after the first image data. Further, a third annotation region is determined among the more than one first annotation region set in the first image data, whose regional similarity with the more than one second annotation region set in the second image data exceeds a first threshold. The update of the input history includes: increasing the number of times the selected tags for the more than one first annotation region are selected according to the number of the third annotation regions.

[0041] According to this structure, when the first and second annotation regions have high similarity, the number of times the selected label for the first annotation region is increased. Thus, since the task of selecting labels for the second annotation region is omitted, the annotation process is simplified.

[0042] (4) In the information processing method described in (2) or (3) above, the annotation indicator may be able to specify more than one annotation region for one tag, and the update of the input history includes: increasing the number of times the tag is selected based on the number of annotation regions specified for the tag.

[0043] This structure effectively increases the number of times labels can be selected for annotation regions that are likely to be frequently included in image data. Consequently, since the display order of multiple labels is updated to ensure they are displayed in easily accessible positions for operators, the task of finding the appropriate label from the label list becomes simpler. As a result, the annotation process is simplified.

[0044] (5) In any of the information processing methods described in (1) to (4) above, the update of the input history may also include: increasing the number of times the selected tag is selected and the number of times similar tags similar to the selected tag are selected.

[0045] When object data is composed of multiple image data that are similar to each other, sometimes a target similar to a target contained in one image data is included in another image data. For example, if one image data contains a bicycle, sometimes another image data contains a motorcycle. Here, according to the structure described above, if "bicycle" is selected as a set label in an annotation task for one image data, the selection frequency of the "bicycle" label increases, and the selection frequency of the similar label "motorcycle" also increases. As a result, the display order of the multiple labels is updated so that not only the "bicycle" label but also the "motorcycle" label is displayed in a position easily selectable by the operator. Therefore, if a motorcycle is included in another image data, the operator can easily select the "motorcycle" label in an annotation task for that other image data. Consequently, the annotation task is simplified.

[0046] (6) In the information processing method described in (2) above, the display order of the object data can also be determined by the order of image similarity from high to low relative to the object data just displayed.

[0047] According to this structure, the probability that the same target is continuously contained in the image data displayed sequentially on the operator's terminal increases. Therefore, the operator can continuously select the same label to specify the target during annotation work on this image data. As described above, in this disclosure, the display order of multiple labels is updated so that labels selected more frequently by the operator are easier for the operator to select. Therefore, the same labels are continuously displayed in positions easily selectable by the operator. As a result, the annotation work is simplified.

[0048] (7) In the information processing method described in (6) above, it is also possible to further reset the number of selections recorded in the input history when the image similarity is below the second threshold.

[0049] When the image similarity between the image data just displayed on the operator's terminal (hereinafter, the previous image data) and the image data displayed on the operator's terminal after the previous image data (hereinafter, the next image data) is below a second threshold, that is, when the image similarity between the two image data is low, the probability that the two image data consecutively contain the same target is low. In this case, the possibility that the label selected by the operator in the annotation work for image data after the next image data is different from the label selected for the previous image data is high. Therefore, in this case, when performing the annotation work for image data after the next image data, the continued display of the label selected in the annotation work for the previous image data in a position easily selected by the operator will not hinder the annotation work for image data after the next image data. Here, according to the above structure, when the image similarity is below the second threshold, the number of selections recorded in the input history is reset. Thus, the continued display of the label selected in the annotation work for the previous image data in a position easily selected by the operator is suppressed when performing the annotation work for image data after the next image data. Thus, the annotation work for image data after the next image data can be simplified.

[0050] (8) In the information processing method described in (2) above, the annotation area may also be a polygon surrounding the target or the outline of the target.

[0051] This structure makes specifying annotation areas easy.

[0052] (9) In any of the information processing methods described in (1) to (8) above, the tag may selectively receive input containing voice data or text data.

[0053] According to this structure, since labels can be selected via voice or text data, operators do not need to manually select appropriate labels. As a result, annotation work becomes simpler.

[0054] (10) In the information processing method described in (9) above, it is also possible to further reset the number of selections recorded in the input history when the selection of the tag based on the voice data or the text data is received.

[0055] When tags are selected using voice or text data, they are more likely to be displayed in positions that are difficult for operators to choose. In other words, the current tag display order is more likely to be inappropriate. Therefore, based on the structure described above, when tag selections are received via voice or text data, the tag selection count is reset. That is, the tag display order is reset once. As a result, since the tag display order is easily updated, it is easier to update the tag display order to the appropriate order. Consequently, the annotation process becomes simpler.

[0056] (11) In the information processing method described in (2) above, the annotation indication may also be an indication that specifies the location of the target by a point. Further, based on the location of the point, the target is extracted from the image data and the annotation area is set for the target.

[0057] According to this structure, the annotation process is made simpler because operators can define the annotation area by specifying the location of the target using points.

[0058] (12) The information processing apparatus in another aspect of the present disclosure includes: a display control unit that displays object data and a tag list containing multiple tags for classifying the object data on an operator terminal; a receiving unit that receives a selection from any one of the multiple tags; a history update unit that updates an input history that records the number of selections for each of the multiple tags based on the selection result of the tag; and a sequence update unit that updates the display order of the multiple tags contained in the tag list according to the number of selections recorded in the input history whenever the display of the object data is switched to the next object data.

[0059] Based on this structure, an information processing apparatus for performing the above-described information processing method can be provided.

[0060] (13) In another aspect of this disclosure, the information processing program causes the processor to perform the following processing: displaying object data and a tag list containing multiple tags for classifying the object data on an operator terminal; receiving a selection from any of the multiple tags; updating an input history that records the number of selections for each of the multiple tags based on the selection result of the tag; and updating the display order of the multiple tags contained in the tag list according to the number of selections recorded in the input history whenever the display of the object data is switched to the next object data.

[0061] Based on this structure, an information processing program that performs the above-described information processing method can be provided.

[0062] This disclosure can also be implemented as an information processing system operating according to such an information processing procedure. Furthermore, it is of course possible to distribute such a computer program via computer-readable non-transitory recording media such as CD-ROMs or communication networks such as the Internet.

[0063] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, constituent elements in the following embodiments that are not described as constituent elements of independent technical solutions representing the highest-level concept are described as arbitrary constituent elements. Furthermore, various contents can be combined in all embodiments.

[0064] (Implementation Method 1)

[0065] Figure 1 This is a block diagram illustrating an example of an annotation task assistance system 1A according to Embodiment 1 of this disclosure. The annotation task assistance system 1A is a system that assists in annotation tasks. An annotation task refers to the task of setting annotation data for object data. Object data may be, for example, data related to images, animations, sounds, or articles. Annotation data is information appended to the object data. For example, an annotation task is performed to generate learning data used when enabling a learning model to perform machine learning. Hereinafter, the object data will be image data (IM). Figure 5 An example will be given. Furthermore, in Implementation 1, an example will be given where the annotation data is a label used to classify targets contained in the object data.

[0066] like Figure 1As shown, the annotation work assistance system 1A mainly includes a worker terminal 10, a network NW, and a server 20 (an example of an information processing device). The worker terminal 10 and the server 20 are configured to communicate with each other via the network NW. The network NW is, for example, a wide area communication network that includes an Internet communication network and a mobile telephone communication network.

[0067] The operator terminal 10 is composed of information terminals such as personal computers, smartphones, and tablet computers. The operator terminal 10 is operated by the operator performing the annotation work.

[0068] The operator terminal 10 includes: an operation unit 101, a display device 102, a control unit 103, and a terminal communication unit 104. The operation unit 101 is, for example, an operating device such as a keyboard and mouse. The display device 102 is, for example, a monitor such as an LCD panel. The control unit 103 is, for example, a CPU or other circuitry responsible for the overall control of the operator terminal 10. The terminal communication unit 104 is a communication circuit that connects the operator terminal 10 to a network NW.

[0069] Server 20 is, for example, a cloud server consisting of one or more computers. Server 20 includes memory 210, processor 220, and server communication unit 240.

[0070] The memory 210 is a storage device capable of storing various types of information, such as RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. The memory 210 includes an object data storage unit 211 and an input history storage unit 212.

[0071] The object data storage unit 211 is a database that stores image data IMs of objects used in annotation jobs. Specifically, in the object data storage unit 211 according to Embodiment 1, an image dataset IMS composed of multiple image data IMs is stored. Figure 4 ).

[0072] Input history is stored in the input history storage unit 212. Figure 2 This is an example of the input history stored in the input history storage unit 212. For example... Figure 2 As shown, the input history records the number of times each tag was selected. For example, in Figure 2 The example shown records that the label "vehicle" was selected 6 times and the label "bicycle" was selected 4 times in the annotation work so far.

[0073] return Figure 1The processor 220 is, for example, a CPU or other circuit. The processor 220 functions by executing a program pre-stored in the memory 210, and includes an image similarity evaluation unit 221, an image display order update unit 222, a display control unit 223, an acquisition unit 224, an input history update unit 225, a list display order update unit 226, and a determination unit 227.

[0074] The image similarity evaluation unit 221 acquires the image dataset IMS stored in the object data storage unit 211 of the memory 210 and evaluates the image similarity between the image data IM constituting the image dataset IMS.

[0075] The image similarity evaluation unit 221 evaluates image similarity by calculating the error between the feature quantities of image data IMs using methods such as absolute error, L2 error, and Cos distance. The image similarity evaluation unit 221 uses, for example, the RGB values ​​of each pixel assigned to each image data IM as feature quantities. Alternatively, the image similarity evaluation unit 221 may use brightness values ​​obtained by converting the RGB values, or edge features obtained by performing edge detection processing on each image data IM as the aforementioned feature quantities. Furthermore, the image similarity evaluation method is not limited to the above examples. For example, the image similarity evaluation unit 221 may input each image data IM into a pre-machined neural network. The image similarity evaluation unit 221 may also acquire multidimensional vectors of each image data IM output from the neural network and evaluate the image similarity between image data IMs based on the vector error obtained by comparing these multidimensional vectors.

[0076] The image display order update unit 222 determines the display order of each image data IM when the display device 102 of the operator terminal 10 displays the image data IM based on the image similarity between each other evaluated by the image similarity evaluation unit 221.

[0077] The display control unit 223 sends instructions to the operator terminal 10 via the server communication unit 240 to display image data IM and a tag list LL containing multiple tags on the display device 102. The display control unit 223 may also, as needed, send information such as the programs required to display the image data IM and the tag list LL along with the aforementioned instructions to the operator terminal 10.

[0078] The acquisition unit 224 acquires tag selection result information and annotation instruction information sent from the operator terminal 10 via the terminal communication unit 104 and the server communication unit 240. The tag selection result information indicates which tag the operator selected from the multiple tags included in the tag list LL during the current annotation operation. The annotation instruction information indicates which area in the image data IM the operator designated as the annotation area AR during the current annotation operation. Figure 6 (information).

[0079] The determination unit 227 determines whether the annotation work has been completed. In addition, the determination unit 227 determines whether the image similarity between the image data IM displayed on the display device 102 in the previous annotation work and the image data IM displayed on the display device 102 in the current annotation work is equal to a predetermined image similarity threshold (an example of the second threshold).

[0080] The input history update unit 225 updates the input history of the tags selected by the operator in the current annotation operation by referring to the tag selection result information obtained by the acquisition unit 224. Specifically, it increases the number of times the tags selected in the current annotation operation are used. In addition, the input history update unit 225 increases the number of times a tag is selected by referring to the annotation instruction information obtained by the acquisition unit 224, based on the number of annotation regions AR set for a tag. Furthermore, if the determination unit 227 determines that the image similarity between the image data IM displayed in the previous annotation operation and the image data IM displayed in the current annotation operation is below the image similarity threshold, the input history update unit 225 resets the number of times each tag is selected in the input history storage unit 212.

[0081] The list display order update unit 226 updates the display order of multiple tags in the tag list LL based on the number of times each tag has been selected. Specifically, the list display order update unit 226 rearranges the display order of multiple tags, for example, in descending order, based on the number of times each tag has been selected. This places tags with more selections, i.e., tags more likely to be selected by the operator, at the top of the tag list LL. As a result, redundant operations can be suppressed when the operator selects tags. "Redundant operations" refer, for example, to operations such as drop-down operations for displaying tags located at the bottom of the tag list LL, or operations such as scrolling the tag list to the bottom of the tag list LL. However, the method of rearranging the multiple tags by the list display order update unit 226 is not limited to this. As long as redundant operations as described above can be reduced, the list display order update unit 226 can also rearrange the display order of multiple tags, for example, in ascending order, based on the number of times each tag has been selected. That is, the list display order update unit 226 updates the display order of the tags in the tag list LL to avoid generating redundant operations.

[0082] Server communication unit 240 is a communication circuit used to connect server 20 to network NW.

[0083] Reference Figure 3 The flowchart below illustrates the processing of the annotation task assistance system 1A as described above. Figure 3 This is a flowchart illustrating an example of the overall processing of the annotation task assistance system 1A according to Embodiment 1.

[0084] In step S101, the image similarity evaluation unit 221 evaluates the image similarity between the image data IM.

[0085] In step S102, the image display order update unit 222 determines the display order of the image data IM in the display device 102 based on the image similarity evaluated by the image similarity evaluation unit 221. Hereinafter, the processing of the image display order update unit 222 according to Embodiment 1 will be described with reference to the following diagram. Figure 4 The processing of the image display order update unit 222 will be explained below. Furthermore, the image initially displayed in the display device 102 will be referred to as the first display image IM1, and the second, third, fourth, and fifth images displayed in the display device 102 will be referred to as the second display image IM2, the third display image IM3, the fourth display image IM4, and the fifth display image IM5, respectively.

[0086] In step S102, the image display order update unit 222 first evaluates the image data IMs with the highest image similarity among all image data IMs included in the image dataset IMS, as evaluated by the image similarity evaluation unit 221, and sets them as the first display image IM1 and the second display image IM2. Next, among the remaining image data IMs excluding the first display image IM1, the image display order update unit 222 sets the image data IM with the highest image similarity to the second display image IM2 as the third display image IM3. Similarly, among the remaining image data IMs excluding the first display image IM1 and the second display image IM2, the image display order update unit 222 sets the image data IM with the highest image similarity to the third display image IM3 as the fourth display image IM4. The same process is repeated by the image display order update unit 222 until the display order of all image data IMs is determined.

[0087] For example, such as Figure 4 As shown, consider the case where the image dataset IMS contains images A, B, C, D, and E. Furthermore, consider the following scenario: the image similarity evaluation unit 221 evaluates that among all the image data IM contained in the image dataset IMS, the image similarity between image A and image B is the highest. Further, consider the following scenario: the image similarity evaluation unit 221 evaluates that the image data IM most similar to image B besides image A is image E, and the image data IM most similar to image E besides images A and B is image C. In this case, as... Figure 4 As shown, the image display order update unit 222 sets image A as the first display image IM1, image B as the second display image IM2, image E as the third display image IM3, image C as the fourth display image IM4, and image D as the fifth display image IM5. In this way, the image display order update unit 222 rearranges the image dataset IMS so that the image data IM with the highest similarity to the image data IM previously displayed by the display device 102 becomes the next image data IM displayed by the display device 102.

[0088] return Figure 3 In step S103, the display control unit 223 causes the display device 102 to display image data IM and tag list LL. For example, the display control unit 223 will... Figure 5 The image data IM showing three figures riding bicycles is displayed as the first display image IM1.

[0089] Next, in step S104, the control unit 103 of the operator terminal 10 receives the tag selection. That is, in step S104, the operator performs the tag selection operation. Figure 5As shown, the operator selected the "bicycle" tag. In this case, the control unit 103 acquires tag selection result information indicating that the "bicycle" tag has been selected. The tag selection result information acquired by the control unit 103 is acquired by the acquisition unit 224 via the terminal communication unit 104 and the server communication unit 240.

[0090] Next, in step S105, the control unit 103 receives an annotation instruction for the image data IM from the operator. Based on the received annotation instruction, the control unit 103 sets an annotation area AR corresponding to the annotation instruction for the image data IM. For example, based on the annotation instruction, the control unit 103 sets an annotation area AR for the first displayed image IM1. Figure 6 The annotation area AR is shown. Figure 6 This is a diagram showing the first display image IM1 with the annotation area AR set. Figure 6 The regions inside the quadrilateral frames FL that surround the three bicycles shown are annotation regions AR. Such annotation regions AR are set, for example, by the operator specifying the upper left and lower right vertices of the annotation region AR in the annotation instructions. Specifically, the operator operates the mouse, which serves as the operation unit 101, to move the mouse cursor MC (… Figure 6 The operator aligns the mouse cursor with the top-left vertex of the annotation region AR. Furthermore, while the mouse is clicked (selected), the operator drags the mouse downwards and to the right, and releases the click to specify the bottom-right vertex of the annotation region AR. Alternatively, the annotation region AR can be defined using polygons other than quadrilaterals, such as triangles or pentagons. Furthermore, the annotation region AR can also be defined using circles or ellipses. Additionally, the annotation region AR can be defined by specifying the outline of the target contained within it. In this case, the operator simply moves the mouse cursor MC along the outline of the target. The annotation instruction information acquired by the control unit 103 is acquired by the acquisition unit 224 via the terminal communication unit 104 and the server communication unit 240.

[0091] Next, in step S106, the determination unit 227 determines whether all annotation work has been completed. Specifically, the determination unit 227 determines whether there are any remaining image data IMs in the image data IMs contained in the image dataset IMS that have not been annotated. If all annotation work has been completed (if yes in step S106), the processing of the annotation work assistance system 1A ends. If not all annotation work has been completed (if no in step S106), the process proceeds to step S107.

[0092] In step S107, the determination unit 227 determines whether the image similarity between the image data IM displayed on the device 102 in the previous annotation operation and the image data IM displayed on the display device 102 in the current annotation operation is below a predetermined image similarity threshold. If the image similarity is below the image similarity threshold (if yes in step S107), the process proceeds to step S108. If the image similarity exceeds the image similarity threshold (if no in step S107), the process proceeds to step S109. Furthermore, if there is no comparison object, such as the first displayed image IM1, the process proceeds to step S109. The explanation of step S108 will be described later.

[0093] In step S109, the input history update unit 225 increases the number of times the operator selects a tag in the current annotation operation based on the tag selection result information. Additionally, the input history update unit 225 determines the number of annotation regions (ARs) set for the tags selected by the operator based on the annotation instruction information. Furthermore, the input history update unit 225 increases the number of times the operator selects a tag based on the number of annotation regions (ARs). Figure 7 This diagram illustrates the processing of the input history update unit 225 when the number of times a tag is selected increases. For example, in step S104, the tag "bicycle" is selected, and in step S105, three annotation areas AR are specified for the tag "bicycle". In this case, as... Figure 7 As shown, inputting the history update unit 225 increases the number of times the "bicycle" tag is selected by 3 points.

[0094] In step S110, the list display order update unit 226 updates the display order of the multiple tags contained in the tag list LL according to the number of times each tag has been selected. For example, as Figure 7 As shown, when the "bicycle" tag is selected the most times, the list display order update unit 226 rearranges the display order of multiple tags in the tag list LL so that the "bicycle" tag is at the top of the tag list LL.

[0095] In step S111, the display control unit 223 sends instructions to the operator terminal 10 to instruct the display device 102 to display the next image data IM and the label list LL with the changed display order. For example, the display control unit 223 instructs the display device 102 to display... Figure 8 The second display image IM2, which shows four figures riding bicycles, and the processing of the list display order update unit 226 display the "bicycle" tag in the topmost tag list LL.

[0096] Next, returning to step S104, the control unit 103 of the operator terminal 10 receives the operator's selection of a tag for the second display image IM2. Here, it is assumed that the operator has again selected the "bicycle" tag.

[0097] In step S105, the control unit 103 receives an annotation instruction for the second display image IM2 from the operator, and sets an annotation area AR for the second display image IM2 based on the annotation instruction. Figure 8 As shown, here, it is assumed that the control unit 103 sets the area inside the four quadrilateral frames FL that overlap with the second display image IM2, which respectively surround the four bicycles, as the annotation area AR.

[0098] In step S106, the determination unit 227 again determines whether the annotation work has been completed. Here, it is assumed that the determination unit 227 determines that the annotation work has not been completed.

[0099] In step S107, the determination unit 227 determines whether the image similarity between the first displayed image IM1 and the second displayed image IM2 is below an image similarity threshold. Here, it is assumed that the determination unit 227 determines that the image similarity between the first displayed image IM1 and the second displayed image IM2 exceeds the image similarity threshold. In this case, the process proceeds to step S109.

[0100] In step S109, the input history update unit 225 increases the number of times the operator selects the tag in the current annotation operation. Figure 9 This diagram illustrates the processing of the input history update unit 225 when the number of times a label is selected is increased. As explained in steps S104 and S105, in the annotation operation for the second displayed image IM2, four annotation regions AR are designated for a label such as "bicycle". In this case, as... Figure 9 As shown, inputting the history update unit 225 further increases the number of times the "bicycle" tag can be selected by 4 points.

[0101] In step S110, the list display order update unit 226 updates the display order of multiple tags contained in the tag list LL. For example... Figure 9 As shown, since the "bicycle" tag is selected the most times, the list display order update unit 226 rearranges the display order of multiple tags in the tag list LL so that the "bicycle" tag is at the top of the tag list LL.

[0102] In step S111, the display control unit 223 sends an instruction to the operator terminal 10 to cause the display device 102 to display the next image data IM, i.e., the third display image IM3, and the label list LL with the changed display order. Specifically, the display control unit 223 causes the display device 102 to display... Figure 10 The third display image, IM3, shows one person.

[0103] Returning to step S104, the control unit 103 of the operator terminal 10 receives the selection of the tag for the third display image IM3 from the operator. Here, as... Figure 10 As shown, the "People" label was selected.

[0104] In step S105, the control unit 103 receives an annotation instruction for the third display image IM3 from the operator, and sets an annotation area AR for the third display image IM3 based on the annotation instruction. Figure 10 As shown, here, the control unit 103 sets the area inside the frame FL of a quadrilateral shape that overlaps with the third display image IM3 and surrounds a person as the annotation area AR.

[0105] In step S106, the determination unit 227 again determines whether all annotation work has been completed. Here, it is assumed that the determination unit 227 determines that all annotation work has not been completed.

[0106] In step S107, the determination unit 227 determines whether the image similarity between the second display image IM2 and the third display image IM3 is below an image similarity threshold. Here, it is assumed that the determination unit 227 determines that the image similarity between the second display image IM2 and the third display image IM3 is below the image similarity threshold. In this case, the process proceeds to step S108.

[0107] In step S108, the input history update unit 225 resets the selection counts of all tags recorded in the input history storage unit 212. Furthermore, after resetting the selection counts of all tags, the selection counts of the tags selected in the annotation job prior to the reset are updated. Figure 11 This diagram illustrates the processing of the input history update unit 225 when the number of tag selections is reset. (See diagram for example.) Figure 11 As shown, after updating the selection count of all tags to 0, the input history update unit 225 increases the selection count of the "person" tag selected in the annotation job before the reset by 1.

[0108] Furthermore, in step S110, the list display order update unit 226 updates the display order of the multiple tags contained in the tag list LL according to the number of times each tag has been selected. For example... Figure 11 As shown, the "person" tag is selected most frequently. Therefore, the list display order update unit 226 rearranges the display order of multiple tags in the tag list LL so that the "person" tag is at the top of the tag list LL. Furthermore, for tags other than the "person" tag, the list display order update unit 226 can rearrange them in a random display order.

[0109] Furthermore, in step S111, the display control unit 223 sends an instruction to the operator terminal 10 to cause the display device 102 to display the next image data IM, namely the fourth display image IM4, and the label list LL with the changed display order. Then, in step S106, the same process is repeated until the determination unit 227 determines that the annotation work for all image data IMs has been completed.

[0110] As explained above, according to the annotation work assistance system 1A of Embodiment 1, the display order of multiple labels is rearranged so that labels that are frequently selected by the operator during the annotation work, i.e., labels that are more likely to be selected by the operator, are easier for the operator to select. Therefore, the appropriate label can be found from the label list LL, simplifying the selection of the appropriate label. As a result, the annotation work becomes simpler.

[0111] Furthermore, according to the above structure, since the labels that are more likely to be selected by the operators are displayed in a position that is easy for the operators to select, label selection errors can be reduced.

[0112] (Implementation Method 2)

[0113] Implementation 2 involves the following process: when increasing the number of times a label is selected by the operator during the annotation process, the number of times similar labels that are similar to those selected by the operator is also increased. Furthermore, in Implementation 2, the same reference numerals are used for the same constituent elements as in Implementation 1, and descriptions are omitted. This also applies to the embodiments described later.

[0114] Figure 12 This is a block diagram illustrating an example of the annotation task assistance system 1B according to Embodiment 2. Figure 12 As shown, the annotation task assistance system 1B according to Embodiment 2 differs from the annotation task assistance system 1A according to Embodiment 1 in that the memory 210 has a similar word dictionary database 213.

[0115] The similar word dictionary database 213 is a database that records similar words for a given word. For example, the similar word dictionary database 213 records similar words such as "motorcycle" and "tricycle" for the word "bicycle". Similarly, the similar word dictionary database 213 records similar words such as "bicycle" and "tricycle" for the word "motorcycle".

[0116] Reference Figure 13 The flowchart below illustrates the processing of the annotation task assistance system 1B as described above. Figure 13 This is a flowchart illustrating an example of the overall processing of the annotation task assistance system 1B according to Embodiment 2.

[0117] Regarding the processing of steps S201 and S202, since they are the same as those of steps S101 and S102 described above, detailed explanations are omitted. That is, in Embodiment 2, similar to Embodiment 1, the image display order update unit 222 determines the display order in which the display device 102 of the operator terminal 10 displays each image data IM based on the image similarity evaluated by the image similarity evaluation unit 221. Furthermore, regarding the processing of steps S203 to S209, since they are the same as those of steps S103 to S109 described above, detailed explanations are omitted. As an example, in the processing of step S203, it is envisioned that... Figure 5 The image data IM shown is displayed on the display device 102 as the first display image IM1. Furthermore, in the processes of steps S204 and S205, it is assumed that the operator selects the "bicycle" tag and specifies... Figure 6 The case of three annotation regions AR as shown. Additionally, in the processing of step S209, it is assumed that the input history update unit 225... Figure 7 The example shown increases the number of times the "bicycle" tag is selected by 3 points.

[0118] In step S210, the input history update unit 225 increases the selection count of similar tags that are similar to the tags selected by the operator in the current annotation task. Specifically, first, the input history update unit 225 checks whether there are words similar to the tag "bicycle" selected by the operator in the current annotation task in the similar word dictionary database 213. As mentioned above, the similar word dictionary database 213 records words such as "motorcycle" and "tricycle" that match the similarity of "bicycle". Next, the input history update unit 225 checks whether the tags "motorcycle" and "tricycle" are included in the tag list LL. Here, it is assumed that the tag "motorcycle" is included in the tag list LL, while the tag "tricycle" is not included in the tag list LL. In this case, the input history update unit 225 processes the tag "motorcycle" as a similar tag. Furthermore, the input history update unit 225 increases the selection count of the similar tag "motorcycle" by the same number as the selection count of the tag "bicycle". Figure 14 This diagram illustrates the processing of the input history update unit 225 when the number of selections for similar tags is increased. As described above, in step S209, the input history update unit 225 increases the number of selections for the "bicycle" tag by 3. Therefore, as... Figure 14 As shown, inputting the history update unit 225 increases the number of selections for the similar tag "motorcycle" by 3 points.

[0119] Since the processes in steps S211 and S212 are the same as those in steps S110 and S111 described above, their explanations are omitted. Hereinafter, the process continues until the determination unit 227 determines that the annotation work for all image data IM has been completed. Figure 13 The process described in the procedure. For example, if image data IM containing a motorcycle is displayed on the display device 102 and the operator selects "motorcycle" as a tag, the number of times the similar tag "bicycle" is selected increases along with the number of times the "motorcycle" tag is selected.

[0120] As explained above, according to the annotation task assistance system 1B of Embodiment 2, when an operator selects the "bicycle" tag in a certain annotation task, the number of selections for the similar tag "motorcycle" also increases. According to this structure, the display order of multiple tags is updated so that not only "bicycle" but also the "motorcycle" tag is displayed in a position easily selectable by the operator. Therefore, in another annotation task, when image data IM containing a target motorcycle is displayed, the operator can easily select the "motorcycle" tag. As a result, the annotation task is simpler. The structure of the annotation task assistance system 1B described above is particularly suitable for cases where object data consists of multiple image data IMs that are similar to each other, as in Embodiment 2. In other words, the structure of the annotation task assistance system 1B is particularly suitable when the image dataset IMS contains a target similar to the target contained in one image data IM, and there is a high probability that the target is contained in another image data IM.

[0121] (Implementation Method 3)

[0122] Implementation method 3 performs the following process: determine the third annotation region AR3, and increase the number of times the label is selected based on the number of the third annotation region AR3.

[0123] Figure 15 This is a block diagram illustrating an example of the annotation work assistance system 1C according to Embodiment 3. For example... Figure 15 As shown, the annotation work assistance system 1C according to Embodiment 3 differs from the annotation work assistance system 1B according to Embodiment 2 in that the processor 220 has a region similarity evaluation unit 228 and the control unit 103 has a region extraction unit 103A. Furthermore, the annotation work assistance system 1C according to Embodiment 3 differs from the annotation work assistance systems 1A and 1B described above in that it lacks an image similarity evaluation unit 221 and an image display order update unit 222.

[0124] When the region extraction unit 103A receives an instruction from the operator to locate a target contained in the image data IM by specifying a point, it extracts the target from the image data IM based on the location of the point and sets an annotation region AR in the vicinity of the target.

[0125] The region similarity evaluation unit 228 determines the third annotation region AR3 based on the region similarity between the first annotation region AR1 and the second annotation region AR2. Specifically, the region similarity evaluation unit 228 evaluates the region similarity between one or more first annotation regions AR1 set in one image data IM and one or more second annotation regions AR2 set in another image data IM different from the one image data IM. Specifically, the region similarity evaluation unit 228 evaluates region similarity by performing the same processing as described by the image similarity evaluation unit 221. That is, the region similarity evaluation unit 228 uses calculation methods such as absolute error, L2 error, and Cos distance to calculate the error of feature quantities between regions, thereby evaluating the region similarity. Furthermore, the region similarity evaluation unit 228 determines the first annotation region AR1 whose region similarity exceeds a predetermined region similarity threshold (an example of the first threshold) as the third annotation region AR3.

[0126] The input history update unit 225 in Implementation 3 increases the number of times the selected tag for the first annotation region AR1 is selected based on the number of the third annotation region AR3.

[0127] Reference Figure 16 The flowchart below illustrates the processing of the annotation task assistance system 1C as described above. Figure 16 This is a flowchart illustrating an example of the overall processing of the annotation task assistance system 1C according to Embodiment 3. Furthermore, as described above, the annotation task assistance system 1C according to Embodiment 3 does not have an image similarity evaluation unit 221 or an image display order update unit 222. Therefore, in Embodiment 3, each image data IM constituting the image dataset IMS is displayed on the display device 102 in a random display order.

[0128] In step S301, the display control unit 223 causes the display device 102 to display image data IM and tag list LL. For example, the display control unit 223 causes the display device 102 to display... Figure 17 The first display image IM1 (an example of the first image data) shows three figures riding bicycles, along with a tag list LL.

[0129] In step S302, the control unit 103 of the operator terminal 10 receives the operator's selection of a tag for the first display image IM1. Here, it is assumed that the operator selected the "bicycle" tag.

[0130] In step S303, the control unit 103 receives an annotation instruction for the first display image IM1 from the operator, and sets an annotation area AR for the first display image IM1 based on the annotation instruction. Here, as... Figure 17As shown, three annotation regions AR are defined by three quadrilateral frames FL that respectively surround the three bicycles. Furthermore, in step S303, annotation instructions are given by assuming that an operator uses a mouse to specify the upper left and lower right vertices of each annotation region AR, as described in step S105 of Embodiment 1. The control unit 103 processes these three annotation regions AR as the first annotation region AR1. Hereinafter, Figure 17 The three first annotation regions AR1 shown are referred to as first annotation region AR1a, first annotation region AR1b, and first annotation region AR1c, respectively.

[0131] In step S304, the input history update unit 225 increases the number of times the operator selects a tag based on the number of first annotation areas AR1. As described above, in step S303, three first annotation areas AR1 are specified for a tag such as "bicycle". Therefore, as Figure 14 As shown, inputting the history update unit 225 increases the number of times the "bicycle" tag is selected by 3 points. At this time, as... Figure 14 As shown, inputting the history update unit 225 can also increase the number of selections for similar tags such as "motorcycle" by 3 points.

[0132] In step S305, the list display order update unit 226 updates the display order of multiple tags contained in the tag list LL according to the number of times each tag has been selected.

[0133] In step S306, the display control unit 223 sends instructions to the operator terminal 10 to instruct the display device 102 to display the next image data IM and the label list LL with the changed display order. For example, the display control unit 223 instructs the display device 102 to display... Figure 18 The image data IM showing a person riding a motorcycle is displayed as the second image IM2 (an example of the second image data).

[0134] In step S307, the control unit 103 receives an annotation instruction for the second display image IM2 from the operator, and sets an annotation area AR for the second display image IM2 based on the annotation instruction. Here, as... Figure 18 As shown, an annotation area AR is defined by a quadrilateral frame FL that surrounds a motorcycle.

[0135] The indication of the second annotation area AR2 in step S307 is achieved by the operator specifying the location of the target contained in the image data IM using a point. Specifically, the operator aligns the mouse cursor MC with... Figure 18In the state of the area represented by point P, the mouse of the operation unit 101 is clicked. Hereinafter, such an operation is referred to as point designation. When a point designation is received, the area extraction unit 103A of the control unit 103 identifies the pixels surrounding the point P designated by the operator to infer the outline of the target designated by the operator. Such technology is exemplified in non-patent literature "Alexander Kirillov, 11 others, "Segment Anything", [online], Meta AI Research, FAIR, [searched on September 22, 2006], Internet<URL:https: / / arxiv.org / pdf / 2304.02643.pdf> "It was made public in the middle. And, as..." Figure 18 As shown, the region extraction unit 103A extracts a region by enclosing the area near the outline of the target specified by the operator using a quadrilateral frame FL formed by a pair of horizontal lines extending in the width direction w and a pair of vertical lines extending in the height direction h. The control unit 103 sets the region thus extracted as the second annotation region AR2. However, the method for indicating the second annotation region AR2 in step S307 is not limited to this; the second annotation region AR2 can also be indicated using the same method as described in step S105 of Embodiment 1.

[0136] return Figure 16 In step S308, the region similarity evaluation unit 228 evaluates the region similarity between the first annotation region AR1 and the second annotation region AR2. (Refer to...) Figure 19 This section will explain in detail the processing of the region similarity evaluation unit 228. Figure 19 The diagram shows three first annotation regions AR1, AR1b, and AR1c set in step S303, and one second annotation region AR2 set in step S307. For example... Figure 19 As shown, the region similarity evaluation unit 228 compares the second annotation region AR2 with each of the first annotation regions AR1 to evaluate the region similarity between the second annotation region AR2 and each of the first annotation regions AR1. Furthermore, the region similarity evaluation unit 228 designates the first annotation region AR1 whose region similarity relative to the second annotation region AR2 exceeds a region similarity threshold as the third annotation region AR3. Moreover, when there are multiple second annotation regions AR2, the region similarity evaluation unit 228 evaluates the region similarity between each of the multiple second annotation regions AR2 and each of the multiple first annotation regions AR1.

[0137] return Figure 16In step S309, the determination unit 227 determines whether the third annotation region AR3 exists. If the third annotation region AR3 exists (if yes in step S309), the process proceeds to step S310. If the third annotation region AR3 does not exist, the process proceeds to step S311.

[0138] In step S310, the input history update unit 225 increases the number of times the label set for the first annotation region AR1 is selected based on the number of third annotation regions AR3. As an example, consider a case where the region similarity evaluation unit 228 evaluates that the region similarity of the first annotation regions AR1a and AR1c relative to the second annotation region AR2 exceeds the region similarity threshold. That is, consider a case where the region similarity evaluation unit 228 determines the first annotation regions AR1a and AR1c as the third annotation region AR3. Figure 20 This diagram illustrates the processing of the input history update unit 225 when the number of times a tag is selected for the first annotation region AR1 increases based on the number of third annotation regions AR3. As described above, the tag "bicycle" was selected as the tag set for the first annotation region AR1. Furthermore, there are two third annotation regions AR3. In this case, as... Figure 20 As shown, inputting the history update unit 225 increases the selection count of the "bicycle" tag by 2 points. At the same time, inputting the history update unit 225 can also increase the selection count of the similar tag "motorcycle" (which is similar to "bicycle") by 2 points.

[0139] return Figure 16 In step S311, the control unit 103 receives the selection of a tag for the second annotation area AR2. That is, in the absence of a third annotation area AR3, the control unit 103 instructs the operator to select a tag set for the second annotation area AR2. Here, it is assumed that the operator selected the tag "motorcycle" as the tag set for the second annotation area AR2. As described above, in step S307, since one second annotation area AR2 is set, the input history update unit 225 increases the selection count of the "motorcycle" tag by 1. At this time, the input history update unit 225 can also increase the selection count of a similar tag, such as "bicycle," which is similar to "motorcycle" by 1.

[0140] In step S312, the determination unit 227 determines whether all annotation work has been completed. If all annotation work has been completed (if yes in step S312), the processing of the annotation work assistance system 1C ends. If not all annotation work has been completed (if no in step S312), the process proceeds to step S313.

[0141] In step S313, the control unit 103 processes the region that was set as the second annotation region AR2 in the processing of step S307 into a new first annotation region AR1. For example, the control unit 103 will process the region that was set as the second annotation region AR2 in the processing of step S307 into a new first annotation region AR1. Figure 18 The second annotation region AR2 of the second display image IM2 shown is processed into a new first annotation region AR1. In addition, at this time, the control unit 103 processes the labels that have been added in the processing of step S310 or step S311, and labels other than similar labels, into labels set for the new first annotation region AR1.

[0142] In step S314, the list display order update unit 226 updates the display order of multiple tags contained in the tag list LL according to the number of times each tag has been selected.

[0143] In step S315, the display control unit 223 causes the display device 102 to display the next image data IM and the tab list LL with the changed display order. For example, the display control unit 223 displays... Figure 21 The third display image, IM3, shows four figures riding bicycles.

[0144] Furthermore, returning to step S307, the control unit 103 receives a new annotation instruction for the third display image IM3 from the operator, and based on this annotation instruction, sets a new second annotation area AR2 for the third display image IM3. For example, the control unit 103 will... Figure 21 The areas inside the four quadrilateral frames FL that surround the four bicycles, as shown, are designated as the new second annotation area AR2.

[0145] Furthermore, proceeding to step S308, the region similarity evaluation unit 228 evaluates the region similarity between the first annotation region AR1 in the second displayed image IM2 and the four second annotation regions AR2 in the third displayed image IM3 to determine the third annotation region AR3. The same process is repeated until the determination unit 227 determines that the annotation work for all image data IMs has been completed.

[0146] According to the annotation task assistance system 1C of Embodiment 3 described above, when the first annotation region AR1 and the second annotation region AR2 have a high degree of regional similarity, for example, when the first annotation region and the second annotation region contain the same target, the number of times the label selected for the first annotation region AR1 is increased. Thus, since the task of selecting labels for the second annotation region is omitted, the annotation task is simplified.

[0147] (Implementation Method 4)

[0148] In Implementation 3, the following example is described: In step S303, in order to indicate the first annotation region AR1, the operator uses the mouse to specify the upper left and lower right vertices of the first annotation region AR1. However, the first annotation region AR1 can also be indicated by the points specified above.

[0149] The structure of the annotation job assistance system according to Embodiment 4 is the same as that of the annotation job assistance system 1C according to Embodiment 3. Furthermore, the processing flow of the annotation job assistance system according to Embodiment 4 is the same as that of the annotation job assistance system 1C according to Embodiment 3. Therefore, hereafter, using... Figure 16 The flowchart shown illustrates the processing of the annotation task assistance system involved in Implementation 4.

[0150] In step S301, the display control unit 223 causes the display device 102 to display image data IM and tag list LL. For example, the display control unit 223 causes the display device 102 to display... Figure 22 The first display image IM1 shows three bicycles as shown.

[0151] In step S302, the control unit 103 of the operator terminal 10 receives the selection of a tag for the first display image IM1. Here, it is assumed that the operator selected the tag "bicycle".

[0152] In step S303, the control unit 103 receives an annotation instruction for the first display image IM1, and based on the annotation instruction, sets a first annotation area AR1 for the first display image IM1. Specifically, the operator aligns the mouse cursor MC with... Figure 22 When the operator clicks the mouse (operation unit 101) while the cursor is positioned over the area represented by point P1, the target location is specified by the point. Similarly, when the operator clicks the mouse with the cursor MC aligned with the areas represented by points P2 and P3, the target location is specified. Upon receiving such an operation, the area extraction unit 103A of the control unit 103 identifies the pixels surrounding points P1, P2, and P3 to infer the outlines of each target specified by the operator. That is, it infers the outlines of each of the three bicycles. Furthermore, the area extraction unit 103A... Figure 22 The three quadrilateral frames FL shown enclose the area near the three bicycles to extract the region. The control unit 103 sets the extracted region as the first annotation region AR1.

[0153] According to the annotation task assistance system described in Embodiment 4 above, the first annotation region AR1 is indicated by specifying the location of the target using a point. Therefore, compared with the case of using a mouse to specify the upper left and lower right vertices of the first annotation region AR1, the task of indicating the first annotation region AR1 becomes simpler. As a result, the annotation task is made simpler.

[0154] The present disclosure can be adapted to the following variations.

[0155] (Variation example)

[0156] (1) This disclosure can also be applied to cases where the object data is sound data, text data, or animation data. That is, this disclosure can also be applied when performing annotation operations such as reproducing sound data as object data and assigning labels such as "kitchen sound," "living room sound," "dog barking," and "train sound" to the sound data. In this case, the display device 102 may, for example, have a built-in speaker. Similarly, this disclosure can also be applied when performing annotation operations such as displaying text data as object data and assigning labels such as "noun," "verb," ​​"adverb," ​​and "conjunction" to each word constituting the text data. Furthermore, this disclosure can also be applied to annotation operations that assign labels to each image data IM constituting animation data.

[0157] (2) Tag selection can also be achieved by inputting voice data into the control unit 103. For example, the operator terminal 10 may also have a microphone (not shown), and the control unit 103 may also have a voice recognition unit (not shown). In this case, the operator speaks words such as "bicycle," "motorcycle," or "person" into the microphone. These voices are recognized by the voice recognition unit of the control unit 103, and a tag corresponding to the voice spoken by the operator is determined. Furthermore, the determined tag is selected as the tag set for image data such as IM.

[0158] Alternatively, label selection can also be achieved by inputting text data into the control unit 103. For example, the display device 102 of the operator terminal 10 can also display a text box for inputting text, where the operator can input text such as "bicycle". In this case, the control unit 103 determines the label corresponding to the text input by the operator, and the determined label is selected as the label set for image data such as IM.

[0159] According to this structure, since labels can be selected based on either voice or text data, operators do not need to manually select appropriate labels. As a result, annotation work becomes much simpler.

[0160] Furthermore, when receiving tag selections based on voice or text data, the number of selections recorded in the input history can be reset. That is, the tag display order can also be reset once. As a result, in subsequent annotation tasks, since the tag display order is easily updated, it is easier to update the tag display order to the appropriate order. Consequently, the annotation task is simplified.

[0161] (3) The image similarity evaluation unit 221 and the image display order update unit 222 are not necessary. For example, the operator may pre-create an image dataset IMS consisting of multiple image data IMs that are similar to each other.

[0162] (4) When a label has moved to the top of the label list LL due to the processing of the list display order update unit 226, the control unit 103 may change the background color of the cell containing the label. Alternatively, the control unit 103 may change the color of the text representing the label. Alternatively, the control unit 103 may make the text representing the label bold. Alternatively, the control unit 103 may make the text representing the label blink. Alternatively, the control unit 103 may make the size of the cell containing the label larger than the size of the cells containing other labels. In this way, the label that has moved to the top of the label list LL can be made more conspicuous. As a result, the user's label selection operation can be made easier.

[0163] (5) In Embodiments 3 and 4, examples were described in which the processor 220 does not have the image similarity evaluation unit 221 and the image display order update unit 222, but the processor 220 may also have these functional units. That is, in Embodiments 3 and 4, the image dataset IMS may also be rearranged based on image similarity.

[0164] (6) In the above embodiments, examples of setting labels for targets contained in image data IM have been described, but labels can also be set for regions contained in image data IM.

[0165] Industrial availability

[0166] This disclosure is useful in the technical field of assisting annotation work.

Claims

1. An information processing method, which is an information processing method in an information processing device. In the information processing method, The object data and a list of tags containing multiple tags for classifying the object data are displayed on the operator's terminal. Receive a selection from any one of the plurality of tags. Based on the selection results of the tags, update the input history that records the number of times each of the multiple tags was selected. Whenever the display of the object data is switched to the next object data, the display order of the multiple tags contained in the tag list is updated according to the number of selections recorded in the input history.

2. The information processing method according to claim 1, wherein, The object data is image data. Furthermore, receive annotation instructions for the image data. The label is selected to categorize the targets contained in the annotation area corresponding to the annotation indication.

3. The information processing method according to claim 2, wherein, The object data includes: first image data, and second image data displayed after the first image data. Furthermore, a third annotation region is determined among one or more first annotation regions set in the first image data, whose regional similarity to one or more second annotation regions set in the second image data exceeds a first threshold. The update of the input history includes: increasing the number of times the selected tag is selected for more than one first annotation region, based on the number of the third annotation regions.

4. The information processing method according to claim 2, wherein, The annotation indicator can specify more than one annotation region for a single tag. The update of the input history includes increasing the number of times the tag is selected based on the number of annotation regions specified for the tag.

5. The information processing method according to claim 1 or 2, wherein, The update of the input history includes increasing the number of times the selected tag is selected and the number of times similar tags similar to the selected tag are selected.

6. The information processing method according to claim 2, wherein, The display order of the object data is determined by ranking the image similarity to the previously displayed object data from highest to lowest.

7. The information processing method according to claim 6, wherein, Furthermore, if the image similarity is below the second threshold, the number of selections recorded in the input history is reset.

8. The information processing method according to claim 2, wherein, The annotation area is the polygon surrounding the target or the outline of the target.

9. The information processing method according to claim 1 or 2, wherein, The selection of the tag receives input including voice data or text data.

10. The information processing method according to claim 9, wherein, Furthermore, upon receiving a selection of a tag based on the voice data or the text data, the number of selections recorded in the input history is reset.

11. The information processing method according to claim 2, wherein, The annotation indicates the location of the target using dots. Furthermore, based on the location of the point, the target is extracted from the image data, and the annotation region is set for the target.

12. An information processing device, comprising: The display control unit displays object data and a list of tags containing multiple tags for classifying the object data on the operator's terminal; The receiving unit receives the selection of any one of the plurality of tags; The history update unit updates the input history that records the number of times each of the multiple tags was selected, based on the selection results of the tags. and The sequence update unit updates the display order of the multiple tags in the tag list according to the number of selections recorded in the input history whenever the display of the object data is switched to the next object data.

13. An information processing program that causes a processor to perform the following processing: The object data and a list of tags containing multiple tags for classifying the object data are displayed on the operator's terminal. Receive a selection from any one of the plurality of tags. Based on the selection results of the tags, update the input history that records the number of times each of the multiple tags was selected. Whenever the display of the object data is switched to the next object data, the display order of the multiple tags contained in the tag list is updated according to the number of selections recorded in the input history.

Images