Information processing device, information processing method, and program

Abstract

To reduce the time it takes for annotations implemented when generating a trained model and enhance annotation accuracy.SOLUTION: An information processing device is provided, comprising an image acquisition unit for acquiring an image including a target object, a tag information acquisition unit for acquiring tag information for identifying the target object, and an annotation unit for applying an annotation related to the target object on the image on the basis of the tag information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program capable of executing annotation. [Background technology]

[0002] Conventionally, there are techniques for detecting various types of information using trained models. For example, a technique has been proposed in which multiple learning devices are used to detect multiple specification areas of an analog meter from a captured image of the analog meter (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-135353 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, when generating a trained model, it is necessary to specify the image area of ​​the target object and perform annotation, which involves tagging (labeling) that image area. In this case, performing annotation manually requires a huge amount of time and effort. Furthermore, in order to improve the detection accuracy of image recognition using a trained model, it is important to improve the accuracy of the annotation.

[0005] The present invention aims to reduce the time required for annotation when generating a trained model and to improve the accuracy of annotation. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device having an image acquisition unit that acquires an image including an object, a tag information acquisition unit that acquires tag information that identifies the object, and an annotation unit that applies annotations related to the object to the image based on the tag information. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to reduce the time required for annotation when generating a trained model and improve the accuracy of annotation. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of use of an information processing system. [Figure 2] FIG. 2 is a top view showing an example of the configuration of a physical tag. [Figure 3] FIG. 3 is a top view showing an example of the configuration of a physical tag related to a barcode. [Figure 4] FIG. 4 is a top view showing an example of the configuration of a physical tag related to a graphic code. [Figure 5] FIG. 5 is a top view showing an example of the configuration of a physical tag relating to another graphic code. [Figure 6] FIG. 6 is a block diagram illustrating an example of the system configuration of the information processing system. [Figure 7] FIG. 7 is a diagram showing each piece of information stored in the setting information DB. [Figure 8A] FIG. 8A is a diagram showing an example of rectangular image region information. [Figure 8B] FIG. 8B is a diagram showing an example of circular image region information. [Figure 9A] FIG. 9A is a diagram showing an example of transition when a figure is subjected to affine transformation and translated. [Figure 9B] FIG. 9B is a diagram showing an example of transition when a graphic is enlarged or reduced by affine transformation. [Figure 9C] FIG. 9C is a diagram showing an example of a transition when a figure is rotated by affine transformation. [Figure 9D] FIG. 9D is a diagram showing an example of transition when a figure is sheared by affine transformation. [Figure 10] FIG. 10 is a diagram showing an example of setting a bounding box for a box-shaped container. [Figure 11] FIG. 11 is a diagram showing an example of transformation when image region information is subjected to affine transformation. [Figure 12] FIG. 12 is a diagram showing an example in which physical tags are provided on three sides of a container. [Figure 13] FIG. 13 is a diagram showing an example of setting a bounding box for a cylindrical container. [Figure 14] FIG. 14 is a diagram showing an example of a display screen on a user terminal. [Figure 15] FIG. 15 is a diagram illustrating an example of an annotation method. [Figure 16] FIG. 16 is a flowchart illustrating an example of annotation processing. [Figure 17A] FIG. 17A is a perspective view showing an example in which physical tags are provided on two edges of a box-shaped container. [Figure 17B] FIG. 17B is a diagram showing an example of image area information when physical tags are provided on two sides of the edge of a box-shaped container. [Figure 18A] FIG. 18A is a perspective view showing an example in which physical tags are provided at two positions on the edge of a cylindrical container. [Figure 18B] FIG. 18B is a diagram showing an example of image area information when physical tags are provided at two positions on the edge of a cylindrical container. [Figure 19A] FIG. 19A is a perspective view showing an example in which a physical tag is provided on one side of the edge of a container according to a modified example. [Figure 19B] FIG. 19B is a diagram showing an example of image area information when a physical tag is provided on one side of the edge of a container according to a modified example. [Figure 20] FIG. 20 is a diagram showing a modified example in which an information processing system is used. [Figure 21] FIG. 21 is a diagram showing an example of identification information provided on the surface of a physical tag. [Figure 22] FIG. 22 is a block diagram showing an example of a system configuration according to a modified example of the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0009] The embodiments described below are not limited to the drawings described by the brief description of the drawings.

[0010] (1) One aspect of the present invention is an information processing device having an image acquisition unit that acquires an image including an object, a tag information acquisition unit that acquires tag information that identifies the object, and an annotation unit that applies annotations regarding the object to the image based on the tag information.

[0011] This allows annotation of an object to be performed on an image based on tag information that identifies the object, thereby reducing the time required for annotation when generating a trained model and improving the accuracy of annotation.

[0012] (2) An aspect of the present invention is the information processing device according to (1), in which the tag information is information acquired from a physical tag.

[0013] This makes it possible to appropriately obtain tag information about each object using the physical tag.

[0014] (3) One aspect of the present invention is the information processing device according to (2), in which the physical tag is associated with a container that contains the object.

[0015] According to this, even if a physical tag is not provided on the object itself, it can be provided in association with the container that contains the object. For example, if the physical tag cannot be directly attached to the container, it can be provided around the container (for example, on a sign behind it). This makes it possible to appropriately obtain tag information for each object.

[0016] (4) One aspect of the present invention is an information processing device according to any one of (1) to (3), wherein the object is contained in a container corresponding to the object, the container is provided with a physical tag capable of reading the tag information, the tag information includes attributes of the object or the container, and the annotation unit performs the annotation on the image based on the attributes included in the tag information.

[0017] According to this, even if the object itself does not have a physical tag, tag information about each object can be appropriately acquired by attaching a physical tag to the container that holds the object. This makes it possible to acquire and use the attributes of the object (or the attributes of the container), and to perform annotation on the image based on the acquired attributes.

[0018] (5) One aspect of the present invention is an information processing device as described in (3) or (4), in which the image acquisition unit acquires the image from an imaging device that images the container and generates an image including the container, the object contained in the container, and the physical tag attached to the container, and the tag information acquisition unit acquires the tag information based on the physical tag included in the image.

[0019] This makes it possible to acquire a captured image including a container that contains an object, and acquire tag information based on a physical tag attached to the container included in the captured image, thereby making it possible to appropriately acquire tag information related to each object.

[0020] (6) One aspect of the present invention is an information processing device according to any one of (1) to (5), wherein the object or the container in which the object is stored is provided with a physical tag capable of reading the tag information, and the physical tag is at least one of a barcode, a two-dimensional code, a color code capable of expressing multiple pieces of information by an arrangement of multiple colors, a graphic code capable of expressing multiple pieces of information by an arrangement of multiple figures each having one of multiple colors, and a wireless tag capable of transmitting the tag information using wireless communication.

[0021] This makes it possible to provide a container with an appropriate physical tag that corresponds to the environment of the installation location of the container that contains the object, thereby making it possible to acquire tag information from the physical tag according to the environment of the installation location of the container.

[0022] (7) One aspect of the present invention is an information processing device according to any one of (1) to (6), further comprising a setting unit that sets an image area including the object from the image based on the tag information, and the annotation unit performs the annotation on the image area.

[0023] This allows only the image area containing the object to be the target of annotation processing, thereby reducing the amount of calculation required for annotation processing. Also, since only the image area containing the object can be the target of annotation processing, the accuracy of annotation can be improved.

[0024] (8) One aspect of the present invention is an information processing device according to any one of (1) to (7), wherein the object is at least one of waste, agricultural products, marine products, and industrial products.

[0025] This makes it possible to reduce the time required for annotation and improve the accuracy of annotation when annotating an image that includes a target object (at least one of waste, agricultural products, marine products, and industrial products).

[0026] (9) One aspect of the present invention is an information processing device according to any one of (1) to (8), further comprising a generation unit that generates a trained model using the annotated image.

[0027] This makes it possible to annotate an image based on tag information that identifies the object, and then generate a trained model using the annotated image. This reduces the time required for the generation process when generating a trained model, and improves the accuracy of the generation process.

[0028] (10) One aspect of the present invention is the information processing device described in (9), in which the generation unit generates multiple trained models for each attribute of the object.

[0029] This makes it possible to select and use an appropriate trained model from among the multiple trained models generated, depending on the attributes of the object.

[0030] (11) One aspect of the present invention is an information processing method including an image acquisition process for acquiring an image including an object, a tag information acquisition process for acquiring tag information that identifies the object, and an annotation process for applying an annotation regarding the object to the image based on the tag information.

[0031] This allows annotation of an object to be performed on an image based on tag information that identifies the object, thereby reducing the time required for annotation when generating a trained model and improving the accuracy of annotation.

[0032] (12) One aspect of the present invention is a program that causes a computer to execute an image acquisition procedure for acquiring an image including an object, a tag information acquisition procedure for acquiring tag information that identifies the object, and an annotation procedure for applying an annotation regarding the object to the image based on the tag information.

[0033] This allows annotation of an object to be performed on an image based on tag information that identifies the object, thereby reducing the time required for annotation when generating a trained model and improving the accuracy of annotation.

[0034] Hereinafter, embodiments will be described with reference to the accompanying drawings.

[0035] [Example of use of information processing system] Fig. 1 is a diagram showing an example of using the information processing system 1. Fig. 1 shows an example in which a plurality of types of waste are stored in a plurality of containers C1 to C11 at a waste collection site WP1. Fig. 1 also shows an example in which physical tags PT1 to PT11 are provided on the plurality of containers C1 to C11, respectively, for identifying the waste stored in each container.

[0036] For example, containers C1 and C5 contain plastic bags, containers C2 and C6 contain paper waste, containers C3 and C7 contain PP (polypropylene) resin, containers C4 and C8 contain hard plastic, container C9 contains glass waste, container C10 contains iron scrap, and container C11 contains circuit boards. The containers C1 to C11 can be made of a material and have a structure that suits the waste to be stored. For example, they can be made of a metal material with a mesh structure that is visible from the outside, or they can be made of a metal material with a box-like or cylindrical structure. Various known techniques can be used to determine the material and structure of containers C1 to C11 depending on the waste to be stored, and therefore detailed explanations will be omitted here.

[0037] In addition, in this embodiment, an example is shown in which waste stored in containers C1 to C11 is identified using physical tags PT1 to PT11 included in the imaging range IM1 of the imaging device 200. The physical tags PT1 to PT11 will be described in detail with reference to FIG. 2. Also, configuration examples of the information processing device 100 and the imaging device 200 will be described in detail with reference to FIG. 6. Also, the bounding box BB1 and the like will be described in detail with reference to FIGS. 10 to 13 and the like.

[0038] [Physical tag configuration example] FIG. 2 is a top view showing an example of the configuration of the physical tag PT1. Here, only the physical tag PT1 is shown as a representative example, but the same applies to the other physical tags PT2 to PT11. Furthermore, this embodiment shows an example in which the physical tags PT1 to PT11 are used to acquire the attributes and positions of the target object (waste) (or the attributes and positions of the corresponding containers C1 to C11). Note that the physical tags PT1 to PT11 can be changed as appropriate depending on the positions at which they are attached to the containers. For example, when the physical tag PT11 is attached to the edge of the opening of the container C11, the physical tag PT11 can be configured along the circumferential edge.

[0039] FIG. 2 shows an example of a color code in which five rectangles of a predetermined color are arranged in a row as the physical tag PT1. This color code can be configured by coloring the rectangle R1 at the end in the arrangement direction black, and coloring the other four rectangles R2 to R5 other than the black rectangle R1 in a color other than black. In this case, it is possible to represent multiple pieces of information by changing the colors of the four rectangles R2 to R5 arranged in a row. For example, an arrangement in which rectangle R2 is blue, rectangle R3 is red, rectangle R4 is yellow, and rectangle R5 is green can be used as a color code representing a "plastic bag."

[0040] Note that, in addition to the color code shown in FIG. 2, other physical tags readable by the imaging device 200 may be used, or other wireless tags may be used. Examples of other physical tags readable by the imaging device 200 are shown in FIGS. 3 to 5. Ordinary barcodes and two-dimensional codes (e.g., QR Code (registered trademark)) may also be used as physical tags. Examples of wireless tags are shown in FIGS. 20 to 22. These physical tags are associated with unique identification information (tag information). For example, the information processing device 100 stores a tag DB (Data Base) 140 indicating the relationship between physical tags and tag information in the storage unit 130 (see FIG. 6), and can acquire tag information associated with the read physical tag using the tag DB 140. Note that the tag DB 140 may be stored in an external device and acquired from the external device for use.

[0041] For example, the physical tag PT1 can be a sheet-like member (not shown) that is attached to the object, and five rectangles R1 to R5, each colored a specific number, can be arranged in a line on this sheet-like member. The rectangles R1 to R5 have the same length L1 in the arrangement direction. The lengths CR1 to CR4 between the rectangles R1 to R5 in the arrangement direction are also the same. The lengths L1 in the arrangement direction may be different from each other, and the lengths CR1 to CR4 in the arrangement direction may also be different from each other. The lengths CR1 to CR4 in the arrangement direction may also be zero.

[0042] [Barcode configuration example] FIG. 3 is a top view showing an example of the configuration of a physical tag PT20 relating to a barcode.

[0043] FIG. 3 shows an example of a color code in which multiple rectangles are arranged in a row as the physical tag PT20. This color code can be configured by placing standard bars (rectangle R21 of length L11 and rectangle R26 of length L12) at both ends to indicate the direction of the arrangement, and arranging four bars of two different lengths (long bars, short bars) between them. Note that the arrangement may also be configured with three types of information: bars of two different lengths (long bars, short bars) and blank spaces where no bars are arranged. In this case, multiple pieces of information can be represented by changing the bars of two different lengths (long bars, short bars) and blank spaces arranged between the standard bars at both ends (rectangles R21, R26).

[0044] 3 shows an example of arranging a long bar (rectangle R22 of length L13), a short bar (rectangle R23 of length L14), a short bar (rectangle R24 of length L14), and a long bar (rectangle R25 of length L13). For example, this arrangement can be used as a barcode representing a "plastic bag." In this way, the physical tag PT20 can represent multiple pieces of information by using the horizontal length of each rectangle.

[0045] [Example of shape code configuration] FIG. 4 is a top view showing an example of the configuration of a physical tag PT30 relating to a graphic code.

[0046] FIG. 4 shows an example of a graphic code in which four types of graphics (circle (oval), triangle, square, inverted triangle) are arranged in a row as the physical tag PT30. This graphic code can be configured by placing standard bars (rectangles R31, R36) at both ends to indicate the direction of the arrangement, and arranging four of the four types of graphics between them. The four types of graphics may also be colored. In this case, multiple pieces of information can be represented by changing the four types of graphics arranged between the standard bars (rectangles R31, R36) at both ends and the colors assigned to them.

[0047] 4 shows an example of arranging a blue circle (oval) graphic R32, a red triangle graphic R33, a yellow square graphic R34, and a green inverted triangle graphic R35. For example, this arrangement can be used as a graphic code representing a "plastic bag." In this way, the physical tag PT30 can represent multiple pieces of information by combining multiple types of graphics and their colors.

[0048] [Example of shape code for changing vertical length] FIG. 5 is a top view showing an example of the configuration of a physical tag PT40 relating to a graphic code whose length in the up-down direction (vertical direction, height direction) is changed.

[0049] FIG. 5 shows an example of a graphic code as a physical tag PT50, in which four types of graphics (circle (oval), triangle, square, inverted triangle) whose lengths can be changed in two directions (vertical and height directions) are arranged in a row. This graphic code can be configured by placing standard bars (rectangles R41 and R46) at both ends to indicate the direction of the arrangement, and arranging four of the four types of graphics whose lengths can be changed in the vertical direction between them. Furthermore, each of the four types of graphics may be assigned a color. In this case, multiple pieces of information can be represented by changing the four types of graphics arranged between the standard bars (rectangles R41 and R46) at both ends, the colors assigned to them, and the vertical lengths.

[0050] 5 shows an example in which a circle (oval) R42 with a long vertical line of blue, a triangle R43 with a short vertical line of red, a square R44 with a short vertical line of yellow, and an inverted triangle R45 with a long vertical line of green are arranged. For example, this arrangement can be used as a graphic code representing a "plastic bag." In this way, the physical tag PT40 can represent multiple pieces of information by combining multiple types of shapes, their colors, and their vertical lengths.

[0051] [Example of information processing system configuration] FIG. 6 is a block diagram showing an example of the system configuration of the information processing system 1. As shown in FIG.

[0052] The information processing system 1 is composed of multiple devices that can be connected via a network N1. FIG. 6 shows an example of the information processing system 1 including an information processing device 100, an imaging device 200, and a user terminal 300. The information processing device 100, the imaging device 200, and the user terminal 300 are each connected to the network N1 by a communication method using wired communication or wireless communication. The network N1 is a network such as a public line network or the Internet. In this case, the wireless communication may be a mobile communication network (e.g., standards such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), and 6G (6th Generation)). Alternatively, at least one of wireless communication standards such as wireless LAN (e.g., Wi-Fi (Wireless Fidelity)), Bluetooth (registered trademark), and ZigBee (registered trademark) may be used. In addition, multiple frequency bands (e.g., UHF band and 2.4 GHz band) may be used in combination.

[0053] Note that the information processing device 100, the imaging device 200, and the user terminal 300 may be directly connected using wired or wireless communication without going through the network N1. Also, although only the imaging device 200 and the user terminal 300 are shown as representative examples in Fig. 6, other imaging devices and other user terminals installed at various locations may also constitute the information processing system 1.

[0054] [Configuration example of information processing device] The information processing device 100 includes a communication unit 110, a control unit 120, and a storage unit 130. The information processing device 100 can be, for example, a server realized by one or more devices.

[0055] The communication unit 110, under the control of the control unit 120, exchanges various types of information with other devices using wired or wireless communication.

[0056] The control unit 120 controls each unit of the information processing device 100 based on a control program stored in the storage unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Specifically, the control unit 120 includes an image acquisition unit 121, a tag information acquisition unit 122, a setting unit 123, an annotation unit 124, and a generation unit 125.

[0057] The image acquisition unit 121 acquires the captured image (including image data that has been subjected to image processing and its accompanying information) generated by the imaging device 200 via the communication unit 110. Then, the image acquisition unit 121 outputs the acquired captured image to the tag information acquisition unit 122 and the setting unit 123.

[0058] Here, the object shown in this embodiment includes an object in which a regular object and an irregular object are mixed. For example, when the object is PET (Polyethyleneterephthalate) resin, it is assumed that a regular object such as a PET bottle and an irregular object such as a roll or plate are mixed. Furthermore, the measurement data of the object shown in this embodiment also includes data measured in a state in which one or more regular or irregular objects are present. For example, when the object is PET resin, it is assumed that one or more regular objects such as a PET bottle are mixed with one or more irregular objects (e.g., roll or plate). Furthermore, it is assumed that the regular objects include a mixture of normal regular objects (e.g., uncrushed PET bottles) and compressed regular objects (e.g., crushed PET bottles).

[0059] The tag information acquisition unit 122 reads a physical tag from the captured image output from the image acquisition unit 121 and acquires tag information based on the read physical tag. Then, the tag information acquisition unit 122 associates the position of the read physical tag (the position in the captured image) with the acquired tag information and outputs them to the setting unit 123, the annotation unit 124, and the generation unit 125. For example, when using the physical tag PT1 shown in FIG. 2, it is possible to read the physical tag PT1 from the captured image using a known image recognition technique (e.g., an object detection technique using pattern matching). Then, the tag information acquisition unit 122 acquires tag information associated with the read physical tag PT1 using the tag DB 140. As described above, the tag DB 140, which indicates the relationship between the physical tag and the tag information, is stored in the storage unit 130.

[0060] The setting unit 123 sets a predetermined image area from the captured image output from the image acquisition unit 121 based on the tag information acquired by the tag information acquisition unit 122, and outputs setting information regarding the set image area to the annotation unit 124 and the generation unit 125. This setting information includes the position, size, shape, etc. of the image area in the captured image. For example, assume a case where an image area corresponding to waste contained in a container C1 is set from a captured image corresponding to an imaging range IM1 (see FIG. 1). In this case, the setting unit 123 identifies the edge of the opening of the container C1 in the captured image based on information about the physical tag PT1 output from the tag information acquisition unit 122 (the position (base point), length, and direction (angle with respect to a specific direction) of the physical tag PT1 in the captured image) and tag information output from the tag information acquisition unit 122 (tag information corresponding to the physical tag PT1).

[0061] Specifically, the setting unit 123 extracts image area information 153 corresponding to the tag information (tag information corresponding to physical tag PT1) output from the tag information acquisition unit 122 from the setting information DB 150 (see FIG. 7). Then, the setting unit 123 identifies the edge of the opening of the container C1 based on the extracted image area information and information related to the physical tag PT1 (the position, length, and angle described above). Next, the setting unit 123 identifies an image area including the identified edge of the opening of the container C1, and sets the image area in the captured image. These setting methods will be described in detail with reference to FIGS. 10 to 13, etc.

[0062] The annotation unit 124 performs annotation on an object included in the captured image output from the image acquisition unit 121 based on the tag information output from the tag information acquisition unit 122. Then, the annotation unit 124 outputs the annotated image to the generation unit 125. For example, the annotation unit 124 tags an image area corresponding to the setting information output from the setting unit 123 with the attribute of the object, the state of the object, etc. corresponding to the tag information output from the tag information acquisition unit 122. Note that, as shown in FIG. 14 , the state of the object can be manually input by the user. The method of performing annotation will be described in detail with reference to FIG. 15 .

[0063] Here, annotation means adding various information to data. For example, the process of adding attribute information of an object to a captured image (image data) including the object can be called annotation. Note that various information added to data is sometimes called a label. Also, adding various information to data is sometimes called tagging. Also, annotated data (tagged data) is sometimes called training data. As will be described later, it is possible to generate a trained model by machine learning using training data.

[0064] The generation unit 125 generates a trained model using an image annotated by the annotation unit 124, and stores the generated trained model in the trained model DB 160. Note that a known generation method can be used to generate the trained model.

[0065] Here, the trained model is an AI (Artificial Intelligence) model that has been machine-learned using teacher data or training data. Examples of the trained model that can be used include SVM (Support Vector Machine), CNN (Convolutional Neural Network), ViT (Vision Transformer), and YOLO (You Only Look Once).

[0066] Furthermore, the trained model shown in this embodiment is generated by learning using images of waste stored in a container, and is capable of detecting, for example, the state (e.g., weight, filling rate) of the waste stored in the container. For example, a trained model can be generated by learning a large amount of training data in which example problems and corresponding correct answers are associated. Furthermore, when new input data is input to the trained model, it is possible to output output data that is the correct answer based on the learning results of the example problems and the corresponding correct answers.

[0067] For example, when learning the filling rate as the state of waste, it is possible to generate a learned model by learning a large amount of tagged data (teacher data) in which the relationship between an image including the target waste (waste contained in a container), the attributes of the waste, and the filling rate of the waste in the container is associated. For example, when the filling rate when there is no waste in the container is set to 0, the filling rate when the container is full of waste is set to 1, and the filling rate of the waste in other containers is learned as a value between 0 and 1. In this case, a value between 0 and 1 is output as the output result of the learned model. That is, when there is no waste in the container, a filling rate of 0 is output, and when the container is full of waste, a filling rate of 1 is output. Also, when there is waste in the container and it is not full, a numerical value t (0 < t < 1) corresponding to the filling rate is output. Similarly, when learning the weight as the state of waste, it is possible to generate a learned model by learning a large amount of tagged data (teacher data) in which the relationship between an image including the target waste (waste contained in a container) and the weight of the waste in this container is associated.

[0068] Note that when generating a learned model, it is possible to cut out a rectangular image region including one container (including the waste) in which the target waste is contained from the captured images of each waste contained in each of a plurality of containers, and use the cut-out image. For example, as shown in FIG. 1, when waste is contained in 11 containers, it is possible to cut out 11 images for each container. Also, for example, as shown in FIG. 1, when 7 types of waste (plastic bags, paper scraps, PP resin, hard plastic, glass scraps, iron scraps, substrates) are contained in 11 containers, it is possible to generate a learned model according to the 7 types of waste. Then, when estimating the state of waste, it is possible to select a learned model according to the type of waste to be estimated, and use the selected learned model to output the state of the waste to be estimated.

[0069] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (e.g., a control program, a tag DB 140, a setting information DB 150 (see FIG. 7), and a trained model DB 160) that are required for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 110. The storage unit 130 can be, for example, a read-only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0070] [Configuration example of imaging device] The imaging device 200 includes a communication unit 201, a control unit 202, a storage unit 203, and an imaging unit 204. As the imaging device 200, for example, a remote camera that can be remotely controlled can be used.

[0071] The communication unit 201 exchanges various types of information with the information processing device 100 using wireless communication under the control of the control unit 202 .

[0072] The control unit 202 controls each unit of the imaging device 200 based on a control program stored in the storage unit 203. The control unit 202 is realized by a processing device such as a CPU or a GPU. For example, the control unit 202 executes control to associate the captured image generated by the imaging unit 204 with imaging device identification information and transmit the image to the information processing device 100. Based on the imaging device identification information, the information processing device 100 can identify that the captured image was generated at the waste collection site WP1.

[0073] The storage unit 203 is a storage medium that stores various types of information. For example, the storage unit 203 stores various types of information (for example, a control program, imaging device identification information for identifying the imaging device 200) that is required for the control unit 202 to perform various processes. The storage unit 203 also stores various types of information acquired via the communication unit 201. The storage unit 203 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0074] The imaging unit 204 captures an image of a subject under the control of the control unit 202 to generate an image (image data), and outputs the generated image to the control unit 202. The imaging unit 204 is configured, for example, with an imaging element (image sensor) that receives light from the subject collected by lenses (e.g., multiple lenses that collect light from the subject), and an image processing unit that performs predetermined image processing on the image data generated by the imaging element. As the imaging element, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type imaging element can be used. For example, the imaging unit 204 captures an image of a subject included in an imaging range IM1 (see FIG. 1 ) to generate a captured image, and outputs the captured image to the control unit 202. The imaging unit 204 may also generate still images periodically or irregularly, or may generate moving images.

[0075] [Example of user terminal configuration] The user terminal 300 includes a communication unit 301 , a control unit 302 , a storage unit 303 , an operation unit 304 , a sound acquisition unit 305 , and a display unit 306 .

[0076] The communication unit 301 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 202.

[0077] The control unit 302 controls each unit based on various programs stored in the storage unit 303. The control unit 302 is realized by a processing device such as a CPU or a GPU. For example, the control unit 302 executes control to transmit status information (e.g., filling rate, weight) input via the operation unit 304 to the information processing device 100.

[0078] The storage unit 303 is a storage medium that stores various types of information. For example, the storage unit 303 stores various types of information (for example, control programs and applications) required for the control unit 202 to perform various processes. The storage unit 303 also stores various types of information acquired via the communication unit 301. The storage unit 303 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0079] The operation unit 304 receives various operations from the user U1 and outputs the received operation contents to the control unit 302. For example, when a user operation to input status information (e.g., filling rate, weight) regarding the status of waste contained in each of the multiple containers C1 to C11 installed at the waste collection site WP1 (see FIG. 1) is received, the operation unit 304 outputs the input information to the control unit 302.

[0080] The sound acquisition unit 305 acquires sounds near or around the user terminal 300 under the control of the control unit 302. As the sound acquisition unit 305, for example, one or more microphones can be used.

[0081] The display unit 306 is a display unit that displays various images under the control of the control unit 302. For example, a display panel such as an organic EL (Electro Luminescence) panel or an LCD (Liquid Crystal Display) panel can be used as the display unit 306. For example, it is possible to display a display screen 310 (see FIG. 14 ) on the display unit 306 for inputting the state of an object to be used for annotation when generating a trained model.

[0082] [Example of settings information DB content] 7 is a diagram schematically showing each piece of information stored in the setting information DB 150. The setting information DB 150 is a database for managing each piece of information used when performing annotation on the waste contained in the containers C1 to C11.

[0083] The setting information DB 150 stores tag information 151, attribute information 152, and image area information 153 in association with each other. Note that each of these pieces of information is an example, and other information may be stored in the setting information DB 150.

[0084] The tag information 151 is tag information acquired based on the physical tags PT1 to PT11 attached to the containers C1 to C11. In Fig. 7, an example is shown in which the tag information corresponding to the physical tag PT1 attached to the container C1 is "TG001," the tag information corresponding to the physical tag PT2 attached to the container C2 is "TG002," and the tag information corresponding to the physical tag PT11 attached to the container C11 is "TG011." Note that other tag information is omitted from the illustration.

[0085] The attribute information 152 is attribute information that indicates the attributes of the object (waste) contained in a container that is provided with a physical tag that corresponds to the tag information stored in the tag information 151. Note that the attributes shown in FIG. 7 are just an example, and other attributes relating to multiple classifications, other subdivided attributes, etc. may be stored. For example, if the object is a resin such as PET, since there is a mixture of standardized objects such as PET bottles and non-standardized objects such as rolls or plates, these may be classified and set as different attributes. Note that the attribute information and tag information correspond to each other, and therefore the attribute information may be included in the tag information.

[0086] Image area information 153 is information used when tag information stored in tag information 151 is acquired and an image area corresponding to the tag information is set. This image area information will be described in detail with reference to Figs. 8A, 8B, 10 to 13, etc. The image area to be set is sometimes called a bounding box. This bounding box refers to an image area that surrounds an object included in a captured image. When a wireless tag is used as a physical tag (see Figs. 20 to 22), each setting information such as attribute information 152 and image area information 153 may be stored in the wireless tag. In this case, the information processing device 100 can acquire and use each setting information stored in the wireless tag.

[0087] [Image area information] 8A and 8B are diagrams schematically showing image region information stored in image region information 153 (see FIG. 7). 8A and 8B show an example in which a figure for a bounding box is set in advance, and the bounding box is set based on this figure and information about the physical tag (for example, the position, length, and angle of the physical tag in the captured image).

[0088] 8A shows an example of constructing image area information based on the relationship between the shape of the edge SE1 of the opening of the container C1 when viewed from above and the physical tag PT1 attached to the edge SE1. The same applies to the relationship between the other containers C2 to C10, whose edges SE1 when viewed from above are rectangular, and the physical tags PT2 to PT10.

[0089] 8B shows an example of constructing image area information based on the shape of the edge SE11 of the opening of the container C11 when viewed from above and the relationship with the physical tag PT11 attached to the edge SE11 of the opening of the container C11. The same applies to the relationship between other containers and physical tags whose edge SE1 has a circular shape when viewed from above.

[0090] 8A, the shape of the edge of the opening of the container C1 to which the physical tag PT1 is attached (the shape of the edge when viewed from above) can be identified based on the physical tag PT1. Here, when using a captured image (a captured image from directly above the container C1) generated by the imaging device 200 provided directly above the container C1, it is assumed that the shape of the edge of the opening of the container C1 included in the captured image matches the shape of the edge SE1 associated with the physical tag PT1. Therefore, the shape of the edge of the opening of the container C1 to which the physical tag PT1 is attached can be identified based on the physical tag PT1.

[0091] However, when using a captured image generated by the imaging device 200 disposed diagonally above the container C1 (a captured image captured from diagonally above the container C1), it is assumed that the shape of the edge of the opening of the container C1 included in the captured image does not match the shape of the edge SE1 associated with the physical tag PT1. Therefore, in this case, the shape of the edge SE1 associated with the physical tag PT1 can be transformed based on the state of the physical tag PT1 included in the captured image (e.g., the position, length, and angle of the physical tag in the captured image), and the transformed shape can be identified as the shape of the edge of the opening of the container C1. For example, an affine transformation can be used for this transformation. Examples of this are shown in FIGS. 9A to 9D.

[0092] Affine Transformation 9A to 9D are diagrams showing an example of transition when a square figure S1 is subjected to affine transformation. In Fig. 9A to 9D, figures S2 to S5 after affine transformation are shown by thick lines.

[0093] Here, when affine transforming each coordinate (x, y) in two-dimensional space to coordinate (x', y'), the following formula is used: Therefore, it is possible to affine transform each coordinate (x, y) in a plane image, which is two-dimensional space, to coordinate (x', y') using the following formula.

[0094]

number

[0095] Figure 9A shows an example of a transition when translating a figure S1 into a square figure S2. When translating the figure S1 in this way, the following affine matrix is ​​used. Note that tx and ty are parameters that specify the distance when translating.

[0096]

number

[0097] Figure 9B shows an example of a transition when scaling the figure S1 to convert it into a square figure S3. When scaling the figure S1 in this way, the following affine matrix is ​​used. Note that sx and sy are parameters that specify the scaling ratio when scaling.

[0098]

number

[0099] 9C shows an example of a transition when rotating the figure S1 to convert it into a square figure S4. When rotating the figure S1 in this way, the following affine matrix is ​​used. Note that θ is a parameter that specifies the angle when rotating.

[0100]

number

[0101] FIG. 9D shows an example of the transition when shearing (skewing) the figure S1 to transform it into a parallelogram figure S5. When shearing and transforming the figure S1 in this way, the following affine matrix is ​​used. Note that sh x , sh y is a parameter that specifies the distance at which shearing occurs.

[0102]

number

[0103] [Bounding box setting example] FIG. 10 is a diagram schematically showing an example of a setting method for setting a bounding box that includes waste (plastic bags) contained in a container C1 included in the imaging range IM1 (see FIG. 1).

[0104] As shown in FIG. 10(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. Specifically, the tag information acquisition unit 122 extracts the physical tags PT1 to PT11 included in the captured image using a known image recognition technique as described above. Next, the tag information acquisition unit 122 acquires tag information corresponding to each of the physical tags PT1 to PT11 extracted from the captured image based on the arrangement order of the colors constituting the physical tags PT1 to PT11 as described above. As described above, the tag information corresponding to each of the physical tags PT1 to PT11 can be acquired using the tag DB 140 (see FIG. 6) that indicates the relationship between physical tags and tag information.

[0105] Next, the setting unit 123 (see FIG. 6) extracts, from the setting information DB 150, image area information 153 (see FIG. 7) corresponding to the tag information (corresponding to the physical tag PT1) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 8A, the shape of a rectangular edge SE1 having the physical tag PT1 provided on one side is extracted.

[0106] As shown in FIG. 10(B), based on the physical tag PT1 extracted from the captured image (corresponding to the imaging range IM1), the setting unit 123 fits the extracted image region information (the shape of the rectangular edge SE1) to the shape of the edge of the opening of the container C1 included in the captured image. For example, the setting unit 123 identifies both ends E1' and E2' of the physical tag PT1 attached to the container C1 included in the imaging range IM1. Then, the setting unit 123 performs affine transformation on the rectangular edge SE1 so that both ends E1 and E2 of the rectangular edge SE1 coincide with both ends E1' and E2' of the physical tag PT1 included in the captured image. This affine transformation will be described in detail with reference to FIG. 11.

[0107] 10(C) shows an example in which the rectangular edge SE1 is affine transformed so that both ends E1 and SE2 of the rectangular edge SE1 coincide with both ends E1' and E2' of the physical tag PT1 included in the captured image (corresponding to the imaging range IM1). That is, the rectangular edge SE1 is affine transformed so that the rectangular edge SE1 coincides with the edge of the opening of the container C1 included in the captured image.

[0108] 10(D1), the setting unit 123 fits the affine-transformed rectangular edge SE1 to the edge of the opening of the container C1 included in the imaging range IM1 based on the position (base point) of the physical tag PT1 in the captured image. Then, the setting unit 123 sets an image area of ​​a predetermined size including the fitted rectangular edge SE1 in the captured image (corresponding to the imaging range IM1). This image area can be used as a bounding box BB1.

[0109] The size of the bounding box BB1 can be set appropriately depending on the size of the fitted rectangular edge SE1. For example, it is possible to select and use the smallest size that can contain the fitted rectangular edge SE1 from among a plurality of preset rectangle sizes. Also, for example, the maximum size (e.g., horizontal length) of the fitted rectangular edge SE1 may be used as a reference, and the size of the rectangle that contains that maximum size may be selected. Note that while FIG. 10(D1) shows an example of setting a rectangular bounding box BB1, other shapes may also be set as the bounding box.

[0110] For example, as shown in Fig. 10(D2), the shape of the fitted rectangular edge SE1 may be set as a bounding box BB2. Alternatively, the bounding box may be set based on the position and size of the physical tag PT1. For example, it is possible to set a rectangular bounding box centered on the position of the physical tag PT1 and with a rectangular size corresponding to the length of the physical tag PT1.

[0111] In this way, the bounding boxes BB1 and BB2 can be set based on the position, size, shape (degree of deformation), etc. of the physical tag PT1. In this case, as described above, the image area can be set to any shape, figure, etc.

[0112] [Affine transformation example] FIG. 11 is a diagram showing an example of transformation when affine transformation is performed on image region information (the shape of rectangular edge SE1) so as to match the edge of the opening of container C1.

[0113] For example, assume that the position of container C1 and the position and orientation (imaging direction) of imaging device 200 are fixed at waste collection site WP1. In this case, the distance from imaging device 200 to container C1 can be represented by r, and the position of container C1 relative to imaging device 200 can be represented by depression angle θ and azimuth angle φ. Furthermore, the spherical coordinates (r, θ, φ) can be converted to Cartesian linear coordinates (x, y, z) using the following equations.

[0114]

number

[0115] Furthermore, the opening surface of container C1 (the surface including the edge of the opening of container C1, the surface including physical tag PT1) is assumed to be on plane P (a virtual plane parallel to the floor surface). The height from the floor surface at the installation location of container C1 to the edge of the opening of container C1 is assumed to be h. Furthermore, it is assumed that the setting unit 123 is capable of grasping in advance the relationship between the three-dimensional space (three-dimensional coordinates) at waste collection site WP1 and the two-dimensional coordinates corresponding to the imaging range IM1.

[0116] In this case, the setting unit 123 identifies both ends E1', E2' of the physical tag PT1 acquired by the tag information acquisition unit 122, and calculates the length D1 of both ends E1', E2'. Next, the setting unit 123 calculates the length D2 (the length of ends E1', E3') of one side of the container C1 on which the physical tag PT1 is provided, based on the ratio (T1 / T2) of the length T1 (see FIG. 8A) of the physical tag PT1 in the image region information (the shape of the rectangular edge SE1) to the length T2 (see FIG. 8A) of one side on which the physical tag PT1 is provided. That is, the length D2 (=D1·T2 / T1) of the ends E1' and E3' on the plane P is calculated. This allows the setting unit 123 to identify the end E3' of the physical tag PT1 in three-dimensional space.

[0117] Next, the setting unit 123 changes the size of the image region information (the shape of the rectangular edge SE1) (see FIG. 8A) so that the length D2 of the ends E1' and E3' on the plane P matches the length T2 of the ends E1 and E3 of the image region information (the shape of the rectangular edge SE1). For example, the size of the image region information (the shape of the rectangular edge SE1) can be changed by an affine transformation (a transformation that enlarges or reduces a figure) shown in FIG. 10B. Then, the setting unit 123 draws the image region information (the shape of the rectangular edge SE1) after the size conversion on the plane P. This allows the setting unit 123 to identify both ends E4' and E5' of the physical tag PT1 in three-dimensional space.

[0118] The setting unit 123 can also identify the ends E1', E3', E4', and E5' of the physical tag PT1 included in the captured image corresponding to the imaging range IM1 based on the relationship between the three-dimensional space (three-dimensional coordinates) at the waste collection site WP1 and the two-dimensional coordinates corresponding to the imaging range IM1. Therefore, the setting unit 123 can calculate the crushed (sheared) area of ​​the opening of the container C1 in the captured image based on the relationship between the line segment between the ends E1' and E3' and the line segment between the ends E4' and E5' of the physical tag PT1 included in the captured image. That is, the shearing amount of the rectangle corresponding to the opening of the container C1 in the captured image can be calculated. The scaling size can be calculated based on the relationship between the length D1 of the line segment between the ends E1' and E2' of the physical tag PT1 and the length T1 of the line segment between the ends E1 and E2 of the image area information (the shape of the rectangular edge SE1). The rotation angle can be calculated based on the angle of the line segment between the ends E1' and E2' of the physical tag PT1 relative to a specific direction (e.g., the horizontal direction in the captured image).

[0119] Using the shear amount, scaling size, and rotation angle thus determined, it is possible to perform affine transformation so that the image region information (the shape of the rectangular edge SE1) coincides with the edge of the opening of the container C1, as shown in Figure 10(C). Furthermore, the image region information (the shape of the rectangular edge SE1) thus affine transformed is placed in the captured image so that the reference points (e.g., end E1' of the physical tag PT1 and end E1 of the physical tag PT1) coincide, as shown in Figures 10(D1) and 10(D2). This allows the bounding box BB1 or BB2 to be set.

[0120] In this way, if the position and imaging direction of the imaging device 200 are fixed and the size of the container C1 can be known in advance, the shear amount can be calculated based on the depression angle and azimuth angle.

[0121] The above example shows how affine transformation is performed to fit image area information (rectangular or circular edge shape) to each container based on the relationship between the three-dimensional space (three-dimensional coordinates) at waste collection point WP1 and the two-dimensional coordinates corresponding to imaging range IM1. It is also possible to measure container C1 included in imaging range IM1 in advance and calculate the shear amount of the opening (rectangular) edge of container C1 in advance based on the measurement results. In this case, the calculation results can be used to perform affine transformation.

[0122] Furthermore, for example, affine transformation can be performed based on the length (size) D1 and angle (e.g., angle relative to the horizontal direction) of both ends E1', E2' of the physical tag PT1. That is, affine transformation can be performed so that the length and angle of both ends E1, E2 of the rectangular edge SE1 coincide with the length and angle of both ends E1', E2' of the physical tag PT1. For example, using the difference value (e.g., angle, length) between end E2 and end E2' when end E1 and end E1' are coincident, an affine transformation can be performed by setting an affine matrix so that the difference value between end E2 and end E2' becomes 0. For each of these affine transformations, a known transformation method can be used.

[0123] The edge of the opening of the container C1 can be detected using a known image recognition technology (e.g., edge detection technology). Therefore, the angle between one of the four edges of the container C1, on which the physical tag PT1 is provided, and another adjacent edge can be obtained. The angle between these two adjacent edges can be used to estimate the shape (degree of deformation) of the rectangle corresponding to the edge of the opening of the container C1. For example, when using a captured image (a captured image from directly above the container C1) generated by the imaging device 200 provided directly above the container C1, the rectangle corresponding to the edge of the opening of the container C1 included in the captured image is rectangular. Therefore, a 90-degree angle can be obtained as the angle between the one side on which the physical tag PT1 is provided and the other adjacent side. On the other hand, when using a captured image (a captured image from diagonally above the container C1) generated by the imaging device 200 provided diagonally above the container C1, the rectangle corresponding to the edge of the opening of the container C1 included in the captured image has a sheared rectangular shape. In this case, a value less than 90 degrees is acquired as the angle between one side on which the physical tag PT1 is provided and another side adjacent thereto.

[0124] By providing physical tags on two or more of the four sides of the container C1, it is possible to calculate the shear amount by calculating the angles of the two or more physical tags without detecting the edge of the opening of the container C1. Examples of providing physical tags on two or more of the four sides of the container C1 are shown in Figures 12, 17A and 17B, 18A and 18B.

[0125] Furthermore, if the positions and orientations of the containers C1 to C11 and the position and orientation (imaging direction) of the imaging device 200 are fixed at the waste collection site WP1, the shapes of the edges of the openings of the containers C1 to C11 included in the imaging range IM1 can be acquired in advance by measurement or the like. Therefore, in such a case, the shapes of the edges of the openings of the containers C1 to C11 included in the imaging range IM1 (deformed rectangular shapes) can be measured and acquired in advance and stored in the image region information 153 (see FIG. 7) corresponding to the tag information 151 (see FIG. 7). For example, the graphic information shown in FIG. 10(C) can be stored in the image region information 153 as the image region information corresponding to the container C1. This makes it possible to omit the above-mentioned processes such as affine transformation, and the graphic information stored in the image region information 153 can be used as is.

[0126] The above example illustrates the affine transformation of the rectangular edge SE1 based on the length and angle of both ends E1', E2' of the physical tag PT1. However, the affine transformation may also be performed based on other information. For example, the rectangular edge SE1 after the affine transformation may be compared with the edge of the opening of the container C1, and whether or not to use the affine-transformed rectangular edge SE1 may be determined based on the degree of similarity. For example, if the degree of similarity is equal to or greater than a threshold, the affine-transformed rectangular edge SE1 is used. If the degree of similarity is less than the threshold, the affine-transformed rectangular edge SE1 is again affine-transformed. The edge of the opening of the container C1 can be detected using known image recognition technology (e.g., edge detection technology). Furthermore, known image recognition technology (e.g., a determination technology based on brightness difference values) can be used to compare images.

[0127] [Example of physical tags on three sides] 11 shows an example of affine transformation in which one physical tag PT1 is attached to one side of one container C1. However, one container may be attached with multiple physical tags. This makes it possible to simplify the calculation of the affine transformation.

[0128] 12 is a diagram showing an example in which physical tags PT1A, PT1B, and PT1C are provided on three sides of the edge of a container C1. In this case, image area information (the shape of the rectangular edge SE1A) corresponding to the physical tags PT1A, PT1B, and PT1C provided on the container C1 is stored in the image area information 153 (see FIG. 7).

[0129] 12 shows a perspective view of the container C1 on the left side, and image area information (the shape of the rectangular edge SE1A) on the right side. In this case, similar to the example shown in FIG. 10(B), the setting unit 123 fits the extracted image area information (the shape of the rectangular edge SE1A) to the shape of the edge of the opening of the container C1 included in the captured image based on the physical tags PT1A, PT1B, and PT1C extracted from the captured image (corresponding to the imaging range IM1). In the example shown in FIG. 12, the physical tags PT1A, PT1B, and PT1C are provided on three sides of the container C1, so the setting unit 123 identifies the ends, lengths, and angles (e.g., the angles between the physical tags) of the physical tags PT1A, PT1B, and PT1C provided on the container C1 included in the imaging range IM1. The setting unit 123 then performs affine transformation on the image region information (the shape of the rectangular edge SE1A) so that the physical tags PT1A, PT1B, and PT1C on three sides of the image region information (the shape of the rectangular edge SE1A) match the physical tags PT1A, PT1B, and PT1C included in the captured image. A known affine transformation method can be used for this affine transformation. As shown in FIG. 12, by providing the physical tags PT1A, PT1B, and PT1C on three sides of the container C1, it is possible to calculate the angle by which the rectangle of the container C1 is deformed, which makes it easier to calculate the shear amount.

[0130] [Example of setting a bounding box for a cylindrical container] FIG. 13 is a diagram schematically showing an example of a setting method for setting a bounding box that includes the object (substrate) contained in the container C11 included in the imaging range IM1 (see FIG. 1).

[0131] 13(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. The method for acquiring this tag information is the same as the example shown in FIG.

[0132] Next, the setting unit 123 (see FIG. 6) extracts, from the setting information DB 150, image area information 153 (see FIG. 7) corresponding to the tag information (corresponding to the physical tag PT11) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 8B, the shape of a circular edge SE11 on which the physical tag PT11 is provided is extracted.

[0133] As shown in FIG. 13(B), based on the physical tag PT11 extracted from the captured image (corresponding to the imaging range IM1), the setting unit 123 fits the extracted image region information (the shape of the rectangular edge SE11) to the shape of the edge of the opening of the container C11 included in the captured image. For example, the setting unit 123 identifies both end portions E11′, E12′ and a central portion M13′ of the physical tag PT11 attached to the container C11 included in the imaging range IM1. Then, the setting unit 123 performs affine transformation on the circumferential edge SE11 so that both end portions E11, E12, and the central portion M13 of the circumferential edge SE11 coincide with both end portions E11′, E12′, and the central portion M13′ of the physical tag PT11 included in the captured image. A known transformation method can be used for this affine transformation.

[0134] For example, an affine transformation can be performed based on the circumferential shape specified by the lengths (sizes) of both ends E11', E12' of the physical tag PT11 and the central portion M13'. That is, an affine transformation can be performed so that the circumferential shape specified by both ends E11, E12 and the central portion M13 of the circumferential edge SE11 matches the circumferential shape specified by both ends E11', E12' and the central portion M13' of the physical tag PT11. The method for calculating the shear amount of the opening (circular shape) of the container C11 is the same as in FIGS. 10 and 11.

[0135] 13(C), the circumferential edge SE11 is affine transformed so that both ends E11, E12 and a central portion M13 of the rectangular shape of the edge SE11 coincide with both ends E11', E12' and a central portion M13' of the physical tag PT11 included in the captured image (corresponding to the imaging range IM1). That is, the circumferential edge SE11 is affine transformed so that the circumferential edge SE11 coincides with the edge of the opening of the container C11 included in the captured image.

[0136] Next, as shown in FIG. 13(D), the setting unit 123 fits the affine-transformed circumferential edge SE11 to the edge of the opening of the container C11 included in the imaging range IM1 based on the position (base point) of the physical tag PT11 in the captured image. Then, the setting unit 123 sets an image area of ​​a predetermined size including the fitted circumferential edge SE11 in the captured image (corresponding to the imaging range IM1). This image area can be used as a bounding box BB11. Alternatively, a bounding box may be set in the same manner as in FIG. 10(D2).

[0137] 10, the circumferential edge SE11 after the affine transformation may be compared with the edge of the opening of the container C11, and whether or not to use the circumferential edge SE11 after the affine transformation may be determined based on the degree of similarity. Also, it is possible to set bounding boxes for edges of other shapes in a similar manner.

[0138] [Example of input using a user terminal] Fig. 14 is a diagram showing an example of inputting the state of waste using the display screen 310 displayed on the display unit 306 of the user terminal 300. Fig. 14 shows an example of inputting the filling rate as the state of waste contained in the containers C1 to C11 installed at the waste collection site WP1 (see Fig. 1). Fig. 15 shows a learning method using the filling rate of the waste input in Fig. 14.

[0139] A container information display area 311, a waste information display area 312, and a filling rate input area 313 are displayed in association with each other on the display screen 310. The display screen 310 can be displayed based on the control of the information processing device 100. The container information display area 311 and the waste information display area 312 can be displayed based on the information stored in the tag DB 140 and the setting information DB 150 of the storage unit 130. FIG. 14 shows an example in which only some (containers C1 to C4) of the containers C1 to C11 installed at the waste collection point WP1 are displayed. The other containers can be displayed using a scroll bar 314.

[0140] Container information for identifying each of the containers C1 to C11 installed at the waste collection point WP1 is displayed in the container information display area 311. Fig. 14 shows an example in which the container information corresponding to container C1 is displayed as "first container," the container information corresponding to container C2 is displayed as "second container," the container information corresponding to container C3 is displayed as "third container," and the container information corresponding to container C4 is displayed as "fourth container."

[0141] The waste information display area 312 displays the names of the waste materials to identify the waste materials contained in the containers C1 to C11 installed at the waste material collection point WP1.

[0142] The filling rate input area 313 displays the filling rate of each waste material input using the operation unit 304. In Fig. 14, the field for the waste material to be input in the filling rate input area 313 is indicated by being surrounded by a thick line 315. Note that information may be input to the filling rate input area 313 based on the user's voice acquired using the sound acquisition unit 305.

[0143] In this way, the inputted filling rate of the waste is transmitted to the information processing device 100 in association with the identification information (tag information 151 (see FIG. 7)) of each waste.

[0144] [Annotation execution example] 15 is a diagram illustrating an example of an annotation method for generating a trained model for waste (plastic bags) contained in a container C1 included in the imaging range IM1 (see FIG. 1). Here, an example is shown in which the attribute of the object (plastic bags) and the filling rate of the object are tagged.

[0145] 15(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. The method of acquiring this tag information is similar to the examples shown in FIGS. 10 to 13, etc.

[0146] Next, the setting unit 123 (see FIG. 6) extracts attribute information 152 and image area information 153 (see FIG. 7) corresponding to the tag information (corresponding to physical tag PT1) acquired by the tag information acquisition unit 122 from the setting information DB 150. For example, "plastic bag" is extracted as the attribute corresponding to physical tag PT1, and the shape of edge SE1 shown in FIG. 8A is extracted as the image area information corresponding to physical tag PT1.

[0147] 15(B), the setting unit 123 sets a bounding box BB1 using the extracted image region information. The method for setting this bounding box BB1 is the same as the example shown in FIG.

[0148] Also, in FIG. 15(B), the attribute "plastic bag" extracted as the attribute corresponding to the physical tag PT1 and the filling rate "89%" acquired in response to a user operation (see FIG. 14) are enclosed in a rectangle TG1. The attribute "plastic bag" and the filling rate "89%" shown in the rectangle TG1 are tagged to the image included in the bounding box BB1. That is, FIG. 15 shows an example in which the attribute "plastic bag" and the filling rate "89%" shown in the rectangle TG1 are used as labels. Note that other labels may be added as necessary.

[0149] Next, the annotation unit 124 (see FIG. 6) tags the image included in the bounding box BB1 with the attribute "plastic bag" and the filling rate "89%" shown in the rectangle TG1.

[0150] Next, the generation unit 125 (see FIG. 6) generates a trained model using the image included in the bounding box BB1 tagged with "plastic bag" and a filling rate of "89%" by the annotation unit 124. A known learning method can be used to generate this trained model.

[0151] Here, if the physical tag PT1 falls within the bounding box BB1, there is a risk that the accuracy of the image data to be learned will decrease. Therefore, for example, taking into consideration the image area corresponding to the bounding box BB1, the physical tag PT1 may be attached to the container C1 (or the periphery of the contained waste) so that the physical tag PT1 does not fall within the bounding box BB1. Also, for example, a masking process may be performed to mask the physical tag PT1 included in the bounding box BB1. Note that known image processing can be used for this masking process.

[0152] [Example of operation of information processing device] Fig. 16 is a flowchart showing an example of annotation processing in the information processing device 100. This annotation processing is executed by the control unit 120 (see Fig. 6) based on a program stored in the storage unit 130 (see Fig. 6). Fig. 16 also shows an example of tagging the attributes and filling rate of each waste contained in containers C1 to C11 installed at the waste collection point WP1 (see Fig. 1). This annotation processing will be explained with appropriate reference to Figs. 1 to 15.

[0153] In step S501 , the image acquisition unit 121 acquires a captured image generated by the imaging device 200 .

[0154] In step S502, the tag information acquisition unit 122 reads the physical tags PT1 to PT11 attached to the containers C1 to C11, respectively, based on the captured images acquired in step S501. A known image recognition technique can be used to read these physical tags.

[0155] In step S503, the tag information acquisition unit 122 acquires tag information of the objects (waste) contained in each of the containers C1 to C11 based on each of the physical tags PT1 to PT11 read in step S502.

[0156] In step S504, the setting unit 123 acquires the attributes of the object based on the tag information of the object acquired in step S503. For example, the setting unit 123 extracts and acquires attribute information 152 corresponding to tag information 151 of the object from the setting information DB 150 (see FIG. 7). Furthermore, the annotation unit 124 acquires the state of the object when tagging the state of the object (e.g., filling rate, weight) together with the attribute of the object. For example, as shown in FIG. 14, it is possible to acquire the state of the object (e.g., filling rate, weight) input by a user operation.

[0157] In step S505, the setting unit 123 sets an image area including the object based on the tag information of the object acquired in step S503. The method for setting this image area can be the same as the examples shown in Figs. 10 to 13, etc.

[0158] In step S506, the annotation unit 124 performs annotation to tag the image region of the object set in step S505 with the information about the object acquired in step S504 (attributes of the object, state of the object). Then, the generation unit 125 performs a learning process on the annotated image region of the object.

[0159] These processes may be executed sequentially for each object, or may be executed in parallel for each object.

[0160] The generation unit 125 sequentially stores the trained models generated in step S506 in the trained model DB 160 of the storage unit 130.

[0161] [Modification of physical tags] The above example shows a case where one physical tag is attached to one container. However, multiple physical tags may be attached to one container. This allows for more accurate setting of the bounding box.

[0162] 17A and 17B are diagrams showing an example in which physical tags PT1a and PT1b are provided on two of the edges of a container C1. FIG. 17A shows a perspective view of the container C1, and FIG. 17B shows the shape of the edge SE1a corresponding to the container C1, which is stored in the image region information 153 (see FIG. 7). In this manner, image region information corresponding to the physical tags PT1a and PT1b provided on the container C1 is stored in the image region information 153. The setting unit 123 can also set a bounding box using the positions, lengths, angles, etc. of the physical tags PT1a and PT1b provided on the two edges of the container C1. In this manner, providing two physical tags PT1a and PT1b makes it possible to obtain the angles of the two sides of the container C1, thereby facilitating the calculation of the shear amount.

[0163] 18A and 18B are diagrams illustrating an example in which physical tags PT11a and PT11b are provided at two opposing positions on the edge of a container C11. FIG. 18A illustrates a perspective view of the container C11, and FIG. 18B illustrates the shape of the edge SE11a corresponding to the container C11, which is stored in the image region information 153 (see FIG. 7). In this manner, image region information corresponding to the physical tags PT11a and PT11b provided on the container C11 is stored in the image region information 153. The setting unit 123 can also set a bounding box using the positions, lengths, angles, and the like of the physical tags PT11a and PT11b provided at two opposing positions on the container C11. In this manner, providing two physical tags PT11a and PT11b makes it possible to acquire the positional relationship between the two opposing positions on the container C11, thereby facilitating the calculation of the shear amount.

[0164] [Container Modification] In the above, examples have been shown in which physical tags are provided on box-shaped and cylindrical containers. However, one or more physical tags may be provided on containers of other shapes. Figures 19A and 19B show examples of other shapes.

[0165] 19A and 19B are diagrams showing an example in which a physical tag PT13 is provided on one side of the edge of an L-shaped container C13 when viewed from above. FIG. 19A shows a perspective view of the container C13, and FIG. 19B shows the shape of the edge SE13 corresponding to the container C13, which is stored in the image area information 153 (see FIG. 7). In this way, the image area information corresponding to the physical tag PT13 provided on the container C13 is stored in the image area information 153. Note that the method of setting the image area is the same as in the above-described example. Furthermore, as in FIGS. 12, 17A, 17B, 18A, and 18B, multiple physical tags may be provided on the container C13.

[0166] [Modification of information processing system] The above describes an example in which tag information of the containers C1 to C11 is acquired based on captured images generated by the imaging device 200. Here, the tag information of the containers C1 to C11 may be acquired using wireless communication. Therefore, the following describes an example in which tag information of the containers C1 to C11 is acquired using wireless communication.

[0167] [Example of use of information processing system] Fig. 20 is a diagram showing an example of use of the information processing system 1a. The information processing system 1a shown in Fig. 20 shows an example in which a communication device 400 is added to the information processing system 1 shown in Fig. 1, and physical tags PTT1 to PTT11 capable of wireless communication are provided on the containers C1 to C11. Note that, since other parts are common to the information processing system 1, the same reference numerals as in Fig. 1 are used and their description will be omitted. In the following, the physical tag PTT1 will be mainly described as an example, but the same applies to the physical tags PTT2 to PTT11.

[0168] The physical tag PTT1 is an integrated circuit (IC) tag capable of wireless communication. For example, the physical tag PTT1 can be provided inside a sheet that can be attached to the container C1. The physical tag PTT1 is an example of RFID (Radio Frequency Identification) and is a device capable of low-power wireless communication. The physical tag PTT1 can be configured as a wireless communication tag (RFID tag) compatible with RFID technology. For example, the physical tag PTT1 can be configured as an IC tag that employs a communication method such as Bluetooth Low Energy (BLE), Bluetooth, ZigBee, Low Power Wide Area (LPWA), or Ultra Wide Band (UWB), which are low-power communication modes. The maximum communication distance of the physical tag PTT1 is not particularly limited, but can be, for example, in the range of several tens of centimeters to several meters. For example, if the physical tag PTT1 complies with the BLE standard, it broadcasts packets at predetermined intervals (for example, every 1 to 10 seconds). A packet transmitted by this physical tag PTT1 includes a unique ID (Identification) (tag ID) that is identification information of the IC tag. Figures 20 to 22 show an example in which this tag ID is used as identification information (tag information) of the container C1 (waste).

[0169] That is, by reading the tag ID of the physical tag PTT1 attached to the container C1, it is possible to obtain the identification information of the waste (vinyl bag) contained in the container C1.

[0170] Note that identification information that allows tag information to be acquired from the captured image generated by the imaging device 200 may be provided on the surface of the sheet provided with the physical tag PTT1. For example, triangular identification information PTT1a (see FIG. 21) that allows identification of the base point, size, and direction may be provided. Note that color codes or the like shown in FIGS. 2 to 5 may also be provided.

[0171] [Physical tag configuration example] 21 is a diagram showing an example of identification information PTT1a provided on the surface of a sheet having a physical tag PT1a. Note that the identification information PTT1a can be colored to identify the attributes of the waste contained in the container C1.

[0172] The identification information PTT1a is a figure formed by an isosceles triangle. The center position of the base of the isosceles triangle is defined as a base point PP1, the length of a line segment PP2 from the base point PP1 to the apex angle of the isosceles triangle is defined as a size PPL1 of the identification information PTT1a, and the angle of the line segment PP2 with respect to the horizontal direction is defined as a direction θ1 of the identification information PTT1a. The base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be determined based on the identification information PTT1a included in the captured image. The base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be used to perform affine transformation on the image region information stored in the image region information 153 (see FIG. 7).

[0173] [Examples of using information processing equipment] Fig. 22 is a block diagram showing an example of the system configuration of the information processing system 1a. The information processing system 1a is a partial modification of the information processing system 1 shown in Fig. 6. Specifically, the information processing system 1a is provided with a communication device 400 for reading the physical tag PTT1. Note that apart from the provision of the communication device 400, the information processing system 1a has the same components as the information processing system 1. Therefore, the same reference numerals are used for the components common to the information processing system 1, and some of these components will not be illustrated or described. Furthermore, the components different from the information processing system 1 will be described below as appropriate.

[0174] The information processing system 1a is configured by a plurality of devices that can be connected via a network N1. Fig. 22 shows, as an example, the information processing system 1a including an information processing device 100, an imaging device 200, a user terminal 300, and a communication device 400. Each of the information processing device 100, the imaging device 200, the user terminal 300, and the communication device 400 is connected to the network N1 by a communication method using wired communication or wireless communication.

[0175] Note that the information processing device 100, the imaging device 200, the user terminal 300, and the communication device 400 may be connected directly using wired or wireless communication without going through the network N1. Also, while Fig. 22 shows only the imaging device 200, the user terminal 300, and the communication device 400 as representative examples, the information processing system 1a may also include other imaging devices, other user terminals, and other communication devices installed at each location. Furthermore, as will be described later, if the position of the physical tag PTT1 can be identified by the communication device 400, the installation of the imaging device 200 may be omitted.

[0176] The communication device 400 and the physical tag PTT1 are connected by a direct connection using wireless communication without going through the network N1. Also, in Fig. 22, only the physical tag PTT1 and the container C1 are shown as representative examples, but the other physical tags PTT2 to PTT11 and the other containers C2 to C11 also constitute the information processing system 1a. Furthermore, as will be described later, when a plurality of communication devices 400 are used, the information processing system 1a is constituted by the plurality of communication devices 400.

[0177] The communication device 400 includes a communication unit 401 , a reading unit 402 , a control unit 403 , and a storage unit 404 .

[0178] The communication unit 401 exchanges various types of information with the information processing device 100 using wireless communication under the control of the control unit 403 .

[0179] The reading unit 402 is a communication unit that receives radio waves from the physical tag PTT1 provided on the container C1 and exchanges various information with the physical tag PTT1. Note that the wireless communication used by the physical tag PTT1 described above can be adopted as the wireless communication exchanged between the reading unit 402 and the physical tag PTT1. For example, the reading unit 402 reads the tag ID of the physical tag PTT1 and outputs this tag ID to the control unit 403.

[0180] The control unit 403 controls each unit of the communication device 400 based on a control program stored in the storage unit 404. The control unit 403 is realized by a processing device such as a CPU or a GPU. For example, the control unit 403 executes control to transmit the tag ID read by the reading unit 402 to the information processing device 100.

[0181] The storage unit 404 is a storage medium that stores various types of information. For example, the storage unit 404 stores various types of information (for example, a control program, location identification information for identifying the location where the communication device 400 is installed) that is required for the control unit 403 to perform various processes. The storage unit 404 also stores various types of information acquired via the communication unit 401. The storage unit 404 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0182] [Example of location measurement using wireless tags] Here, a description will be given of a measurement method for measuring the position of a container using a wireless tag as the physical tag PTT 1. For example, the position of the wireless tag (position of the container) can be measured using a known indoor positioning system or the like.

[0183] For example, it is possible to use a measurement method that estimates the position of a wireless tag based on radio waves (radio waves emitted by the wireless tag) received by multiple communication devices. For example, three or more receivers can be installed at a waste collection site WP1 (see Figure 20), and the radio waves (radio waves emitted by the wireless tag) received by these receivers can be acquired. The position of the wireless tag can then be estimated by triangulation (cross-azimuth method) using the radio wave intensities.

[0184] Furthermore, for example, a measurement method can be used in which angle information between one or more communication devices and a wireless tag is calculated based on communication between the communication devices and the wireless tag, and the position of the wireless tag is estimated based on the angle information. For example, a reception angle detection technique (AoA (Angle of Arrival)) or a radiation angle detection technique (AoD (Angle of Departure)) can be used to calculate the angle information. For example, assume that containers C1 to C11 are located within a one-floor facility (waste collection site WP1) and an AoA receiver is installed on the ceiling of that floor. In this case, it is possible to calculate the angle of incidence of the radio waves (radio waves emitted by the wireless tag) received by the receiver installed on the ceiling. Then, the position of each wireless tag present on the floor of the waste collection site WP1 can be estimated based on the angle of incidence. Alternatively, the position of each wireless tag may be estimated using a receiver that can detect the position of the wireless tag that emitted the radio waves based on the directionality and reception strength of the received radio waves.

[0185] Using these position estimation techniques, it is possible to estimate the position of the physical tag PTT1 in three-dimensional space (waste collection point WP1). The estimation result (tag information and position) is transmitted by the communication device 400 to the information processing device 100. Note that the communication device 400 may transmit measurement data of the received radio waves to the information processing device 100, and the information processing device 100 may estimate the position of the physical tag PTT1 using the above-mentioned position estimation method.

[0186] Furthermore, by fixing the imaging range IM1 of the imaging device 200, it is possible to associate a three-dimensional space (the waste collection point WP1) with a two-dimensional space (the captured image corresponding to the imaging range IM1). Therefore, the tag information acquisition unit 122 can estimate the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) based on the position of the physical tag PTT1 at the waste collection point WP1.

[0187] In this way, the tag information acquisition unit 122 acquires the tag ID (tag information) of the physical tag PTT1 acquired by the communication device 400 and the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) estimated by the above-mentioned estimation method. Then, the tag information acquisition unit 122 associates the position of the physical tag PTT1 (position in the captured image) with the acquired tag information and outputs them to the setting unit 123, the annotation unit 124, and the generation unit 125.

[0188] The setting unit 123 sets a predetermined image area from the captured image output from the image acquiring unit 121 based on the position and tag information of the physical tag PTT1 acquired by the tag information acquiring unit 122. For example, it is possible to set an image area of ​​a predetermined range based on the position of the physical tag PTT1 from the captured image corresponding to the imaging range IM1 (see FIG. 20). For example, it is possible to set a rectangular image area with the position of the physical tag PTT1 as the center and a size according to the attribute corresponding to the tag information.

[0189] 21, the identification information PTT1a of the physical tag PTT1 can be read, and based on this identification information PTT1a, the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be identified. Therefore, the setting unit 123 can affine transform the image region information (corresponding to the tag information of the physical tag PTT1) stored in the image region information 153 (see FIG. 7) using the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a. In this case, the setting unit 123 identifies the edge of the opening of the container C1 in the captured image based on the information about the physical tag PTT1 (the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a) and the tag information output from the tag information acquisition unit 122 (tag information corresponding to the physical tag PTT1).

[0190] Specifically, the setting unit 123 extracts image area information 153 corresponding to the tag information (tag information corresponding to the physical tag PTT1) output from the tag information acquisition unit 122 from the setting information DB 150 (see FIG. 7). Then, the setting unit 123 identifies the edge of the opening of the container C1 based on the extracted image area information and information about the physical tag PT1 (base point PP1, size PPL1 of the identification information PTT1a, and direction θ1 of the identification information PTT1a). This identification method is the same as the identification method shown in FIGS. 10 to 13, etc. Next, the setting unit 123 identifies an image area including the edge of the opening of the identified container C1, and sets the image area in the captured image.

[0191] The annotation unit 124 performs annotation on the object included in the captured image output from the image acquisition unit 121 based on the tag information output from the tag information acquisition unit 122. Here, if it is possible to estimate the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) using wireless communication, it is possible to associate the physical tag PTT1 with the image area including it based on the position. Also, if it is not possible to estimate the position of the physical tag PTT1 using wireless communication, it is possible to associate the identification information PTT1a of the physical tag PTT1 with the image area specified based on the color of the identification information PTT1a.

[0192] [Example of effect of this embodiment] In recent years, image recognition processing using AI has become widespread. To improve the accuracy of image recognition, the accuracy of annotation, which identifies and tags (labels) image regions containing objects, is crucial. For example, image annotation can be broadly categorized into three categories: object detection, region extraction (semantic segmentation), and image classification. Object detection is an annotation method that encloses objects in an image using rectangular regions (box-shaped regions) called bounding boxes and assigns annotations based on the category. Region extraction (semantic segmentation) is an annotation method used to strictly process only objects. It assigns tags to each pixel of image data or video data and detects image regions. Image classification is an annotation method that determines which category an image belongs to based on its content.

[0193] For example, adopting area extraction (semantic segmentation) to improve the accuracy of AI image recognition requires specifying the area pixel by pixel, making it difficult to automate. This requires manual human effort to specify the image area, which requires a huge amount of labor. There is also a risk of inaccurate information being entered due to human error. Furthermore, even if annotation methods other than area extraction (semantic segmentation) are adopted, the attributes of each waste item and the image area containing each waste item must be manually specified by humans, which increases the time required for annotation and may reduce the accuracy of the annotation.

[0194] Therefore, in this embodiment, containers C1 to C11 that store waste are provided with physical tags PT1 to PT11 and PTT1 to PTT11 from which tag information can be read. An imaging device 200 (or a communication device 400) is also installed at the waste disposal site. The information processing device 100 acquires tag information that identifies each waste based on a captured image (or wireless communication by the communication device 400) that includes the containers C1 to C11 generated by the imaging device 200. Next, the information processing device 100 automatically determines the attributes of each waste and automatically sets an image area that includes each waste. This means that automatic annotation of the attributes of each waste and the image area that includes each waste can be achieved. Furthermore, the automatically annotated image data can be used as data for the next AI (training data, validation data, test data).

[0195] In this way, since there is no need to manually specify the attributes of each waste or set the image area containing each waste, annotation time can be significantly reduced and annotation accuracy can be improved. For example, it is possible to prevent a decrease in the accuracy of setting image areas and a decrease in the accuracy of specifying attribute information due to inaccurate information input caused by human error, thereby improving annotation accuracy. Furthermore, annotation can be performed accurately even for images not taken under ideal conditions. As a result, annotation of waste can be easily performed at many waste treatment sites, and efficient generation of trained models can be achieved.

[0196] [Examples of application to other objects] In the above, waste has been described as an example of an object contained in a container. That is, an example has been shown in which annotation is performed on waste (object) contained in a container. However, this embodiment can also be applied to objects other than waste as objects contained in a container. For example, this embodiment can also be applied when annotation is performed on agricultural products, marine products, industrial products, etc. as objects contained in a container. This makes it possible to easily perform annotation on objects at the processing site of each object, and to realize efficient generation of trained models.

[0197] In the above, an example of performing annotation on an object contained in a container has been shown. However, this embodiment can also be applied to an object that is not contained in a container. In this case, for example, a physical tag can be attached to the object itself or its surroundings (for example, a sign behind it), and annotation can be performed on the object using this physical tag.

[0198] [Example of executing processing on other devices or systems] Although the above describes an example in which tag information acquisition processing, setting processing, annotation processing, generation processing, etc. are executed in the information processing device 100, all or part of each of these processes may be executed in other devices. In this case, an information processing system is configured by the devices that execute part of each of these processes. For example, at least part of each process can be executed using devices available to the user (e.g., smartphones, tablet terminals, personal computers), various information processing devices such as servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0199] Furthermore, a part (or all) of the information processing system capable of executing the functions of the information processing device 100 (or the information processing system 1, 1a) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0200] [Configuration example and effect example of this embodiment] The main effects of the information processing device 100, the information processing method, and the program according to the embodiment of the present invention will be described below.

[0201] The information processing device 100 has an image acquisition unit 121 that acquires an image including waste (an example of a target object), a tag information acquisition unit 122 that acquires tag information that identifies the waste, and an annotation unit 124 that applies annotations related to the waste to the image based on the tag information.

[0202] This allows annotation of waste to be performed on images based on tag information that identifies the waste, thereby reducing the time required for annotation when generating a trained model and improving the accuracy of annotation.

[0203] The tag information can be information obtained from the physical tags PT1 to PT11, PTT1 to PTT11.

[0204] This makes it possible to appropriately acquire tag information about each waste by using the physical tags PT1 to PT11 and PTT1 to PTT11.

[0205] The physical tags PT1 to PT11 and PTT1 to PTT11 may be provided in association with the containers C1 to C11 that contain waste (an example of a target object).

[0206] According to this, even if a physical tag is not attached to the waste itself, it is possible to attach a physical tag in association with the containers C1 to C11 that store the waste. For example, if the physical tags cannot be attached directly to the containers C1 to C11, it is possible to attach the physical tags PT1 to PT11 and PTT1 to PTT11 around the containers C1 to C11 (for example, on a sign behind them). This makes it possible to properly obtain tag information for each waste.

[0207] Waste (an example of a target object) is stored in containers C1 to C11 corresponding to the waste, and the containers C1 to C11 are provided with physical tags PT1 to PT11 and PTT1 to PTT11 from which tag information can be read. The tag information includes attributes of the waste or the containers C1 to C11. The annotation unit 124 performs annotation on the image based on the attributes included in the tag information.

[0208] According to this, even if the waste itself does not have a physical tag, tag information for each waste can be appropriately acquired by attaching physical tags PT1 to PT11 and PTT1 to PTT11 to the containers C1 to C11 that store the waste. This makes it possible to acquire and use the attributes of the waste (or the attributes of the containers C1 to C11), and to perform annotation on the image based on the acquired attributes.

[0209] The image acquiring unit 121 acquires images of the containers C1 to C11 from the imaging device 200, which captures images of the containers C1 to C11 and generates images including the containers C1 to C11, the waste (an example of a target object) contained in the containers C1 to C11, and the physical tags PT1 to PT11 and PTT1 to PTT11 attached to the containers C1 to C11. The tag information acquiring unit 122 acquires tag information based on the physical tags PT1 to PT11 and PTT1 to PTT11 included in the captured images.

[0210] This makes it possible to acquire a captured image including the containers C1 to C11 that store waste, and acquire tag information based on the physical tags PT1 to PT11 and PTT1 to PTT11 attached to the containers C1 to C11 included in the captured image, thereby making it possible to appropriately acquire tag information for each waste.

[0211] Waste (an example of a target object) or containers C1 to C11 containing the target object are provided with physical tags PT1 to PT11 and PTT1 to PTT11 from which tag information can be read. The physical tags PT1 to PT11 and PTT1 to PTT11 can be at least one of a barcode (see FIGS. 2 and 3), a two-dimensional code, a color code capable of expressing a plurality of pieces of information by an arrangement of a plurality of colors (see FIGS. 2 and 3), a graphic code capable of expressing a plurality of pieces of information by an arrangement of a plurality of graphics to which one of a plurality of colors is applied (see FIGS. 4 and 5), and a wireless tag capable of transmitting tag information by wireless communication (see FIGS. 20 to 22).

[0212] This makes it possible to provide the containers C1 to C11 with appropriate physical tags according to the environment of the installation location of the containers C1 to C11 that store waste. This makes it possible to acquire tag information from the physical tags PT1 to PT11 and PTT1 to PTT11 according to the environment of the installation location of the containers C1 to C11.

[0213] The information processing device 100 further includes a setting unit 123 that sets an image area including waste (an example of a target object) from the image based on the tag information. An annotation unit 124 applies annotation to the image area.

[0214] This allows only the image area containing waste to be the target of annotation processing, thereby reducing the amount of calculation required for annotation processing. Also, since only the image area containing waste can be the target of annotation processing, the accuracy of annotation can be improved.

[0215] In addition to waste, at least one of agricultural products, marine products, and industrial products may be the target object.

[0216] This makes it possible to reduce the time required for annotation and improve the accuracy of annotation when annotating an image that includes a target object (at least one of waste, agricultural products, marine products, and industrial products).

[0217] The information processing device 100 further includes a generation unit 125 that generates a trained model using annotated images.

[0218] This makes it possible to annotate images related to waste based on tag information that identifies the waste, and then generate a trained model using the annotated images. This reduces the time required for the generation process when generating a trained model and improves the accuracy of the generation process.

[0219] The generation unit 125 generates a plurality of trained models for each attribute of waste (an example of an object).

[0220] This makes it possible to select and use an appropriate trained model from among the multiple trained models generated, depending on the attributes of the waste.

[0221] An information processing method according to an embodiment of the present invention includes an image acquisition process (step S501) for acquiring an image including waste (an example of a target object), a tag information acquisition process (steps S502, S503) for acquiring tag information for identifying the waste, and an annotation process (step S506) for applying annotations related to the waste to the image based on the tag information.

[0222] This allows annotation of waste to be performed on images based on tag information that identifies the waste, thereby reducing the time required for annotation when generating a trained model and improving the accuracy of annotation.

[0223] A program according to an embodiment of the present invention is a program that causes a computer to execute an image acquisition procedure (step S501) for acquiring an image including waste (an example of a target object), a tag information acquisition procedure (steps S502, S503) for acquiring tag information for identifying the waste, and an annotation procedure (step S506) for applying annotations related to the waste to the image based on the tag information.

[0224] This allows annotation of waste to be performed on images based on tag information that identifies the waste, thereby reducing the time required for annotation when generating a trained model and improving the accuracy of annotation.

[0225] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that makes it possible to realize this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0226] Each process described in this embodiment is executed based on a program that causes a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to an information processing device can store the program in the storage device of the information processing device. This makes it possible to cause the updated information processing device to execute each process described in this embodiment.

[0227] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments. [Explanation of symbols]

[0228] 1, 1a Information Processing System 100 Information processing device 110 Communications Department 120 control section 121 Image acquisition unit 122 Tag information acquisition unit 123 Settings 124 Annotation Section 125 Generation part 130 Storage section 140 Tag DB 150 Setting information DB 160 trained models DBDB 200 Imaging device 201 Communications Department 202 Control section 203 Storage section 204 Imaging unit 300 User Terminals 301 Communications Department 302 Control Unit 303 Storage section 304 Operation section 305 Sound acquisition section 306 Display section 400 Communication Equipment 401 Communications Department 402 Reading unit 403 Control Unit 404 Storage section C1~C11, C13 containers N1 Network PT1~PT11, PT1a, PT11a, PT1A, PT1B, PT1C, PT13, PT20, PT30, PT40, PTT1~PTT11 Physical tags WP1 Waste collection point

Claims

1. an image acquisition unit that acquires an image including an object; a tag information acquisition unit that acquires tag information that identifies the object; an annotation unit that performs annotation on the image regarding the object based on the tag information; An information processing device having the above.

2. 2. The information processing device according to claim 1, The tag information is information obtained from a physical tag. Information processing device.

3. 3. The information processing device according to claim 2, The physical tag is associated with a container that contains the object. Information processing device.

4. 2. The information processing device according to claim 1, The object is contained in a container appropriate for the object, The container is provided with a physical tag capable of reading the tag information, the tag information includes attributes of the object or the container; the annotation unit performs the annotation on the image based on the attribute included in the tag information; Information processing device.

5. 5. The information processing device according to claim 4, the image acquisition unit acquires the image from an imaging device that captures an image of the container and generates an image including the container, the object contained in the container, and the physical tag attached to the container as the image; the tag information acquisition unit acquires the tag information based on the physical tag included in the captured image. Information processing device.

6. 2. The information processing device according to claim 1, The object or a container in which the object is accommodated is provided with a physical tag capable of reading the tag information, The physical tag is at least one of a barcode, a two-dimensional code, a color code capable of expressing a plurality of pieces of information by an arrangement of a plurality of colors, a graphic code capable of expressing a plurality of pieces of information by an arrangement of a plurality of graphics to which any of a plurality of colors is applied, and a wireless tag capable of transmitting the tag information by wireless communication. Information processing device.

7. 2. The information processing device according to claim 1, a setting unit that sets an image area including the object from the image based on the tag information; the annotation unit performs the annotation on the image region; Information processing device.

8. 8. The information processing device according to claim 1, The object is at least one of waste, agricultural products, marine products, and industrial products. Information processing device.

9. 8. The information processing device according to claim 1, Further comprising a generation unit that generates a trained model using the annotated image. Information processing device.

10. 10. The information processing device according to claim 9, The generation unit generates a plurality of the trained models for each attribute of the object. Information processing device.

11. an image acquisition process for acquiring an image including the object; a tag information acquisition process for acquiring tag information for identifying the object; An annotation process for performing annotations regarding the object on the image based on the tag information; An information processing method including:

12. an image acquisition step for acquiring an image including the object; a tag information acquisition step of acquiring tag information that identifies the object; an annotation procedure for performing annotations on the image regarding the object based on the tag information; A program that causes a computer to execute the following.

Images