Information processing device, information processing method, and program

Abstract

To suppress the computation volume associated with estimation processing for estimating the condition of a target object using multiple trained models.SOLUTION: An information processing device is provided, for estimating the condition of a target object on the basis of measurement data of the target object using multiple trained models. The information processing device comprises a measurement data acquisition unit for acquiring the measurement data of the target object, a tag information acquisition unit for acquiring tag information for identifying the target object, a selection unit for selecting at least one trained model from among the multiple trained models on the basis of the tag information, and an estimation unit for inputting the measurement data to the selected at least one trained model to estimate the condition of the target object based on an output from the at least one trained model.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program that can use a trained model. [Background technology]

[0002] Conventionally, there are techniques for detecting various types of information using trained models. For example, a technique has been proposed in which multiple learning devices are used to detect multiple specification areas of an analog meter from a captured image of the analog meter (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-135353 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described conventional technology, when an analog meter has n specifications, the system is configured with n learning devices corresponding to the respective specifications. Furthermore, each learning device detects only the target specification area. Therefore, when detecting n specification areas from an image of an analog meter having n specifications, it is necessary to simultaneously use n learning devices to perform the detection process, which increases the amount of calculation required for the detection process.

[0005] The present invention aims to reduce the amount of calculation required for the estimation process when estimating the state of an object using multiple trained models. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device that estimates a state of an object based on measurement data of the object using multiple trained models. The information processing device includes a measurement data acquisition unit that acquires measurement data of the object, a tag information acquisition unit that acquires tag information that identifies the object, a selection unit that selects at least one trained model from the multiple trained models based on the tag information, and an estimation unit that inputs the measurement data to the selected trained model and estimates the state of the object based on an output from the trained model. [Effects of the Invention]

[0007] According to one aspect of the present invention, when estimating the state of an object using multiple trained models, the amount of calculation involved in the estimation process can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of use of an information processing system. [Figure 2] FIG. 2 is a top view showing an example of the configuration of a physical tag. [Figure 3] FIG. 3 is a top view showing an example of the configuration of a physical tag related to a barcode. [Figure 4] FIG. 4 is a top view showing an example of the configuration of a physical tag related to a graphic code. [Figure 5] FIG. 5 is a top view showing an example of the configuration of a physical tag relating to another graphic code. [Figure 6] FIG. 6 is a block diagram illustrating an example of the system configuration of the information processing system. [Figure 7] FIG. 7 is a diagram showing each piece of information stored in the trained model DB. [Figure 8] FIG. 8 is a diagram showing each piece of information stored in the selection information DB. [Figure 9A] FIG. 9A is a diagram showing an example of rectangular image region information. [Figure 9B] FIG. 9B is a diagram showing an example of circular image region information. [Figure 10A] FIG. 10A is a diagram showing an example of transition when a figure is subjected to affine transformation and translation. [Figure 10B] FIG. 10B is a diagram showing an example of transition when a figure is enlarged or reduced by affine transformation. [Figure 10C] FIG. 10C is a diagram showing an example of a transition when a figure is rotated by affine transformation. [Figure 10D] FIG. 10D is a diagram showing an example of transition when a figure is sheared by affine transformation. [Figure 11] FIG. 11 is a diagram showing an example of setting a bounding box for a box-shaped container. [Figure 12] FIG. 12 is a diagram showing an example of transformation when image region information is subjected to affine transformation. [Figure 13] FIG. 13 is a diagram showing an example in which physical tags are provided on three sides of a container. [Figure 14] FIG. 14 is a diagram showing an example of setting a bounding box for a cylindrical container. [Figure 15] FIG. 15 is a diagram illustrating an example of selecting a trained model. [Figure 16] FIG. 16 is a diagram showing an example of a display screen on a user terminal. [Figure 17] FIG. 17 is a flowchart illustrating an example of the estimation process. [Figure 18A] FIG. 18A is a perspective view showing an example in which physical tags are provided on two edges of a box-shaped container. [Figure 18B] FIG. 18B is a diagram showing an example of image area information when physical tags are provided on two sides of the edge of a box-shaped container. [Figure 19A] FIG. 19A is a perspective view showing an example in which physical tags are provided at two positions on the edge of a cylindrical container. [Figure 19B] FIG. 19B is a diagram showing an example of image region information when physical tags are provided at two positions on the edge of a cylindrical container. [Figure 20A] FIG. 20A is a perspective view showing an example in which a physical tag is provided on one side of the edge of a container according to a modified example. [Figure 20B] FIG. 20B is a diagram showing an example of image area information when a physical tag is provided on one side of the edge of a container according to a modified example. [Figure 21] FIG. 21 is a diagram showing a modified example in which an information processing system is used. [Figure 22] FIG. 22 is a diagram showing an example of identification information provided on the surface of a physical tag. [Figure 23] FIG. 23 is a block diagram showing an example of a system configuration according to a modified example of the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0009] The embodiments described below are not limited to the drawings described by the brief description of the drawings.

[0010] (1) One aspect of the present invention is an information processing device that estimates the state of an object based on measurement data of the object using multiple trained models, and includes: a measurement data acquisition unit that acquires the measurement data of the object; a tag information acquisition unit that acquires tag information that identifies the object; a selection unit that selects at least one trained model from the multiple trained models based on the tag information; and an estimation unit that inputs the measurement data to the selected trained model and estimates the state of the object based on the output from the trained model.

[0011] According to this, when using a trained model to estimate the state (e.g., weight, filling rate) of an object (e.g., waste) from an image including the object, it is possible to select and use a trained model corresponding to the object from among multiple trained models based on tag information that identifies the object. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of the object from an image including the object, thereby reducing the amount of calculation required for the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0012] (2) An aspect of the present invention is the information processing device according to (1), in which the tag information is information acquired from a physical tag.

[0013] This makes it possible to appropriately obtain tag information about each object using the physical tag.

[0014] (3) One aspect of the present invention is the information processing device according to (2), in which the physical tag is associated with a container that contains the object.

[0015] This allows a physical tag to be associated with a container that contains the object, even if the object itself does not have a physical tag. For example, if the physical tag cannot be attached directly to the container, it can be attached to the periphery of the container (e.g., on a sign behind it). This makes it possible to appropriately acquire tag information for each object. This makes it possible to select and use an appropriate trained model for each object, thereby improving the accuracy of the estimation process for estimating the state of the object.

[0016] (4) One aspect of the present invention is an information processing device according to any one of (1) to (3), wherein a container for storing the object is provided with a physical tag capable of reading the tag information, the measurement data acquisition unit acquires the measurement data measuring the state inside the container, and the estimation unit uses the measurement data to estimate the state of the object inside the container.

[0017] According to this, even if the object itself does not have a physical tag, tag information about each object can be appropriately acquired by attaching a physical tag to the container that holds the object. This makes it possible to select and use an appropriate trained model according to the object, thereby improving the accuracy of the estimation process for estimating the state of the object.

[0018] (5) One aspect of the present invention is an information processing device as described in (3) or (4), wherein the tag information is information that can identify the attributes of the object, and the physical tag is provided on the container according to the attributes of the object contained in the container.

[0019] According to this, since physical tags are provided according to the attributes of the objects contained in the container, it is possible to appropriately acquire tag information according to the attributes of each object. This makes it possible to select and use an appropriate trained model according to the attributes of the object, thereby improving the accuracy of the estimation process for estimating the state of the object.

[0020] (6) One aspect of the present invention is an information processing device according to any one of (3) to (5), in which the state of the object is the filling rate of the object in the container.

[0021] This makes it possible to select an appropriate trained model from among multiple trained models according to the attributes of the object and estimate the filling rate of the object from an image including the object. This eliminates the need to simultaneously use multiple trained models to perform an estimation process for estimating the filling rate of the object from an image including the object, making it possible to reduce the amount of calculation involved in the estimation process. In other words, it is possible to reduce the load of the estimation process and shorten the processing time.

[0022] (7) One aspect of the present invention is an information processing device according to any one of (1) to (6), wherein a container for storing the object is provided with a physical tag capable of reading the tag information, the measurement data acquisition unit is an image acquisition unit that acquires an image of the container from an imaging device that captures an image including the object, and the tag information acquisition unit acquires the tag information based on the physical tag included in the image.

[0023] This makes it possible to acquire a captured image including a container that contains an object, and acquire tag information based on a physical tag attached to the container included in the captured image. This makes it possible to select and use an appropriate trained model according to the object, thereby improving the accuracy of the estimation process for estimating the state of the object.

[0024] (8) One aspect of the present invention is an information processing device described in any of (1) to (7), wherein a container for storing the object is provided with a physical tag capable of reading the tag information, and the physical tag is at least one of a barcode, a two-dimensional code, a color code capable of expressing multiple pieces of information by an arrangement of multiple colors, a graphic code capable of expressing multiple pieces of information by an arrangement of multiple figures each having one of multiple colors, and a wireless tag capable of transmitting the tag information using wireless communication.

[0025] This makes it possible to provide a container with an appropriate physical tag that corresponds to the environment of the installation location of the container that contains the object. This makes it possible to acquire tag information from the physical tag according to the environment of the installation location of the container. This makes it possible to select and use an appropriate trained model that corresponds to the object, thereby improving the accuracy of the estimation process for estimating the state of the object.

[0026] (9) One aspect of the present invention is an information processing device described in any of (1) to (8), wherein the measurement data acquisition unit is an image acquisition unit that acquires the image from an imaging device that captures an image of the object and generates an image including the object, and further includes a setting unit that sets an image area including the object from the image based on the tag information, and the estimation unit inputs the image in which the image area is set to the selected trained model, and sets the output from the trained model as the state of the object.

[0027] According to this, an image in which an image region including the object is set can be input to a trained model corresponding to the object, and the output from the trained model can be used as the state of the object (e.g., weight, fill rate). This eliminates the need to subject all images generated by capturing an object to estimation processing, making it possible to reduce the amount of calculation involved in the estimation processing. Furthermore, since only images including the object can be subject to estimation processing and the output result can be used as the state of the object (e.g., weight, fill rate), it is possible to improve the accuracy of estimating the state of the object.

[0028] (10) One aspect of the present invention is an information processing device according to any one of (1) to (9), wherein the object is at least one of waste, agricultural products, marine products, and industrial products.

[0029] According to this, when using a trained model to estimate the state (e.g., weight, filling rate) of an object (at least one of waste, agricultural products, marine products, and industrial products) from an image including the object, it is possible to select and use a trained model corresponding to the object from among multiple trained models based on tag information that identifies the object. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of the object from an image including the object, thereby reducing the amount of calculation required for the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0030] (11) One aspect of the present invention is an information processing device described in any of (1) to (10), wherein the state of the object is the weight of the object, the plurality of trained models include a trained model corresponding to the volume of the object, a trained model corresponding to the shape of the object, and a trained model corresponding to the density of the object, the selection unit selects a trained model corresponding to the volume of the object, a trained model corresponding to the shape of the object, and a trained model corresponding to the density of the object from among the plurality of trained models based on the tag information, and the estimation unit inputs the measurement data to each selected trained model and estimates the weight of the object based on the output from each trained model.

[0031] For example, when the target object is a resin such as PET, it is possible that a mixture of standardized objects such as PET bottles and non-standardized objects such as rolls or plates is present. In this case, the volume of the standardized object differs from the volume of the non-standardized object, which may result in a large error in the weight calculated based on the volume. Furthermore, it is also possible that a mixture of standardized objects (e.g., uncrushed PET bottles) and compressed standardized objects (e.g., crushed PET bottles) is present. In this case, the volume of the standardized object differs from the volume of the compressed standardized object, which may result in a large error in the weight calculated based on the volume. Therefore, it is possible to improve the estimation accuracy by estimating the weight of an object using multiple trained models according to the attributes of the object.

[0032] (12) One aspect of the present invention is an information processing method for estimating a state of an object based on measurement data of the object using multiple trained models, the information processing method including: a measurement data acquisition process for acquiring the measurement data of the object; a tag information acquisition process for acquiring tag information that identifies the object; a selection process for selecting at least one trained model from the multiple trained models based on the tag information; and a state estimation process for inputting the measurement data into the selected trained model and estimating the state of the object based on the output from the trained model.

[0033] According to this, when using a trained model to estimate the state (e.g., weight, filling rate) of an object (e.g., waste) from an image including the object, it is possible to select and use a trained model corresponding to the object from among multiple trained models based on tag information that identifies the object. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of the object from an image including the object, thereby reducing the amount of calculation required for the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0034] (13) One aspect of the present invention is a program that causes a computer to execute an information processing method for estimating the state of an object based on measurement data of the object using multiple trained models, the program causing a computer to execute the following steps: a measurement data acquisition step for acquiring measurement data of the object; a tag information acquisition step for acquiring tag information that identifies the object; a selection step for selecting at least one trained model from the multiple trained models based on the tag information; and a state estimation step for inputting the measurement data into the selected trained model and estimating the state of the object based on the output from the trained model.

[0035] According to this, when using a trained model to estimate the state (e.g., weight, filling rate) of an object (e.g., waste) from an image including the object, it is possible to select and use a trained model corresponding to the object from among multiple trained models based on tag information that identifies the object. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of the object from an image including the object, thereby reducing the amount of calculation required for the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0036] Hereinafter, embodiments will be described with reference to the accompanying drawings.

[0037] [Example of use of information processing system] Fig. 1 is a diagram showing an example of using the information processing system 1. Fig. 1 shows an example in which a plurality of types of waste are stored in a plurality of containers C1 to C11 at a waste collection site WP1. Fig. 1 also shows an example in which physical tags PT1 to PT11 are provided on the plurality of containers C1 to C11, respectively, for identifying the waste stored in each container.

[0038] For example, containers C1 and C5 contain plastic bags, containers C2 and C6 contain paper waste, containers C3 and C7 contain PP (polypropylene) resin, containers C4 and C8 contain hard plastic, container C9 contains glass waste, container C10 contains iron scrap, and container C11 contains circuit boards. The containers C1 to C11 can be made of a material and have a structure that suits the waste to be stored. For example, they can be made of a metal material with a mesh structure that is visible from the outside, or they can be made of a metal material with a box-like or cylindrical structure. Various known techniques can be used to determine the material and structure of containers C1 to C11 depending on the waste to be stored, and therefore detailed explanations will be omitted here.

[0039] In addition, in this embodiment, an example is shown in which waste stored in containers C1 to C11 is identified using physical tags PT1 to PT11 included in the imaging range IM1 of the imaging device 200. The physical tags PT1 to PT11 will be described in detail with reference to FIG. 2. Also, configuration examples of the information processing device 100 and the imaging device 200 will be described in detail with reference to FIG. 6. Also, the bounding box BB1 and the like will be described in detail with reference to FIGS. 11 to 14, etc.

[0040] [Physical tag configuration example] FIG. 2 is a top view showing an example of the configuration of the physical tag PT1. Here, only the physical tag PT1 is shown as a representative example, but the same applies to the other physical tags PT2 to PT11. Furthermore, this embodiment shows an example in which the physical tags PT1 to PT11 are used to acquire the attributes and positions of the target object (waste) (or the attributes and positions of the corresponding containers C1 to C11). Note that the physical tags PT1 to PT11 can be changed as appropriate depending on the positions at which they are attached to the containers. For example, when the physical tag PT11 is attached to the edge of the opening of the container C11, the physical tag PT11 can be configured along the circumferential edge.

[0041] FIG. 2 shows an example of a color code in which five rectangles of a predetermined color are arranged in a row as the physical tag PT1. This color code can be configured by coloring the rectangle R1 at the end in the arrangement direction black, and coloring the other four rectangles R2 to R5 other than the black rectangle R1 in a color other than black. In this case, it is possible to represent multiple pieces of information by changing the colors of the four rectangles R2 to R5 arranged in a row. For example, an arrangement in which rectangle R2 is blue, rectangle R3 is red, rectangle R4 is yellow, and rectangle R5 is green can be used as a color code representing a "plastic bag."

[0042] Note that, other than the color code shown in FIG. 2, other physical tags readable by the imaging device 200 may be used, or other wireless tags may be used. Examples of other physical tags readable by the imaging device 200 are shown in FIGS. 3 to 5. Also, normal barcodes and two-dimensional codes (e.g., QR Code (registered trademark)) may be used as physical tags. Also, examples of wireless tags are shown in FIGS. 21 to 23. Also, these physical tags are associated with unique identification information (tag information). For example, the information processing device 100 can store a tag DB (Data Base) indicating the relationship between physical tags and tag information in the storage unit 130 (see FIG. 6), and can acquire tag information associated with the read physical tag using the tag DB. Note that the tag DB may be stored in an external device and acquired from the external device for use.

[0043] For example, the physical tag PT1 can be a sheet-like member (not shown) that is attached to the object, and five rectangles R1 to R5, each colored a specific number, can be arranged in a line on this sheet-like member. The rectangles R1 to R5 have the same length L1 in the arrangement direction. The lengths CR1 to CR4 between the rectangles R1 to R5 in the arrangement direction are also the same. The lengths L1 in the arrangement direction may be different from each other, and the lengths CR1 to CR4 in the arrangement direction may also be different from each other. The lengths CR1 to CR4 in the arrangement direction may also be zero.

[0044] [Barcode configuration example] FIG. 3 is a top view showing an example of the configuration of a physical tag PT20 relating to a barcode.

[0045] FIG. 3 shows an example of a color code in which multiple rectangles are arranged in a row as the physical tag PT20. This color code can be configured by placing standard bars (rectangle R21 of length L11 and rectangle R26 of length L12) at both ends to indicate the direction of the arrangement, and arranging four bars of two different lengths (long bars, short bars) between them. Note that the arrangement may also be configured with three types of information: bars of two different lengths (long bars, short bars) and blank spaces where no bars are arranged. In this case, multiple pieces of information can be represented by changing the bars of two different lengths (long bars, short bars) and blank spaces arranged between the standard bars at both ends (rectangles R21, R26).

[0046] 3 shows an example of arranging a long bar (rectangle R22 of length L13), a short bar (rectangle R23 of length L14), a short bar (rectangle R24 of length L14), and a long bar (rectangle R25 of length L13). For example, this arrangement can be used as a barcode representing a "plastic bag." In this way, the physical tag PT20 can represent multiple pieces of information by using the horizontal length of each rectangle.

[0047] [Example of shape code configuration] FIG. 4 is a top view showing an example of the configuration of a physical tag PT30 relating to a graphic code.

[0048] FIG. 4 shows an example of a graphic code in which four types of graphics (circle (oval), triangle, square, inverted triangle) are arranged in a row as the physical tag PT30. This graphic code can be configured by placing standard bars (rectangles R31, R36) at both ends to indicate the direction of the arrangement, and arranging four of the four types of graphics between them. The four types of graphics may also be colored. In this case, multiple pieces of information can be represented by changing the four types of graphics arranged between the standard bars (rectangles R31, R36) at both ends and the colors assigned to them.

[0049] 4 shows an example of arranging a blue circle (oval) graphic R32, a red triangle graphic R33, a yellow square graphic R34, and a green inverted triangle graphic R35. For example, this arrangement can be used as a graphic code representing a "plastic bag." In this way, the physical tag PT30 can represent multiple pieces of information by combining multiple types of graphics and their colors.

[0050] [Example of shape code for changing vertical length] FIG. 5 is a top view showing an example of the configuration of a physical tag PT40 relating to a graphic code whose length in the up-down direction (vertical direction, height direction) is changed.

[0051] FIG. 5 shows an example of a graphic code as a physical tag PT50, in which four types of graphics (circle (oval), triangle, square, inverted triangle) whose lengths can be changed in two directions (vertical and height directions) are arranged in a row. This graphic code can be configured by placing standard bars (rectangles R41 and R46) at both ends to indicate the direction of the arrangement, and arranging four of the four types of graphics whose lengths can be changed in the vertical direction between them. Furthermore, each of the four types of graphics may be assigned a color. In this case, multiple pieces of information can be represented by changing the four types of graphics arranged between the standard bars (rectangles R41 and R46) at both ends, the colors assigned to them, and the vertical lengths.

[0052] 5 shows an example in which a circle (oval) R42 with a long vertical line of blue, a triangle R43 with a short vertical line of red, a square R44 with a short vertical line of yellow, and an inverted triangle R45 with a long vertical line of green are arranged. For example, this arrangement can be used as a graphic code representing a "plastic bag." In this way, the physical tag PT40 can represent multiple pieces of information by combining multiple types of shapes, their colors, and their vertical lengths.

[0053] [Example of information processing system configuration] FIG. 6 is a block diagram showing an example of the system configuration of the information processing system 1. As shown in FIG.

[0054] The information processing system 1 is composed of multiple devices that can be connected via a network N1. FIG. 6 shows an example of the information processing system 1 including an information processing device 100, an imaging device 200, and a user terminal 300. The information processing device 100, the imaging device 200, and the user terminal 300 are each connected to the network N1 by a communication method using wired communication or wireless communication. The network N1 is a network such as a public line network or the Internet. In this case, the wireless communication may be a mobile communication network (e.g., standards such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), and 6G (6th Generation)). Alternatively, at least one of wireless communication standards such as wireless LAN (e.g., Wi-Fi (Wireless Fidelity)), Bluetooth (registered trademark), and ZigBee (registered trademark) may be used. In addition, multiple frequency bands (e.g., UHF band and 2.4 GHz band) may be used in combination.

[0055] Note that the information processing device 100, the imaging device 200, and the user terminal 300 may be directly connected using wired or wireless communication without going through the network N1. Also, although only the imaging device 200 and the user terminal 300 are shown as representative examples in Fig. 6, other imaging devices and other user terminals installed at various locations may also constitute the information processing system 1.

[0056] [Configuration example of information processing device] The information processing device 100 includes a communication unit 110, a control unit 120, and a storage unit 130. The information processing device 100 can be, for example, a server realized by one or more devices.

[0057] The communication unit 110, under the control of the control unit 120, exchanges various types of information with other devices using wired or wireless communication.

[0058] The control unit 120 controls each unit of the information processing device 100 based on a control program stored in the storage unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Specifically, the control unit 120 includes an image acquisition unit 121, a tag information acquisition unit 122, a setting unit 123, a selection unit 124, an estimation unit 125, and an output control unit 126.

[0059] The image acquisition unit 121 acquires the captured image (including image data that has been subjected to image processing and its accompanying information) generated by the imaging device 200 via the communication unit 110. Then, the image acquisition unit 121 outputs the acquired captured image to the tag information acquisition unit 122 and the setting unit 123. The image acquisition unit 121 is an example of a measurement data acquisition unit that acquires measurement data of an object (e.g., waste).

[0060] Here, the object shown in this embodiment includes an object in which a regular object and an irregular object are mixed. For example, when the object is PET (Polyethyleneterephthalate) resin, it is assumed that a regular object such as a PET bottle and an irregular object such as a roll or plate are mixed. Furthermore, the measurement data of the object shown in this embodiment also includes data measured in a state in which one or more regular or irregular objects are present. For example, when the object is PET resin, it is assumed that one or more regular objects such as a PET bottle are mixed with one or more irregular objects (e.g., roll or plate). Furthermore, it is assumed that the regular objects include a mixture of normal regular objects (e.g., uncrushed PET bottles) and compressed regular objects (e.g., crushed PET bottles).

[0061] The tag information acquisition unit 122 reads a physical tag from the captured image output from the image acquisition unit 121 and acquires tag information based on the read physical tag. Then, the tag information acquisition unit 122 associates the position of the read physical tag (the position in the captured image) with the acquired tag information and outputs them to the setting unit 123 and the selection unit 124. For example, when using the physical tag PT1 shown in FIG. 2, it is possible to read the physical tag PT1 from the captured image using a known image recognition technique (for example, an object detection technique using pattern matching). Then, the tag information acquisition unit 122 acquires tag information associated with the read physical tag PT1 using a tag DB. As described above, the tag DB indicating the relationship between the physical tag and the tag information is stored in the storage unit 130.

[0062] The setting unit 123 sets a predetermined image area from the captured image output from the image acquisition unit 121 based on the tag information acquired by the tag information acquisition unit 122, and outputs setting information regarding the set image area to the estimation unit 125. This setting information includes the position, size, shape, etc. of the image area in the captured image. For example, assume a case where an image area corresponding to waste contained in a container C1 is set from a captured image corresponding to an imaging range IM1 (see FIG. 1). In this case, the setting unit 123 identifies the edge of the opening of the container C1 in the captured image based on information regarding the physical tag PT1 output from the tag information acquisition unit 122 (the position (base point), length, and direction (angle with respect to a specific direction) of the physical tag PT1 in the captured image) and tag information output from the tag information acquisition unit 122 (tag information corresponding to the physical tag PT1).

[0063] Specifically, the setting unit 123 extracts image area information 154 corresponding to the tag information (tag information corresponding to physical tag PT1) output from the tag information acquisition unit 122 from the selection information DB 150 (see FIG. 8). Then, the setting unit 123 identifies the edge of the opening of the container C1 based on the extracted image area information and information related to the physical tag PT1 (the position, length, and angle described above). This identification method will be described in detail with reference to FIGS. 11 to 14, etc. Next, the setting unit 123 identifies an image area including the identified edge of the opening of the container C1, and sets the image area in the captured image. This setting method will be described in detail with reference to FIGS. 11 to 15, etc.

[0064] The selection unit 124 selects a trained model stored in the trained model DB 140 based on the tag information acquired by the tag information acquisition unit 122, and outputs the selection result to the estimation unit 125. Specifically, the selection unit 124 selects trained model identification information 153 corresponding to the tag information 151 acquired by the tag information acquisition unit 122 from the selection information DB 150 (see FIG. 8 ). Next, the selection unit 124 outputs the selection result to the estimation unit 125. That is, the selection unit 124 outputs the selected trained model identification information to the estimation unit 125.

[0065] Here, the trained model is an AI (Artificial Intelligence) model that has been machine-learned using teacher data or training data. Examples of the trained model that can be used include SVM (Support Vector Machine), CNN (Convolutional Neural Network), ViT (Vision Transformer), and YOLO (You Only Look Once).

[0066] Furthermore, the trained model shown in this embodiment is trained using images of waste stored in a container, and is capable of detecting the state (e.g., weight, filling rate) of the waste stored in the container. For example, a trained model can be generated by training a large amount of training data in which example problems and corresponding correct answers are associated. Furthermore, when new input data is input to the trained model, it is possible to output output data that is the correct answer based on the learning results of the example problems and the corresponding correct answers.

[0067] For example, when learning the filling rate as the state of waste, it is possible to generate a learned model by learning a large amount of teacher data in which the relationship between an image including the target waste (the waste contained in the container) and the filling rate of the waste in this container is associated. For example, when there is no waste in the container, the filling rate is set to 0, when the container is full of waste, the filling rate is set to 1, and the filling rate of the waste in other containers is learned as a value between 0 and 1. In this case, a value between 0 and 1 is output as the output result of the learned model. That is, when there is no waste in the container, the filling rate 0 is output, and when the container is full of waste, the filling rate 1 is output. Also, when there is waste in the container and it is not full, a numerical value t (0 < t < 1) corresponding to the filling rate is output. Similarly, when learning the weight as the state of waste, it is possible to generate a learned model by learning a large amount of teacher data in which the relationship between an image including the target waste (the waste contained in the container) and the weight of the waste in this container is associated.

[0068] When generating the learned model, it is possible to cut out a rectangular image region including one container (including the waste) in which the target waste is contained from the captured images of each waste contained in each of a plurality of containers, and use the cut-out image. For example, as shown in FIG. 1, when waste is contained in 11 containers, it is possible to cut out 11 images for each container. Also, for example, as shown in FIG. 1, when 7 types of waste (plastic bags, paper scraps, PP resin, hard plastics, glass scraps, iron scraps, substrates) are contained in 11 containers, it is possible to generate a learned model according to the 7 types of waste. Then, when estimating the state of the waste, it is possible to select a learned model corresponding to the type of the waste to be estimated, and use the selected learned model to output the state of the waste to be estimated. This selection example will be described in detail with reference to FIG. 15.

[0069] The estimation unit 125 estimates the state of waste contained in a container to which a physical tag corresponding to the tag information acquired by the tag information acquisition unit 122 is attached, and stores the estimation result in the estimation result DB 160 of the storage unit 130. Specifically, the estimation unit 125 estimates the state of waste contained in a container included in the captured image output from the image acquisition unit 121 using the trained model selected by the selection unit 124. For example, the estimation unit 125 can identify an image area set in the captured image based on the setting information output from the setting unit 123. Therefore, the estimation unit 125 inputs the captured image in which the image area is set by the setting unit 123 into the trained model selected by the selection unit 124, and estimates the output result from the trained model as the state of the waste contained in that image area. This waste estimation method will be described in detail with reference to FIG. 15.

[0070] The output control unit 126 executes control to output various types of information from the user terminal 300. For example, in response to a request from the user terminal 300, the output control unit 126 provides the inference result stored in the inference result DB 160 of the storage unit 130 to the user terminal 300 and causes the display unit 306 to display the inference result.

[0071] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (e.g., a control program, a trained model DB 140 (see FIG. 7), a selection information DB 150 (see FIG. 8), an estimation result DB 160, and a tag DB) that are required for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 110. The storage unit 130 can be, for example, a read-only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0072] [Configuration example of imaging device] The imaging device 200 includes a communication unit 201, a control unit 202, a storage unit 203, and an imaging unit 204. As the imaging device 200, for example, a remote camera that can be remotely controlled can be used.

[0073] The communication unit 201 exchanges various types of information with the information processing device 100 using wireless communication under the control of the control unit 202 .

[0074] The control unit 202 controls each unit of the imaging device 200 based on a control program stored in the storage unit 203. The control unit 202 is realized by a processing device such as a CPU or a GPU. For example, the control unit 202 executes control to associate the captured image generated by the imaging unit 204 with imaging device identification information and transmit the image to the information processing device 100. Based on the imaging device identification information, the information processing device 100 can identify that the captured image was generated at the waste collection site WP1.

[0075] The storage unit 203 is a storage medium that stores various types of information. For example, the storage unit 203 stores various types of information (for example, a control program, imaging device identification information for identifying the imaging device 200) that is required for the control unit 202 to perform various processes. The storage unit 203 also stores various types of information acquired via the communication unit 201. The storage unit 203 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0076] The imaging unit 204 captures an image of a subject under the control of the control unit 202 to generate an image (image data), and outputs the generated image to the control unit 202. The imaging unit 204 is configured, for example, with an imaging element (image sensor) that receives light from the subject collected by lenses (e.g., multiple lenses that collect light from the subject), and an image processing unit that performs predetermined image processing on the image data generated by the imaging element. As the imaging element, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type imaging element can be used. For example, the imaging unit 204 captures an image of a subject included in an imaging range IM1 (see FIG. 1 ) to generate a captured image, and outputs the captured image to the control unit 202. The imaging unit 204 may also generate still images periodically or irregularly, or may generate moving images.

[0077] [Example of user terminal configuration] The user terminal 300 includes a communication unit 301 , a control unit 302 , a storage unit 303 , an operation unit 304 , a sound output unit 305 , and a display unit 306 .

[0078] The communication unit 301 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 202.

[0079] The control unit 302 controls each unit based on various programs stored in the storage unit 303. The control unit 302 is realized by a processing device such as a CPU, a GPU, etc. For example, the control unit 302 executes control to display status information relating to the status of the object provided from the information processing device 100 on the display unit 306.

[0080] The storage unit 303 is a storage medium that stores various types of information. For example, the storage unit 303 stores various types of information (for example, control programs and applications) required for the control unit 202 to perform various processes. The storage unit 303 also stores various types of information acquired via the communication unit 301. The storage unit 303 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0081] The operation unit 304 receives various operations from the user U1 and outputs the received operation details to the control unit 302. For example, when a user operation is received to cause the display unit 306 to display status information regarding the status of waste stored in each of the multiple containers C1 to C11 installed at the waste collection site WP1 (see FIG. 1), the operation unit 304 outputs operation information regarding the user operation to the control unit 302.

[0082] The sound output unit 305 outputs various sounds based on the control of the control unit 302. As the sound output unit 305, for example, one or more speakers can be used.

[0083] The display unit 306 displays various images under the control of the control unit 302. For example, a display panel such as an organic EL (Electro Luminescence) panel or an LCD (Liquid Crystal Display) panel can be used as the display unit 306. An example of the display on the display unit 306 is shown in FIG. 16.

[0084] [Example of contents of trained model DB] 7 is a diagram schematically illustrating each piece of information stored in the trained model DB 140. The trained model DB 140 is a database for managing a plurality of trained models used to estimate the state of the waste contained in the containers C1 to C11.

[0085] Trained model DB 140 stores trained model identification information 141, trained models 142, and attribute information 143 in association with each other. Note that each of these pieces of information is an example, and other information may also be stored in trained model DB 140.

[0086] Trained model identification information 141 is identification information for identifying each trained model stored in trained model 142.

[0087] The trained model 142 is a trained model used to estimate the state of the waste contained in the containers C1 to C11. In this embodiment, an example is shown in which one or more trained models are selected from a plurality of trained models according to the attributes of the waste. This selection example is shown in FIG. 15. Note that, although FIG. 7 shows an example in which a trained model is stored in the trained model DB 140 and used, a trained model stored in an external device may also be used.

[0088] Attribute information 143 is attribute information that indicates the attributes of the object (waste) that uses the trained model stored in trained model 142.

[0089] [Example of the contents of the selection information DB] 8 is a diagram showing each piece of information stored in the selection information DB 150. The selection information DB 150 is a database for managing each piece of information used to estimate the state of the waste contained in the containers C1 to C11.

[0090] The selection information DB 150 stores tag information 151, attribute information 152, trained model identification information 153, and image region information 154 in association with each other. Note that each of these pieces of information is an example, and other information may be stored in the selection information DB 150.

[0091] The tag information 151 is tag information acquired based on the physical tags PT1 to PT11 attached to the containers C1 to C11. In Fig. 8, an example is shown in which the tag information corresponding to the physical tag PT1 attached to the container C1 is "TG001", the tag information corresponding to the physical tag PT2 attached to the container C2 is "TG002", and the tag information corresponding to the physical tag PT11 attached to the container C11 is "TG011". Note that other tag information is not shown in the figure.

[0092] The attribute information 152 is attribute information that indicates the attributes of the object (waste) that uses the trained model whose identification information is stored in the trained model identification information 153. The attribute information 152 corresponds to the attribute information 143 (see FIG. 7).

[0093] The trained model identification information 153 is identification information that indicates the trained model to be used for the image area corresponding to the tag information stored in the tag information 151 when that tag information is acquired. The trained model identification information 153 corresponds to the trained model identification information 141 (see FIG. 7). Furthermore, the same trained model is often used for waste with the same attributes. Therefore, the trained model identification information 153 may also be the same for waste with the same attribute information 152.

[0094] The image area information 154 is information used when tag information stored in the tag information 151 is acquired and an image area corresponding to the tag information is set. This image area information will be described in detail with reference to FIGS. 9A, 9B, 11 to 14, etc. The image area to be set is sometimes referred to as a bounding box. This bounding box refers to an image area that surrounds an object included in a captured image. When a wireless tag is used as a physical tag, each piece of selection information such as the attribute information 152, the trained model identification information 153, and the image area information 154 may be stored in the wireless tag. In this case, the information processing device 100 can acquire and use each piece of selection information stored in the wireless tag.

[0095] [Image area information] 9A and 9B are diagrams schematically showing image region information stored in image region information 154 (see FIG. 8). 9A and 9B show an example in which a figure for a bounding box is set in advance, and the bounding box is set based on this figure and information about the physical tag (for example, the position, length, and angle of the physical tag in the captured image).

[0096] 9A shows an example of constructing image area information based on the relationship between the shape of the edge SE1 of the opening of container C1 when viewed from above and the physical tag PT1 attached to the edge SE1. The same applies to the relationship between the other containers C2 to C10, whose edges SE1 when viewed from above are rectangular, and the physical tags PT2 to PT10.

[0097] 9B shows an example of constructing image area information based on the shape of the edge SE11 of the opening of the container C11 when viewed from above and the relationship with the physical tag PT11 attached to the edge SE11 of the opening of the container C11. The same applies to the relationship between other containers and physical tags whose edge SE1 has a circular shape when viewed from above.

[0098] 9A, the shape of the edge of the opening of the container C1 to which the physical tag PT1 is attached (the shape of the edge when viewed from above) can be identified based on the physical tag PT1. Here, when using a captured image (a captured image from directly above the container C1) generated by the imaging device 200 provided directly above the container C1, it is assumed that the shape of the edge of the opening of the container C1 included in the captured image matches the shape of the edge SE1 associated with the physical tag PT1. Therefore, the shape of the edge of the opening of the container C1 to which the physical tag PT1 is attached can be identified based on the physical tag PT1.

[0099] However, when using a captured image generated by an imaging device 200 disposed diagonally above the container C1 (a captured image captured from diagonally above the container C1), it is assumed that the shape of the edge of the opening of the container C1 included in the captured image does not match the shape of the edge SE1 associated with the physical tag PT1. Therefore, in this case, the shape of the edge SE1 associated with the physical tag PT1 can be transformed based on the state of the physical tag PT1 included in the captured image (e.g., the position, length, and angle of the physical tag in the captured image), and the transformed shape can be identified as the shape of the edge of the opening of the container C1. For example, an affine transformation can be used as this transformation. Examples of this are shown in FIGS. 10A to 10D.

[0100] Affine Transformation 10A to 10D are diagrams showing an example of transition when a square figure S1 is subjected to affine transformation. In Fig. 10A to 10D, figures S2 to S5 after affine transformation are shown by thick lines.

[0101] Here, when affine transforming each coordinate (x, y) in two-dimensional space to coordinate (x', y'), the following formula is used: Therefore, it is possible to affine transform each coordinate (x, y) in a plane image, which is two-dimensional space, to coordinate (x', y') using the following formula.

[0102]

number

[0103] 10A shows an example of a transition when translating a figure S1 into a square figure S2. When translating the figure S1 in this way, the following affine matrix is ​​used. Note that tx and ty are parameters that specify the distance when translating.

[0104]

number

[0105] Figure 10B shows an example of a transition when scaling the figure S1 to convert it into a square figure S3. When scaling the figure S1 in this way, the following affine matrix is ​​used. Note that sx and sy are parameters that specify the scaling ratio when scaling.

[0106]

number

[0107] 10C shows an example of a transition when rotating the figure S1 to convert it into a square figure S4. When rotating the figure S1 in this way, the following affine matrix is ​​used. Note that θ is a parameter that specifies the angle when rotating.

[0108]

number

[0109] FIG. 10D shows an example of the transition when shearing (skewing) the figure S1 to transform it into a parallelogram figure S5. When shearing and transforming the figure S1 in this way, the following affine matrix is ​​used. Note that sh x , sh y is a parameter that specifies the distance at which shearing occurs.

[0110]

number

[0111] [Bounding box setting example] FIG. 11 is a diagram schematically showing an example of a setting method for setting a bounding box that includes waste (plastic bags) contained in a container C1 included in the imaging range IM1 (see FIG. 1).

[0112] As shown in FIG. 11(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. Specifically, the tag information acquisition unit 122 extracts the physical tags PT1 to PT11 included in the captured image using a known image recognition technique as described above. Next, the tag information acquisition unit 122 acquires tag information corresponding to each of the physical tags PT1 to PT11 extracted from the captured image based on the arrangement order of the colors constituting the physical tags PT1 to PT11 as described above. As described above, the tag information corresponding to each of the physical tags PT1 to PT11 can be acquired using the tag DB in the storage unit 130 that indicates the relationship between the physical tags and tag information.

[0113] Next, the setting unit 123 (see FIG. 6) extracts, from the selection information DB 150, image area information 154 (see FIG. 8) corresponding to the tag information (corresponding to the physical tag PT1) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 9A, the shape of a rectangular edge SE1 having the physical tag PT1 provided on one side is extracted.

[0114] 11(B), based on the physical tag PT1 extracted from the captured image (corresponding to the imaging range IM1), the setting unit 123 fits the extracted image region information (the shape of the rectangular edge SE1) to the shape of the edge of the opening of the container C1 included in the captured image. For example, the setting unit 123 identifies both ends E1' and E2' of the physical tag PT1 attached to the container C1 included in the imaging range IM1. Then, the setting unit 123 performs affine transformation on the rectangular edge SE1 so that both ends E1 and E2 of the rectangular edge SE1 coincide with both ends E1' and E2' of the physical tag PT1 included in the captured image. This affine transformation will be described in detail with reference to FIG. 12.

[0115] 11(C) shows an example in which the rectangular edge SE1 is affine transformed so that both ends E1 and SE2 of the rectangular edge SE1 coincide with both ends E1' and E2' of the physical tag PT1 included in the captured image (corresponding to the imaging range IM1). That is, the rectangular edge SE1 is affine transformed so that the rectangular edge SE1 coincides with the edge of the opening of the container C1 included in the captured image.

[0116] 11(D1), the setting unit 123 fits the affine-transformed rectangular edge SE1 to the edge of the opening of the container C1 included in the imaging range IM1 based on the position (base point) of the physical tag PT1 in the captured image. Then, the setting unit 123 sets an image area of ​​a predetermined size including the fitted rectangular edge SE1 in the captured image (corresponding to the imaging range IM1). This image area can be used as a bounding box BB1.

[0117] The size of the bounding box BB1 can be set appropriately depending on the size of the fitted rectangular edge SE1. For example, it is possible to select and use the smallest size that can contain the fitted rectangular edge SE1 from among a plurality of preset rectangle sizes. Also, for example, the maximum size (e.g., horizontal length) of the fitted rectangular edge SE1 may be used as a reference, and the size of the rectangle that contains that maximum size may be selected. Note that while FIG. 11(D1) shows an example of setting a rectangular bounding box BB1, other shapes may also be set as the bounding box.

[0118] For example, as shown in Fig. 11(D2), the shape of the fitted rectangular edge SE1 may be set as a bounding box BB2. Alternatively, the bounding box may be set based on the position and size of the physical tag PT1. For example, it is possible to set a rectangular bounding box centered on the position of the physical tag PT1 and with a rectangular size corresponding to the length of the physical tag PT1.

[0119] In this way, the bounding boxes BB1 and BB2 can be set based on the position, size, shape (degree of deformation), etc. of the physical tag PT1. In this case, as described above, the image area can be set to any shape, figure, etc.

[0120] [Affine transformation example] FIG. 12 is a diagram showing an example of transformation when affine transformation is performed on image region information (the shape of rectangular edge SE1) so as to match the edge of the opening of container C1.

[0121] For example, assume that the position of container C1 and the position and orientation (imaging direction) of imaging device 200 are fixed at waste collection site WP1. In this case, the distance from imaging device 200 to container C1 can be represented by r, and the position of container C1 relative to imaging device 200 can be represented by depression angle θ and azimuth angle φ. Furthermore, the spherical coordinates (r, θ, φ) can be converted to Cartesian linear coordinates (x, y, z) using the following equations.

[0122]

number

[0123] Furthermore, the opening surface of container C1 (the surface including the edge of the opening of container C1, the surface including physical tag PT1) is assumed to be on plane P (a virtual plane parallel to the floor surface). The height from the floor surface at the installation location of container C1 to the edge of the opening of container C1 is assumed to be h. Furthermore, it is assumed that the setting unit 123 is capable of grasping in advance the relationship between the three-dimensional space (three-dimensional coordinates) at waste collection site WP1 and the two-dimensional coordinates corresponding to the imaging range IM1.

[0124] In this case, the setting unit 123 identifies both ends E1', E2' of the physical tag PT1 acquired by the tag information acquisition unit 122, and calculates the length D1 of both ends E1', E2'. Next, the setting unit 123 calculates the length D2 (the length of ends E1', E3') of one side of the container C1 on which the physical tag PT1 is provided, based on the ratio (T1 / T2) of the length T1 (see FIG. 9A) of the physical tag PT1 in the image region information (the shape of the rectangular edge SE1) to the length T2 (see FIG. 9A) of one side on which the physical tag PT1 is provided. That is, the length D2 (=D1·T2 / T1) of the ends E1' and E3' on the plane P is calculated. This allows the setting unit 123 to identify the end E3' of the physical tag PT1 in three-dimensional space.

[0125] Next, the setting unit 123 changes the size of the image region information (the shape of the rectangular edge SE1) (see FIG. 9A) so that the length D2 of the ends E1' and E3' on the plane P matches the length T2 of the ends E1 and E3 of the image region information (the shape of the rectangular edge SE1). For example, the size of the image region information (the shape of the rectangular edge SE1) can be changed by an affine transformation (a transformation that enlarges or reduces a figure) shown in FIG. 10B. Then, the setting unit 123 draws the image region information (the shape of the rectangular edge SE1) after the size conversion on the plane P. This allows the setting unit 123 to identify both ends E4' and E5' of the physical tag PT1 in three-dimensional space.

[0126] The setting unit 123 can also identify the ends E1', E3', E4', and E5' of the physical tag PT1 included in the captured image corresponding to the imaging range IM1 based on the relationship between the three-dimensional space (three-dimensional coordinates) at the waste collection site WP1 and the two-dimensional coordinates corresponding to the imaging range IM1. Therefore, the setting unit 123 can calculate the crushed (sheared) area of ​​the opening of the container C1 in the captured image based on the relationship between the line segment between the ends E1' and E3' and the line segment between the ends E4' and E5' of the physical tag PT1 included in the captured image. That is, the shearing amount of the rectangle corresponding to the opening of the container C1 in the captured image can be calculated. The scaling size can be calculated based on the relationship between the length D1 of the line segment between the ends E1' and E2' of the physical tag PT1 and the length T1 of the line segment between the ends E1 and E2 of the image area information (the shape of the rectangular edge SE1). The rotation angle can be calculated based on the angle of the line segment between the ends E1' and E2' of the physical tag PT1 relative to a specific direction (e.g., the horizontal direction in the captured image).

[0127] Using the shear amount, scaling size, and rotation angle thus determined, it is possible to perform affine transformation so that the image region information (the shape of the rectangular edge SE1) coincides with the edge of the opening of the container C1, as shown in Figure 11(C). Furthermore, the image region information (the shape of the rectangular edge SE1) thus affine transformed is placed in the captured image so that the reference points (e.g., end E1' of the physical tag PT1 and end E1 of the physical tag PT1) coincide, as shown in Figures 11(D1) and 11(D2). This allows the bounding box BB1 or BB2 to be set.

[0128] In this way, if the position and imaging direction of the imaging device 200 are fixed and the size of the container C1 can be known in advance, the shear amount can be calculated based on the depression angle and azimuth angle.

[0129] The above example shows how affine transformation is performed to fit image area information (rectangular or circular edge shape) to each container based on the relationship between the three-dimensional space (three-dimensional coordinates) at waste collection point WP1 and the two-dimensional coordinates corresponding to imaging range IM1. It is also possible to measure container C1 included in imaging range IM1 in advance and calculate the shear amount of the opening (rectangular) edge of container C1 in advance based on the measurement results. In this case, the calculation results can be used to perform affine transformation.

[0130] Furthermore, for example, affine transformation can be performed based on the length (size) D1 and angle (e.g., angle relative to the horizontal direction) of both ends E1', E2' of the physical tag PT1. That is, affine transformation can be performed so that the length and angle of both ends E1, E2 of the rectangular edge SE1 coincide with the length and angle of both ends E1', E2' of the physical tag PT1. For example, using the difference value (e.g., angle, length) between end E2 and end E2' when end E1 and end E1' are coincident, an affine transformation can be performed by setting an affine matrix so that the difference value between end E2 and end E2' becomes 0. For each of these affine transformations, a known transformation method can be used.

[0131] The edge of the opening of the container C1 can be detected using a known image recognition technology (e.g., edge detection technology). Therefore, the angle between one of the four edges of the container C1, on which the physical tag PT1 is provided, and another adjacent edge can be obtained. The angle between these two adjacent edges can be used to estimate the shape (degree of deformation) of the rectangle corresponding to the edge of the opening of the container C1. For example, when using a captured image (a captured image from directly above the container C1) generated by the imaging device 200 provided directly above the container C1, the rectangle corresponding to the edge of the opening of the container C1 included in the captured image is rectangular. Therefore, a 90-degree angle can be obtained as the angle between the one side on which the physical tag PT1 is provided and the other adjacent side. On the other hand, when using a captured image (a captured image from diagonally above the container C1) generated by the imaging device 200 provided diagonally above the container C1, the rectangle corresponding to the edge of the opening of the container C1 included in the captured image has a sheared rectangular shape. In this case, a value less than 90 degrees is acquired as the angle between one side on which the physical tag PT1 is provided and another side adjacent thereto.

[0132] By providing physical tags on two or more of the four sides of the container C1, it is possible to calculate the shear amount by calculating the angles of the two or more physical tags without detecting the edge of the opening of the container C1. Examples of providing physical tags on two or more of the four sides of the container C1 are shown in Figures 13, 18A and 18B, and 19A and 19B.

[0133] Furthermore, if the positions and orientations of the containers C1 to C11 and the position and orientation (imaging direction) of the imaging device 200 are fixed at the waste collection site WP1, the shapes of the edges of the openings of the containers C1 to C11 included in the imaging range IM1 can be acquired in advance by measurement or the like. Therefore, in such a case, the shapes of the edges of the openings of the containers C1 to C11 included in the imaging range IM1 (deformed rectangular shapes) can be measured and acquired in advance and stored in the image region information 154 (see FIG. 8) corresponding to the tag information 151 (see FIG. 8). For example, the graphic information shown in FIG. 11(C) can be stored in the image region information 154 as the image region information corresponding to the container C1. This makes it possible to omit the above-mentioned processes such as affine transformation, and the graphic information stored in the image region information 154 can be used as is.

[0134] The above example illustrates the affine transformation of the rectangular edge SE1 based on the length and angle of both ends E1', E2' of the physical tag PT1. However, the affine transformation may also be performed based on other information. For example, the rectangular edge SE1 after the affine transformation may be compared with the edge of the opening of the container C1, and whether or not to use the affine-transformed rectangular edge SE1 may be determined based on the degree of similarity. For example, if the degree of similarity is equal to or greater than a threshold, the affine-transformed rectangular edge SE1 is used. If the degree of similarity is less than the threshold, the affine-transformed rectangular edge SE1 is again affine-transformed. The edge of the opening of the container C1 can be detected using known image recognition technology (e.g., edge detection technology). Furthermore, known image recognition technology (e.g., a determination technology based on brightness difference values) can be used to compare images.

[0135] [Example of physical tags on three sides] 12 shows an example of affine transformation in which one physical tag PT1 is attached to one side of one container C1. However, one container may be attached with multiple physical tags. This makes it possible to simplify the calculation of the affine transformation.

[0136] 12 is a diagram showing an example in which physical tags PT1A, PT1B, and PT1C are provided on three sides of the edge of container C1. In this case, image region information (the shape of rectangular edge SE1A) corresponding to the physical tags PT1A, PT1B, and PT1C provided on container C1 is stored in image region information 154 (see FIG. 8).

[0137] 12 shows a perspective view of the container C1 on the left side, and image area information (the shape of the rectangular edge SE1A) on the right side. In this case, similar to the example shown in FIG. 11(B), the setting unit 123 fits the extracted image area information (the shape of the rectangular edge SE1A) to the shape of the edge of the opening of the container C1 included in the captured image based on the physical tags PT1A, PT1B, and PT1C extracted from the captured image (corresponding to the imaging range IM1). In the example shown in FIG. 12, the physical tags PT1A, PT1B, and PT1C are provided on three sides of the container C1, so the setting unit 123 identifies the ends, lengths, and angles (e.g., the angles between the physical tags) of the physical tags PT1A, PT1B, and PT1C provided on the container C1 included in the imaging range IM1. The setting unit 123 then performs affine transformation on the image region information (the shape of the rectangular edge SE1A) so that the physical tags PT1A, PT1B, and PT1C on three sides of the image region information (the shape of the rectangular edge SE1A) match the physical tags PT1A, PT1B, and PT1C included in the captured image. A known affine transformation method can be used for this affine transformation. As shown in FIG. 12, by providing the physical tags PT1A, PT1B, and PT1C on three sides of the container C1, it is possible to calculate the angle by which the rectangle of the container C1 is deformed, which makes it easier to calculate the shear amount.

[0138] [Example of setting a bounding box for a cylindrical container] FIG. 14 is a diagram schematically showing an example of a setting method for setting a bounding box that includes the object (substrate) contained in the container C11 included in the imaging range IM1 (see FIG. 1).

[0139] 14(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. The method for acquiring this tag information is the same as the example shown in FIG.

[0140] Next, the setting unit 123 (see FIG. 6) extracts, from the selection information DB 150, image region information 154 (see FIG. 8) corresponding to the tag information (corresponding to the physical tag PT11) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 9B, the shape of a circular edge SE11 on which the physical tag PT11 is provided is extracted.

[0141] As shown in FIG. 14(B), based on the physical tag PT11 extracted from the captured image (corresponding to the imaging range IM1), the setting unit 123 fits the extracted image region information (the shape of the rectangular edge SE11) to the shape of the edge of the opening of the container C11 included in the captured image. For example, the setting unit 123 identifies both end portions E11′, E12′ and a central portion M13′ of the physical tag PT11 attached to the container C11 included in the imaging range IM1. Then, the setting unit 123 performs affine transformation on the circumferential edge SE11 so that both end portions E11, E12, and the central portion M13 of the circumferential edge SE11 coincide with both end portions E11′, E12′, and the central portion M13′ of the physical tag PT11 included in the captured image. A known transformation method can be used for this affine transformation.

[0142] For example, it is possible to perform affine transformation based on the circumferential shape specified by the lengths (sizes) of both ends E11', E12' of the physical tag PT11 and the central portion M13'. That is, it is possible to perform affine transformation so that the circumferential shape specified by both ends E11, E12 and the central portion M13 of the circumferential edge SE11 matches the circumferential shape specified by both ends E11', E12' and the central portion M13' of the physical tag PT11.

[0143] 14(C), the circumferential edge SE11 is affine transformed so that both ends E11, E12 and a central portion M13 of the rectangular shape of the edge SE11 coincide with both ends E11', E12' and a central portion M13' of the physical tag PT11 included in the captured image (corresponding to the imaging range IM1). That is, the circumferential edge SE11 is affine transformed so that the circumferential edge SE11 coincides with the edge of the opening of the container C11 included in the captured image.

[0144] Next, as shown in FIG. 14(D), the setting unit 123 fits the affine-transformed circumferential edge SE11 to the edge of the opening of the container C11 included in the imaging range IM1 based on the position (base point) of the physical tag PT11 in the captured image. Then, the setting unit 123 sets an image area of ​​a predetermined size including the fitted circumferential edge SE11 in the captured image (corresponding to the imaging range IM1). This image area can be used as a bounding box BB11. Alternatively, a bounding box may be set in the same manner as in FIG. 11(D2).

[0145] 11, the circumferential edge SE11 after the affine transformation may be compared with the edge of the opening of the container C11, and whether or not to use the circumferential edge SE11 after the affine transformation may be determined based on the degree of similarity. Bounding boxes can also be set for edges of other shapes in a similar manner.

[0146] [Example of selecting a trained model] 15 is a diagram showing an example of a selection method for selecting a trained model to be used when estimating the state of waste (plastic bags) contained in a container C1 included in the imaging range IM1 (see FIG. 1). Here, an example is shown in which the filling rate of waste is calculated as the state of the waste (plastic bags).

[0147] 15(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. The method for identifying this tag information is similar to the examples shown in FIGS. 11 to 14, etc.

[0148] Next, the selection unit 124 (see FIG. 6) extracts, from the selection information DB 150, trained model identification information 153 (see FIG. 8) corresponding to the tag information (corresponding to the physical tag PT1) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 15(B), trained model identification information "LL4" corresponding to the tag information (corresponding to the physical tag PT1) is selected. In this case, the fourth trained model LL4 corresponding to the trained model identification information "LL4" is selected from the trained model DB 140 (see FIG. 7) and used.

[0149] Next, the estimation unit 125 uses the trained model selected by the selection unit 124 to estimate the state of the waste contained in the container included in the captured image output from the image acquisition unit 121. For example, as shown in FIG. 15(C), the estimation unit 125 inputs the captured image in which the image area (bounding box BB1) is set by the setting unit 123 into the fourth trained model LL4 selected by the selection unit 124, and estimates the output result from the fourth trained model LL4 as the state (filling rate) of the waste (plastic bags) included in the image area (bounding box BB1). Then, the estimation unit 125 stores the estimation result in the estimation result DB 160 of the storage unit 130.

[0150] [Example of display on user terminal] Fig. 16 is a diagram showing an example of a display screen 310 displayed on the display unit 306 of the user terminal 300. Fig. 16 shows an example of displaying the estimated results when the filling rate of waste stored in the containers C1 to C11 installed at the waste collection site WP1 (see Fig. 1) is estimated.

[0151] A container information display area 311, a waste information display area 312, and a filling rate display area 313 are displayed in association with each other on the display screen 310. Each of these pieces of information can be displayed based on each piece of estimation result information stored in the estimation result DB 160 of the storage unit 130. Note that Fig. 16 shows an example in which only some (containers C1 to C4) of the containers C1 to C11 installed at the waste collection point WP1 are displayed. The other containers can be displayed using a scroll bar 314.

[0152] Container information for identifying each of the containers C1 to C11 installed at the waste collection point WP1 is displayed in the container information display area 311. Fig. 16 shows an example in which the container information corresponding to container C1 is displayed as "first container," the container information corresponding to container C2 is displayed as "second container," the container information corresponding to container C3 is displayed as "third container," and the container information corresponding to container C4 is displayed as "fourth container."

[0153] The waste information display area 312 displays the names of the waste materials to identify the waste materials contained in the containers C1 to C11 installed at the waste material collection point WP1.

[0154] The filling rate display area 313 displays the filling rate of each waste estimated by the estimation unit 125. When images generated by the imaging device 200 are acquired in real time and the filling rates of the waste are sequentially estimated, it is possible to change the display content of the filling rate display area 313 each time the estimation process is executed.

[0155] Furthermore, each piece of information that can be displayed on the display screen 310 can be output from the sound output unit 305. For example, each piece of information for each object (for example, audio information SS1) can be output from the sound output unit 305.

[0156] [Example of operation of information processing device] Fig. 17 is a flowchart showing an example of estimation processing in the information processing device 100. This estimation processing is executed by the control unit 120 (see Fig. 6) based on a program stored in the storage unit 130 (see Fig. 6). Fig. 17 also shows an example of estimating the filling rate of each waste contained in containers C1 to C11 installed at the waste collection site WP1 (see Fig. 1). This estimation processing is executed for each control cycle. This estimation processing will be explained with appropriate reference to Figs. 1 to 16.

[0157] In step S501 , the image acquisition unit 121 acquires a captured image generated by the imaging device 200 .

[0158] In step S502, the tag information acquisition unit 122 reads the physical tags PT1 to PT11 attached to the containers C1 to C11, respectively, based on the captured images acquired in step S501. A known image recognition technique can be used to read these physical tags.

[0159] In step S503, the tag information acquisition unit 122 acquires tag information of the objects (waste) contained in each of the containers C1 to C11 based on each of the physical tags PT1 to PT11 read in step S502.

[0160] In step S504, the setting unit 123 sets an image area including the object based on the tag information of the object acquired in step S503. The method for setting this image area can be the same as the examples shown in Figs. 11 to 14, etc.

[0161] In step S505, the selection unit 124 selects a trained model to be used when estimating the filling rate of the object based on the tag information of the object acquired in step S503. The method for selecting this trained model can be the same as the example shown in FIG.

[0162] In step S506, the estimation unit 125 uses the trained model selected in step S505 to estimate the filling rate of the object for which tag information was acquired in step S503. Specifically, the estimation unit 125 inputs the captured image for which the image area was set in step S504 to the selected trained model, and estimates the response result as the filling rate of the object.

[0163] These processes may be executed sequentially for each object, or may be executed in parallel for each object.

[0164] The estimation unit 125 sequentially stores the estimation results estimated in step S506 in the estimation result DB 160 of the storage unit 130. Furthermore, the output control unit 126 provides the estimation results stored in the estimation result DB 160 of the storage unit 130 to the user terminal 300 in response to a request from the user terminal 300. Furthermore, the control unit 302 of the user terminal 300 causes the display unit 306 to display the estimation results provided from the information processing device 100. For example, as shown in FIG. 16, it is possible to cause the display unit 306 to display a display screen 310. Furthermore, the control unit 302 of the user terminal 300 causes the sound output unit 305 to output audio information related to the estimation results provided from the information processing device 100. For example, as shown in FIG. 16, it is possible to cause the sound output unit 305 to output audio information SS1.

[0165] [Modification of physical tags] The above example shows a case where one physical tag is attached to one container. However, multiple physical tags may be attached to one container. This allows for more accurate setting of the bounding box.

[0166] 18A and 18B are diagrams illustrating an example in which physical tags PT1a and PT1b are provided on two edges of a container C1. FIG. 18A illustrates a perspective view of the container C1, and FIG. 18B illustrates the shape of the edge SE1a corresponding to the container C1, which is stored in the image region information 154 (see FIG. 8). In this manner, image region information corresponding to the physical tags PT1a and PT1b provided on the container C1 is stored in the image region information 154 (see FIG. 8). The setting unit 123 can set a bounding box using the positions, lengths, angles, etc. of the physical tags PT1a and PT1b provided on the two edges of the container C1. In this manner, providing two physical tags PT1a and PT1b makes it possible to obtain the angles of the two edges of the container C1, thereby facilitating the calculation of the shear amount.

[0167] 19A and 19B are diagrams illustrating an example in which physical tags PT11a and PT11b are provided at two opposing positions on the edge of a container C11. FIG. 19A illustrates a perspective view of the container C11, and FIG. 19B illustrates the shape of the edge SE11a corresponding to the container C11, which is stored in the image region information 154 (see FIG. 8). In this manner, image region information corresponding to the physical tags PT11a and PT11b provided on the container C11 is stored in the image region information 154 (see FIG. 8). The setting unit 123 can set a bounding box using the positions, lengths, angles, etc. of the physical tags PT11a and PT11b provided at two opposing positions on the container C11. In this manner, providing two physical tags PT11a and PT11b makes it possible to acquire the positional relationship between the two opposing positions on the container C11, thereby facilitating the calculation of the shear amount.

[0168] [Container Modification] In the above, examples have been shown in which a physical tag is provided on a box-shaped or cylindrical container. However, one or more physical tags may be provided on containers of other shapes. Figures 20A and 20B show examples of other shapes.

[0169] 20A and 20B are diagrams showing an example in which a physical tag PT13 is provided on one side of the edge of an L-shaped container C13 when viewed from above. FIG. 20A shows a perspective view of the container C13, and FIG. 20B shows the shape of the edge SE13 corresponding to the container C13, which is stored in the image area information 154 (see FIG. 8). In this manner, the image area information corresponding to the physical tag PT13 provided on the container C13 is stored in the image area information 154 (see FIG. 8). Note that the method of setting the image area is the same as in the above-described example. Furthermore, as in FIGS. 18A, 18B, 19A, and 19B, multiple physical tags may be provided on the container C13.

[0170] [Modification of information processing system] The above describes an example in which tag information of the containers C1 to C11 is acquired based on captured images generated by the imaging device 200. Here, the tag information of the containers C1 to C11 may be acquired using wireless communication. Therefore, the following describes an example in which tag information of the containers C1 to C11 is acquired using wireless communication.

[0171] [Example of use of information processing system] Fig. 21 is a diagram showing an example of use of the information processing system 1a. The information processing system 1a shown in Fig. 21 shows an example in which a communication device 400 is added to the information processing system 1 shown in Fig. 1, and physical tags PTT1 to PTT11 capable of wireless communication are provided on the containers C1 to C11. Note that, since other parts are common to the information processing system 1, the same reference numerals as in Fig. 1 are used and their description will be omitted. In the following, the physical tag PTT1 will be mainly described as an example, but the same applies to the physical tags PTT2 to PTT11.

[0172] The physical tag PTT1 is an integrated circuit (IC) tag capable of wireless communication. For example, the physical tag PTT1 can be provided inside a sheet that can be attached to the container C1. The physical tag PTT1 is an example of RFID (Radio Frequency Identification) and is a device capable of low-power wireless communication. The physical tag PTT1 can be configured as a wireless communication tag (RFID tag) compatible with RFID technology. For example, the physical tag PTT1 can be configured as an IC tag that employs a communication method such as Bluetooth Low Energy (BLE), Bluetooth, ZigBee, Low Power Wide Area (LPWA), or Ultra Wide Band (UWB), which are low-power communication modes. The maximum communication distance of the physical tag PTT1 is not particularly limited, but can be, for example, in the range of several tens of centimeters to several meters. For example, if the physical tag PTT1 complies with the BLE standard, it broadcasts packets at predetermined intervals (for example, every 1 to 10 seconds). A packet transmitted by this physical tag PTT1 includes a unique ID (Identification) (tag ID) that is identification information of the IC tag. Figures 21 to 23 show an example in which this tag ID is used as identification information (tag information) of the container C1 (waste).

[0173] That is, by reading the tag ID of the physical tag PTT1 attached to the container C1, it is possible to obtain the identification information of the waste (vinyl bag) contained in the container C1.

[0174] Note that identification information that allows tag information to be acquired from the captured image generated by the imaging device 200 may be provided on the surface of the sheet provided with the physical tag PTT1. For example, triangular identification information PTT1a (see FIG. 22) that allows identification of the base point, size, and direction may be provided. Note that color codes or the like shown in FIGS. 2 to 5 may also be provided.

[0175] [Physical tag configuration example] 22 is a diagram showing an example of identification information PTT1a provided on the surface of a sheet having a physical tag PT1a. Note that the identification information PTT1a can be colored to identify the attributes of the waste contained in the container C1.

[0176] The identification information PTT1a is a figure formed by an isosceles triangle. The center position of the base of the isosceles triangle is defined as a base point PP1, the length of a line segment PP2 from the base point PP1 to the apex angle of the isosceles triangle is defined as a size PPL1 of the identification information PTT1a, and the angle of the line segment PP2 with respect to the horizontal direction is defined as a direction θ1 of the identification information PTT1a. The base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be specified based on the identification information PTT1a included in the captured image. The base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be used to perform affine transformation on the image region information stored in the image region information 154 (see FIG. 8).

[0177] [Examples of using information processing equipment] Fig. 23 is a block diagram showing an example of the system configuration of the information processing system 1a. The information processing system 1a is a partial modification of the information processing system 1 shown in Fig. 6. Specifically, the information processing system 1a is provided with a communication device 400 for reading the physical tag PTT1. Note that the information processing system 1a has the same components as the information processing system 1 except for the provision of the communication device 400. Therefore, the components common to the information processing system 1 are given the same reference numerals, and some of these components will not be illustrated or described. Furthermore, the components different from the information processing system 1 will be described below as appropriate.

[0178] The information processing system 1a is configured by a plurality of devices that can be connected via a network N1. Fig. 23 shows, as an example, the information processing system 1a including an information processing device 100, an imaging device 200, a user terminal 300, and a communication device 400. Each of the information processing device 100, the imaging device 200, the user terminal 300, and the communication device 400 is connected to the network N1 by a communication method using wired communication or wireless communication.

[0179] Note that the information processing device 100, the imaging device 200, the user terminal 300, and the communication device 400 may be connected directly using wired or wireless communication without going through the network N1. Also, while Fig. 23 shows only the imaging device 200, the user terminal 300, and the communication device 400 as representative examples, the information processing system 1a may also include other imaging devices, other user terminals, and other communication devices installed at each location. Furthermore, as will be described later, if the position of the physical tag PTT1 can be identified by the communication device 400, the installation of the imaging device 200 may be omitted.

[0180] The communication device 400 and the physical tag PTT1 are connected by a direct connection using wireless communication without going through the network N1. Also, in Fig. 23, only the physical tag PTT1 and the container C1 are shown as representative examples, but the other physical tags PTT2 to PTT11 and the other containers C2 to C11 also constitute the information processing system 1a. Furthermore, as will be described later, when a plurality of communication devices 400 are used, the information processing system 1a is constituted by the plurality of communication devices 400.

[0181] The communication device 400 includes a communication unit 401 , a reading unit 402 , a control unit 403 , and a storage unit 404 .

[0182] The communication unit 401 exchanges various types of information with the information processing device 100 using wireless communication under the control of the control unit 403 .

[0183] The reading unit 402 is a communication unit that receives radio waves from the physical tag PTT1 provided on the container C1 and exchanges various information with the physical tag PTT1. Note that the wireless communication used by the physical tag PTT1 described above can be adopted as the wireless communication exchanged between the reading unit 402 and the physical tag PTT1. For example, the reading unit 402 reads the tag ID of the physical tag PTT1 and outputs this tag ID to the control unit 403.

[0184] The control unit 403 controls each unit of the communication device 400 based on a control program stored in the storage unit 404. The control unit 403 is realized by a processing device such as a CPU or a GPU. For example, the control unit 403 executes control to transmit the tag ID read by the reading unit 402 to the information processing device 100.

[0185] The storage unit 404 is a storage medium that stores various types of information. For example, the storage unit 404 stores various types of information (for example, a control program, location identification information for identifying the location where the communication device 400 is installed) that is required for the control unit 403 to perform various processes. The storage unit 404 also stores various types of information acquired via the communication unit 401. The storage unit 404 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0186] [Example of location measurement using wireless tags] Here, a description will be given of a measurement method for measuring the position of a container using a wireless tag as the physical tag PTT 1. For example, the position of the wireless tag (position of the container) can be measured using a known indoor positioning system or the like.

[0187] For example, it is possible to use a measurement method that estimates the position of a wireless tag based on radio waves (radio waves emitted by the wireless tag) received by multiple communication devices. For example, three or more receivers can be installed at a waste collection site WP1 (see Figure 21), and the radio waves (radio waves emitted by the wireless tag) received by these receivers can be acquired. The position of the wireless tag can then be estimated by triangulation (cross-azimuth method) using the radio wave intensities.

[0188] Furthermore, for example, a measurement method can be used in which angle information between one or more communication devices and a wireless tag is calculated based on communication between the communication devices and the wireless tag, and the position of the wireless tag is estimated based on the angle information. For example, a reception angle detection technique (AoA (Angle of Arrival)) or a radiation angle detection technique (AoD (Angle of Departure)) can be used to calculate the angle information. For example, assume that containers C1 to C11 are located within a one-floor facility (waste collection site WP1) and an AoA receiver is installed on the ceiling of that floor. In this case, it is possible to calculate the angle of incidence of the radio waves (radio waves emitted by the wireless tag) received by the receiver installed on the ceiling. Then, the position of each wireless tag present on the floor of the waste collection site WP1 can be estimated based on the angle of incidence. Alternatively, the position of each wireless tag may be estimated using a receiver that can detect the position of the wireless tag that emitted the radio waves based on the directionality and reception strength of the received radio waves.

[0189] Using these position estimation techniques, it is possible to estimate the position of the physical tag PTT1 in three-dimensional space (waste collection point WP1). The estimation result (tag information and position) is transmitted by the communication device 400 to the information processing device 100. Note that the communication device 400 may transmit measurement data of the received radio waves to the information processing device 100, and the information processing device 100 may estimate the position of the physical tag PTT1 using the above-mentioned position estimation method.

[0190] Furthermore, by fixing the imaging range IM1 of the imaging device 200, it is possible to associate a three-dimensional space (the waste collection point WP1) with a two-dimensional space (the captured image corresponding to the imaging range IM1). Therefore, the tag information acquisition unit 122 can estimate the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) based on the position of the physical tag PTT1 at the waste collection point WP1.

[0191] In this way, the tag information acquisition unit 122 acquires the tag ID (tag information) of the physical tag PTT1 acquired by the communication device 400 and the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) estimated by the above-mentioned estimation method. Then, the tag information acquisition unit 122 associates the position of the physical tag PTT1 (position in the captured image) with the acquired tag information and outputs them to the setting unit 123 and the selection unit 124.

[0192] The setting unit 123 sets a predetermined image area from the captured image output from the image acquisition unit 121 based on the position and tag information of the physical tag PTT1 acquired by the tag information acquisition unit 122. For example, it is possible to set an image area of ​​a predetermined range based on the position of the physical tag PTT1 from the captured image corresponding to the imaging range IM1 (see FIG. 1). For example, it is possible to set a rectangular image area with the position of the physical tag PTT1 as the center and a size according to the attribute corresponding to the tag information.

[0193] 22, the identification information PTT1a of the physical tag PTT1 can be read, and based on this identification information PTT1a, the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be identified. Therefore, the setting unit 123 can affine transform the image region information (corresponding to the tag information of the physical tag PTT1) stored in the image region information 154 (see FIG. 8) using the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a. In this case, the setting unit 123 identifies the edge of the opening of the container C1 in the captured image based on the information about the physical tag PTT1 (the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a) and the tag information output from the tag information acquisition unit 122 (tag information corresponding to the physical tag PTT1).

[0194] Specifically, the setting unit 123 extracts image area information 154 corresponding to the tag information (tag information corresponding to the physical tag PTT1) output from the tag information acquisition unit 122 from the selection information DB 150 (see FIG. 8). Then, the setting unit 123 identifies the edge of the opening of the container C1 based on the extracted image area information and information about the physical tag PT1 (base point PP1, size PPL1 of the identification information PTT1a, and direction θ1 of the identification information PTT1a). This identification method is the same as the identification method shown in FIGS. 11 to 14, etc. Next, the setting unit 123 identifies an image area including the edge of the opening of the identified container C1, and sets the image area in the captured image.

[0195] The selection unit 124 selects a trained model stored in the trained model DB 140 based on the tag information acquired by the tag information acquisition unit 122. Specifically, the selection unit 124 selects trained model identification information 153 corresponding to the tag information 151 acquired by the tag information acquisition unit 122 from the selection information DB 150 (see FIG. 8).

[0196] The estimation unit 125 uses the trained model selected by the selection unit 124 to estimate the state of waste stored in a container included in the captured image output from the image acquisition unit 121. Here, if it is possible to estimate the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) using wireless communication, it is possible to associate the trained model with the image area input thereto based on the position. Also, if it is not possible to estimate the position of the physical tag PTT1 using wireless communication, it is possible to associate the trained model corresponding to the identification information PTT1a of the physical tag PTT1 with the image area identified based on the color of the identification information PTT1a.

[0197] [Example of estimating waste weight using multiple trained models] In the above example, the state of waste (filling rate, weight) is estimated using one trained model according to the attributes of the waste contained in a container. However, it is also possible to estimate the state of waste (filling rate, weight) using multiple trained models according to the attributes of the waste.

[0198] Therefore, an example of estimating the weight of waste using multiple trained models will be described below. For example, it is possible to use a trained model (volume) that estimates the volume of waste contained in a container, a trained model (shape) that estimates the shape of the waste contained in the container, and a trained model (density) that estimates the density of the waste contained in the container.

[0199] The trained model (capacity) is, for example, a model that learns the volume (filling rate) as a state of waste. Specifically, it is possible to generate the trained model (capacity) by training a large amount of training data that associates images including the target waste (waste contained in a container) with the filling rate and weight of the waste in the container. The trained model (shape) is, for example, a model that learns the shape (volume) as a state of waste. Specifically, it is possible to generate the trained model (shape) by training a large amount of training data that associates images including the target waste (waste contained in a container) with the shape and weight of the waste in the container. The trained model (density) is, for example, a model that learns the density as a state of waste. Specifically, it is possible to generate the trained model (density) by training a large amount of training data that associates images including the target waste (waste contained in a container) with the density and weight of the waste in the container.

[0200] In addition, combinations of these trained models (trained model (volume), trained model (shape), trained model (density)) are generated for each waste attribute.

[0201] Furthermore, when estimating the weight of waste, it is possible to use the output values ​​(respective output values ​​for the volume, shape, and density of the target waste) from three trained models (trained model (volume), trained model (shape), trained model (density)) according to the attributes of the target waste. In other words, it is possible to estimate the weight of waste based on the three output values.

[0202] [Example of using a trained model (capacity)] For example, it is possible to estimate the weight of waste using the output result (filling rate v) of the trained model (capacity) and the following equation 1. w v =f(v)=k1·v …Equation 1

[0203] Here, the filling rate v is output as a value between 0 and 1, where 1 is the value when the container for the target waste is full. Also, k1 is a coefficient set based on the weight at a filling rate of 1 (100%).

[0204] For example, suppose the weight of PET garbage is to be estimated as waste. In this case, the ratio of PET bottles of various sizes and the filling rate are taken into consideration, and it is assumed that the weight when full is 60 kg. In this case, it is possible to set k1 = 60. The coefficient k1 can be set appropriately based on experiments, simulations, etc.

[0205] [Example using a trained model (shape)] For example, it is possible to estimate the weight of atypical waste using the output result (volume s) of the trained model (shape) and the following equation 2. w s =f(s)=k2·s …Equation 2

[0206] Here, k2 is a coefficient set based on specific gravity.

[0207] For example, let us consider the case where the weight of PET waste is to be estimated. In this case, using a known image recognition technique, it is possible to detect PET waste (non-standard waste) other than standard PET bottles (uncompressed and compressed shapes) based on their shape from among the objects contained in the captured image. For example, in the case of a standard PET board (board thickness 5 mm), the volume s when the length h (cm) × width I (cm) is 0.05 × h × I (cm 3 In this case, if k2 = 1.4 (specific gravity), the weight w s is 1.4×s(g).

[0208] In this way, when atypical waste is detected, it is possible to add the weight of the atypical waste estimated using the trained model (shape).

[0209] [Example using a trained model (density)] For example, it is possible to estimate the weight of waste using the output result (density d) of the trained model (density) and the following equation 3. w d =f(d)=k1·d·Δv …Equation 3

[0210] Here, the density d is the reciprocal of the compression rate. For example, if the compression rate is 30%, then d = 1 / 0.3.

[0211] For example, suppose you want to estimate the weight of PET waste. In this case, you can check the captured images in chronological order and measure the volume Δv and density d at the point when the waste density changes. Then, you can subtract these measurement results to estimate the weight of the waste.

[0212] Taking these into consideration, the weight w of the waste can be calculated using the following equation 4. Note that the following equation 4 is an example in which captured images generated in time series are used, and k is a value indicating the time axis.

[0213]

number

[0214] The first term of Equation 4 represents the estimated weight of standard waste. The second term of Equation 4 represents the estimated weight of non-standard waste. The third term of Equation 4 represents the estimated weight of compressed standard waste. That is, when non-standard waste is detected based on its shape from among the objects included in the captured image, the volume at that time is subtracted from the first term. When a change in density of compressed or other waste is detected from among the objects included in the captured image, the compressed volume at that time is subtracted from the first term. The weight based on the subtracted volume of the non-standard waste is then calculated using the second term. The weight based on the volume of compressed or other waste is then calculated using the third term. Note that Equation 4 is an example shown for ease of explanation, and other equations determined based on experiments, simulations, etc. may also be used.

[0215] In this way, the information processing device 100 can estimate the weight of the waste using multiple trained models according to the attributes of the target waste (a trained model corresponding to the volume of the waste, a trained model corresponding to the shape of the waste, and a trained model corresponding to the density of the waste). This makes it possible to improve the accuracy of estimating the weight of the waste.

[0216] [Example of effect of this embodiment] In recent years, various efforts to realize a circular economy have been widely underway. For example, there is a demand to recycle waste as a resource. For this reason, it is important to visualize the entire process of waste collection, processing, recycled material production, etc., and to achieve more advanced waste treatment and recycling. However, at many waste treatment sites, it is difficult to grasp the timing and amount of waste generated, which makes it difficult to plan efficient collection and processing.

[0217] Therefore, in this embodiment, containers C1 to C11 that store waste are provided with physical tags PT1 to PT11 and PTT1 to PTT11 that can read tag information. An imaging device 200 (or a communication device 400) is also installed at the waste disposal site. The information processing device 100 acquires tag information that identifies each waste based on an image including the containers C1 to C11 generated by the imaging device 200 (or wireless communication by the communication device 400), and estimates the state of the waste (e.g., weight, filling rate) using the tag information. Specifically, the information processing device 100 uses the tag information to set an image area including the waste to be estimated from the image (or wireless communication) including the containers C1 to C11, and selects at least one trained model from among multiple trained models. The information processing device 100 then inputs the captured image in which the image area has been set into the selected trained model, and estimates the state of the waste (e.g., weight, filling rate) based on the output from the trained model.

[0218] This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of waste from images containing the waste, thereby reducing the computational load associated with the estimation processing and increasing processing speed. In other words, the estimation processing load can be reduced, shortening processing time. Furthermore, since the state of waste (e.g., weight, fill rate) is estimated using the image region containing the waste to be estimated and a trained model corresponding to the attributes of the waste to be estimated, the accuracy of the estimation of the state of the waste can be improved. This makes it easy to grasp the timing and amount of waste generated at many waste disposal sites, facilitating efficient collection and disposal planning. In other words, more accurate weight estimation makes it easier to plan waste collection timing, optimal vehicle dispatch arrangements, and collection routes so as not to exceed the load capacity of waste collection vehicles. This contributes to the realization of a circular economy.

[0219] [Examples of application to other objects] In the above, waste has been described as an example of an object contained in a container. That is, an example has been shown in which the state (e.g., weight, filling rate) of waste (object) contained in a container is estimated. However, this embodiment can also be applied to objects other than waste contained in a container. For example, this embodiment can also be applied to the case in which the state (e.g., weight, filling rate) of agricultural products, marine products, industrial products, etc., as objects contained in a container is estimated. This makes it possible to easily grasp the time and amount of objects produced at the processing site of each object, making it easier to plan efficient transportation, processing, etc. In other words, it is possible to realize a circular economy.

[0220] In the above, an example has been shown in which the state (e.g., weight, filling rate) of an object contained in a container is estimated. However, this embodiment can also be applied to an object that is not contained in a container. In this case, for example, a physical tag can be attached to the object itself or its surroundings (e.g., a sign behind it), and the state (e.g., weight, filling rate) of the object can be estimated using this physical tag.

[0221] [Example of executing processing on other devices or systems] Although the above describes an example in which the tag information acquisition process, selection process, setting process, estimation process, etc. are executed in the information processing device 100, all or part of each of these processes may be executed in other devices. In this case, an information processing system is configured by the devices that execute part of each of these processes. For example, at least part of each process can be executed using devices available to the user (e.g., smartphones, tablet terminals, personal computers), various information processing devices such as servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0222] Furthermore, a part (or all) of the information processing system capable of executing the functions of the information processing device 100 (or the information processing system 1, 1a) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0223] [Configuration example and effect example of this embodiment] The main effects of the information processing device 100, the information processing system 1, 1a, the information processing method, and the program according to the embodiments of the present invention will be described below.

[0224] The information processing device 100 is an information processing device that estimates the state (e.g., weight, filling rate) of waste based on measurement data of waste (an example of a target object) using multiple trained models (first trained model LL1 to Nth trained model LLN (where N is a natural number)). The information processing device 100 includes an image acquisition unit 121 (an example of a measurement data acquisition unit) that acquires an image (image data) containing waste (an example of measurement data of waste), a tag information acquisition unit 122 that acquires tag information that identifies the waste, a selection unit 124 that selects at least one trained model from the multiple trained models based on the tag information, and an estimation unit 125 that inputs an image (including waste) to the selected trained model and estimates the state (e.g., weight, filling rate) of the waste based on the output from the trained model.

[0225] According to this, when using a trained model to estimate the state of waste (e.g., weight, filling rate) from an image containing waste, it is possible to select and use a trained model corresponding to the waste from among multiple trained models based on tag information that identifies the waste. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of waste from an image containing the waste, making it possible to reduce the amount of calculation involved in the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0226] The tag information can be information obtained from the physical tags PT1 to PT11, PTT1 to PTT11.

[0227] This makes it possible to appropriately acquire tag information about each waste by using the physical tags PT1 to PT11 and PTT1 to PTT11.

[0228] The physical tags PT1 to PT11 and PTT1 to PTT11 may be provided in association with the containers C1 to C11 that contain waste (an example of a target object).

[0229] According to this, even if physical tags are not attached to the waste itself, they can be attached in association with the containers C1 to C11 that store the waste. For example, if they cannot be attached directly to the containers C1 to C11, physical tags PT1 to PT11 and PTT1 to PTT11 can be attached around the containers C1 to C11 (for example, on a sign behind them). This makes it possible to appropriately acquire tag information for each waste. This makes it possible to select and use an appropriate trained model according to the waste, thereby improving the accuracy of the estimation process for estimating the state of the waste.

[0230] Containers C1 to C11 that store waste (an example of a target object) are provided with physical tags PT1 to PT11 and PTT1 to PTT11 that can read tag information. An image acquisition unit 121 (an example of a measurement data acquisition unit) acquires images (an example of measured measurement data) that capture the state inside the containers C1 to C11. An estimation unit 125 uses the images to estimate the state (e.g., weight, filling rate) of the waste inside the containers C1 to C11.

[0231] According to this, even if the waste itself does not have a physical tag, tag information for each waste can be appropriately acquired by attaching physical tags PT1 to PT11 and PTT1 to PTT11 to the containers C1 to C11 that contain the waste. This makes it possible to select and use an appropriate trained model according to the waste, thereby improving the accuracy of the estimation process for estimating the state of the waste.

[0232] The tag information is information that can identify the attributes of waste (an example of an object). Furthermore, the physical tags PT1 to PT11 and PTT1 to PTT11 are provided on the containers C1 to C11 according to the attributes of the waste contained in the containers.

[0233] According to this, physical tags PT1 to PT11 and PTT1 to PTT11 are provided according to the attributes of the waste contained in the containers C1 to C11, so it is possible to appropriately acquire tag information according to the attributes of each waste. This makes it possible to select and use an appropriate trained model according to the attributes of the waste, and it is possible to improve the accuracy of the estimation process for estimating the state of the waste.

[0234] The state of waste (an example of a target object) may be expressed as the filling rate of waste in the containers C1 to C11.

[0235] This makes it possible to select an appropriate trained model from multiple trained models according to the attributes of the waste and estimate the waste filling rate from an image containing the waste. This eliminates the need to simultaneously use multiple trained models to perform an estimation process to estimate the waste filling rate from an image containing the waste, making it possible to reduce the amount of calculation involved in the estimation process. In other words, it is possible to reduce the load on the estimation process and shorten the processing time.

[0236] Containers C1 to C11 that store waste (an example of a target object) are provided with physical tags PT1 to PT11 and PTT1 to PTT11 from which tag information can be read. The image acquisition unit 121 (an example of a measurement data acquisition unit) is an image acquisition unit that acquires images from an imaging device 200 that captures images of the containers C1 to C11 and generates images including the waste. The tag information acquisition unit 122 acquires tag information based on the physical tags PT1 to PT11 and PTT1 to PTT11 included in the images.

[0237] This makes it possible to acquire a captured image including the containers C1 to C11 that store waste, and acquire tag information based on the physical tags PT1 to PT11 and PTT1 to PTT11 attached to the containers C1 to C11 included in the captured image. This makes it possible to select and use an appropriate trained model according to the waste, thereby improving the accuracy of the estimation process for estimating the state of the waste.

[0238] Containers C1 to C11 that store waste (an example of a target object) are provided with physical tags PT1 to PT11 and PTT1 to PTT11 from which tag information can be read. The physical tags PT1 to PT11 and PTT1 to PTT11 can be at least one of a barcode, a two-dimensional code, a color code that can express a plurality of pieces of information using an arrangement of a plurality of colors, a graphic code that can express a plurality of pieces of information using an arrangement of a plurality of graphics that are given any of a plurality of colors, and a wireless tag that can transmit tag information using wireless communication.

[0239] This makes it possible to provide appropriate physical tags to the containers C1 to C11 that store waste, depending on the environment of the installation location of the containers C1 to C11, etc. This makes it possible to acquire tag information from the physical tags PT1 to PT11 and PTT1 to PTT11 depending on the environment of the installation location of the containers C1 to C11, etc. This makes it possible to select and use an appropriate trained model depending on the waste, and improves the accuracy of the estimation process for estimating the state of the waste.

[0240] The image acquisition unit 121 (an example of a measurement data acquisition unit) is an image acquisition unit that acquires an image from an imaging device 200 that captures an image of waste (an example of an object) and generates an image including the waste. The information processing device 100 further has a setting unit 123 that sets an image area including the waste from the image based on tag information. The estimation unit 125 inputs the image in which the image area has been set to a selected trained model, and sets the output from the trained model as the state of the waste (e.g., weight, filling rate).

[0241] According to this, an image containing waste can be input into a trained model corresponding to the waste, and the output from the trained model can be the state of the waste (e.g., weight, filling rate). This eliminates the need to subject all images generated by capturing images of the waste to the estimation process, making it possible to reduce the amount of calculation required for the estimation process. Furthermore, since only the image area containing the waste can be subject to the estimation process and the output result can be used as the state of the waste (e.g., weight, filling rate), it is possible to improve the accuracy of estimating the state of the waste.

[0242] In addition to waste, the condition (for example, weight, filling rate) of at least one of agricultural products, marine products, and industrial products may be estimated.

[0243] According to this, when using a trained model to estimate the state (e.g., weight, filling rate) of an object (at least one of waste, agricultural products, marine products, and industrial products) from an image including the object, it is possible to select and use a trained model corresponding to the object from among multiple trained models based on tag information that identifies the object. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of the object from an image including the object, thereby reducing the amount of calculation required for the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0244] The state of waste (an example of a target object) can be represented by the weight of the waste. In this case, the multiple trained models include a trained model corresponding to the volume (filling rate) of the waste, a trained model corresponding to the shape of the waste, and a trained model corresponding to the density of the waste. Furthermore, the selection unit 124 selects, from the multiple trained models based on the tag information, a trained model corresponding to the volume of the waste, a trained model corresponding to the shape of the waste, and a trained model corresponding to the density of the waste. The estimation unit 125 inputs measurement data to each selected trained model and estimates the weight of the waste based on the output from each trained model.

[0245] For example, when waste is made of resins such as PET, it is possible that the waste contains a mixture of standardized objects such as PET bottles and non-standardized objects such as rolls or plates. In this case, the volume of the standardized objects differs from the volume of the non-standardized objects, which may result in a large error in the weight calculated based on the volume. It is also possible that the waste contains a mixture of standardized objects (e.g., uncrushed PET bottles) and compressed standardized objects (e.g., crushed PET bottles). In this case, the volume of the standardized objects differs from the volume of the compressed standardized objects, which may result in a large error in the weight calculated based on the volume. Therefore, it is possible to improve the estimation accuracy by estimating the weight of waste using multiple trained models according to the attributes of the waste.

[0246] An information processing method according to an embodiment of the present invention is an information processing method for estimating the state (e.g., weight, filling rate) of waste based on measurement data of waste (an example of a target object) using multiple trained models (first trained model LL1 to Nth trained model LLN (where N is a natural number)). This information processing method includes an image acquisition process (an example of measurement data acquisition process) (step S501) for acquiring an image (image data) containing waste (an example of measurement data of waste), a tag information acquisition process (steps S502 and S503) for acquiring tag information for identifying the waste, a selection process (step S505) for selecting at least one trained model from multiple trained models based on the tag information, and an estimation process (step S506) for inputting an image (including waste) into the selected trained model and estimating the state (e.g., weight, filling rate) of the waste based on output from the trained model.

[0247] According to this, when using a trained model to estimate the state of waste (e.g., weight, filling rate) from an image containing waste, it is possible to select and use a trained model corresponding to the waste from among multiple trained models based on tag information that identifies the waste. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of waste from an image containing the waste, making it possible to reduce the amount of calculation involved in the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0248] A program according to an embodiment of the present invention is a program that causes a computer to execute an information processing method for estimating the state (e.g., weight, filling rate) of waste based on measurement data of the waste (an example of a target object) using multiple trained models (first trained model LL1 to Nth trained model LLN (where N is a natural number)). The program causes the computer to execute an image acquisition procedure (an example of a measurement data acquisition procedure) (step S501) for acquiring an image (image data) containing waste (an example of measurement data of the waste), a tag information acquisition procedure (steps S502 and S503) for acquiring tag information that identifies the waste, a selection procedure (step S505) for selecting at least one trained model from multiple trained models based on the tag information, and an estimation procedure (step S506) for inputting an image (including the waste) into the selected trained model and estimating the state (e.g., weight, filling rate) of the waste based on output from the trained model.

[0249] According to this, when using a trained model to estimate the state of waste (e.g., weight, filling rate) from an image containing waste, it is possible to select and use a trained model corresponding to the waste from among multiple trained models based on tag information that identifies the waste. This eliminates the need to simultaneously use multiple trained models to perform estimation processing to estimate the state of waste from an image containing the waste, making it possible to reduce the amount of calculation involved in the estimation processing. In other words, it is possible to reduce the load on the estimation processing and shorten the processing time.

[0250] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that makes it possible to realize this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0251] Each process described in this embodiment is executed based on a program that causes a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to an information processing device can store the program in the storage device of the information processing device. This makes it possible to cause the updated information processing device to execute each process described in this embodiment.

[0252] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments. [Explanation of symbols]

[0253] 1, 1a Information Processing System 100 Information processing device 110 Communications Department 120 control section 121 Image acquisition unit 122 Tag information acquisition unit 123 Settings 124 Selection Section 125 Estimation part 126 Output control section 130 Storage section 140 trained model DB 150 Selection Information DB 160 Estimation result DB 200 Imaging device 201 Communications Department 202 Control section 203 Storage section 204 Imaging unit 300 User Terminals 301 Communications Department 302 Control Unit 303 Storage section 304 Operation section 305 Sound output unit 306 Display section 400 Communication Equipment 401 Communications Department 402 Reading unit 403 Control Unit 404 Storage section C1~C11, C13 containers N1 Network PT1~PT11, PT1a, PT11a, PT13, PT20, PT30, PT40, PTT1~PTT11 Physical tags WP1 Waste collection point

Claims

1. An information processing device that estimates a state of an object based on measurement data of the object using a plurality of trained models, a measurement data acquisition unit that acquires measurement data of the object; a tag information acquisition unit that acquires tag information that identifies the object; a selection unit that selects at least one trained model from the plurality of trained models based on the tag information; an estimation unit that inputs the measurement data into the selected trained model and estimates the state of the object based on an output from the trained model; An information processing device having the above.

2. 2. The information processing device according to claim 1, The tag information is information obtained from a physical tag. Information processing device.

3. 3. The information processing device according to claim 2, The physical tag is associated with a container that contains the object. Information processing device.

4. 2. The information processing device according to claim 1, a container for accommodating the object is provided with a physical tag capable of reading the tag information; the measurement data acquisition unit acquires the measurement data measuring the state inside the container, The estimation unit estimates a state of the object in the container using the measurement data. Information processing device.

5. 5. The information processing device according to claim 4, the tag information is information that can identify an attribute of the object, The physical tag is provided on the container according to an attribute of the object contained in the container. Information processing device.

6. 5. The information processing device according to claim 4, The state of the object is a filling rate of the object in the container. Information processing device.

7. 2. The information processing device according to claim 1, a container for accommodating the object is provided with a physical tag capable of reading the tag information; the measurement data acquisition unit is an image acquisition unit that acquires an image from an imaging device that captures an image of the container and generates an image including the object, the tag information acquisition unit acquires the tag information based on the physical tag included in the image; Information processing device.

8. 2. The information processing device according to claim 1, a container for accommodating the object is provided with a physical tag capable of reading the tag information; The physical tag is at least one of a barcode, a two-dimensional code, a color code capable of expressing a plurality of pieces of information by an arrangement of a plurality of colors, a graphic code capable of expressing a plurality of pieces of information by an arrangement of a plurality of graphics to which any of a plurality of colors is applied, and a wireless tag capable of transmitting the tag information by wireless communication. Information processing device.

9. 2. The information processing device according to claim 1, the measurement data acquisition unit is an image acquisition unit that acquires an image from an imaging device that captures an image of the object and generates an image including the object, a setting unit that sets an image area including the object from the image based on the tag information; The estimation unit inputs the image in which the image area is set to the selected trained model, and sets an output from the trained model as the state of the object. Information processing device.

10. 10. The information processing device according to claim 1, The object is at least one of waste, agricultural products, marine products, and industrial products. Information processing device.

11. 10. The information processing device according to claim 1, the state of the object is the weight of the object, the plurality of trained models include a trained model corresponding to a volume of the object, a trained model corresponding to a shape of the object, and a trained model corresponding to a density of the object; the selection unit selects, from the plurality of trained models based on the tag information, a trained model corresponding to a volume of the object, a trained model corresponding to a shape of the object, and a trained model corresponding to a density of the object; The estimation unit inputs the measurement data into each selected trained model and estimates the weight of the object based on an output from each trained model. Information processing device.

12. An information processing method for estimating a state of an object based on measurement data of the object using a plurality of trained models, a measurement data acquisition process for acquiring measurement data of the object; a tag information acquisition process for acquiring tag information for identifying the object; A selection process of selecting at least one trained model from the plurality of trained models based on the tag information; a state estimation process of inputting the measurement data into the selected trained model and estimating the state of the object based on an output from the trained model; An information processing method including:

13. A program that causes a computer to execute an information processing method for estimating a state of an object based on measurement data of the object using a plurality of trained models, a measurement data acquisition step of acquiring measurement data of the object; a tag information acquisition step of acquiring tag information that identifies the object; a selection step of selecting at least one trained model from the plurality of trained models based on the tag information; a state estimation step of inputting the measurement data into the selected trained model and estimating the state of the object based on an output from the trained model; A program that causes a computer to execute the following.

Images