Information processing device, information processing method, and program

Abstract

To improve the accuracy of estimating the weight of an object using images.SOLUTION: An information processing device includes: an image acquisition unit configured to acquire images including a container that accommodates an object; a selection unit configured to select, from among multiple image processes used to estimate the weight of the object, an image process that corresponds to the volume of the object and an image process that corresponds to at least one of shape and density of the object based on attributes of the object; and an estimation unit configured to estimate the weight of the object from the image using each selected image process.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program for estimating the weight of an object using an image. [Background technology]

[0002] Conventionally, there are techniques for estimating the weight of an object using an image. For example, a technique has been proposed in which the type and amount of target garbage are identified based on an image of the target garbage, and the weight of the target garbage is estimated based on the identification result (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-087134 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned conventional technology makes it possible to estimate the weight of target waste using its volume. However, when the target waste is a resin such as PET (Poly Ethylene Terephthalate), the volume of an uncompressed PET bottle differs from the volume of a compressed PET bottle, which can lead to a large error in the weight based on the volume. It is also conceivable that PET bottles may be mixed with other resins, such as rolls of PET film and PET sheets. In this case, standardized items (e.g., PET bottles) and non-standardized items (e.g., PET film rolls and PET sheets) are mixed together, which can lead to a large error in the weight based on the volume.

[0005] The present invention aims to improve the estimation accuracy when estimating the weight of an object using an image. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device having an image acquisition unit that acquires an image including a storage unit that stores an object, a selection unit that selects, from among multiple image processes used when estimating the weight of the object, an image process that corresponds to the volume of the object and an image process that corresponds to at least one of the shape and density of the object based on the attributes of the object, and an estimation unit that estimates the weight of the object from the image using each selected image process. [Effects of the Invention]

[0007] According to an aspect of the present invention, when estimating the weight of an object using an image, the estimation accuracy can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of use of an information processing system. [Figure 2] FIG. 2 is a top view showing an example of the configuration of a physical tag. [Figure 3] FIG. 3 is a top view showing an example of the configuration of a physical tag related to a barcode. [Figure 4] FIG. 4 is a top view showing an example of the configuration of a physical tag related to a graphic code. [Figure 5] FIG. 5 is a top view showing an example of the configuration of a physical tag relating to another graphic code. [Figure 6] FIG. 6 is a block diagram illustrating an example of the system configuration of the information processing system. [Figure 7] FIG. 7 is a diagram showing each piece of information stored in the setting information DB. [Figure 8] FIG. 8 is a diagram showing each piece of information stored in the trained model DB. [Figure 9A] FIG. 9A is a diagram showing an example of rectangular image region information. [Figure 9B] FIG. 9B is a diagram showing an example of circular image region information. [Figure 10A] FIG. 10A is a diagram showing an example of transition when a figure is subjected to affine transformation and translation. [Figure 10B] FIG. 10B is a diagram showing an example of transition when a figure is enlarged or reduced by affine transformation. [Figure 10C] FIG. 10C is a diagram showing an example of a transition when a figure is rotated by affine transformation. [Figure 10D] FIG. 10D is a diagram showing an example of transition when a figure is sheared by affine transformation. [Figure 11] FIG. 11 is a diagram showing an example of setting a bounding box for a box-shaped container. [Figure 12] FIG. 12 is a diagram showing an example of transformation when image region information is subjected to affine transformation. [Figure 13] FIG. 13 is a diagram showing an example in which physical tags are provided on three sides of a container. [Figure 14] FIG. 14 is a diagram showing an example of setting a bounding box for a cylindrical container. [Figure 15] FIG. 15 is a diagram illustrating an example of selecting a group of trained models. [Figure 16] FIG. 16 is a diagram showing an example of the transition of waste and estimation results. [Figure 17] FIG. 17 is a diagram showing an example of the transition of waste and estimation results when an irregular object is detected. [Figure 18] FIG. 18 is a diagram showing an example of transition of waste and estimation results when a change in density of a standardized object is detected. [Figure 19] FIG. 19 is a diagram showing an example of a display screen on a user terminal. [Figure 20] FIG. 20 is a flowchart illustrating an example of the estimation process. [Figure 21] FIG. 21 is a flowchart showing an example of the correction process. [Figure 22A] FIG. 22A is a perspective view showing an example in which physical tags are provided on two edges of a box-shaped container. [Figure 22B] FIG. 22B is a diagram showing an example of image area information when physical tags are provided on two sides of the edge of a box-shaped container. [Figure 23A]FIG. 23A is a perspective view showing an example in which physical tags are provided at two positions on the edge of a cylindrical container. [Figure 23B] FIG. 23B is a diagram showing an example of image area information when physical tags are provided at two positions on the edge of a cylindrical container. [Figure 24A] FIG. 24A is a perspective view showing an example in which a physical tag is provided on one side of the edge of a container according to a modified example. [Figure 24B] FIG. 24B is a diagram showing an example of image area information when a physical tag is provided on one side of the edge of a container according to a modified example. [Figure 25] FIG. 25 is a diagram showing a modified example in which an information processing system is used. [Figure 26] FIG. 26 is a diagram showing an example of identification information provided on the surface of a physical tag. [Figure 27] FIG. 27 is a block diagram showing an example of a system configuration according to a modified example of the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0009] The embodiments described below are not limited to the drawings described by the brief description of the drawings.

[0010] (1) One aspect of the present invention is an information processing device having an image acquisition unit that acquires an image including a storage unit that stores an object; a selection unit that selects, from among multiple image processes used when estimating the weight of the object, an image process that corresponds to the volume of the object and an image process that corresponds to at least one of the shape and density of the object based on the attributes of the object; and an estimation unit that estimates the weight of the object from the image using each selected image process.

[0011] According to this, when estimating the weight of an object from an image including the object, it is possible to select and use image processing corresponding to the object from among multiple image processing methods based on the attributes of the object. This eliminates the need to simultaneously execute image processing that does not correspond to the attributes of the object to execute estimation processing to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation processing. Furthermore, because the weight of the object is estimated using multiple image processing methods corresponding to the attributes of the object simultaneously, it is possible to improve the estimation accuracy.

[0012] (2) One aspect of the present invention is an information processing device according to (1), wherein the image processing is image processing using a trained model used to estimate the weight of the object, and the selection unit selects, from among the plurality of trained models, a trained model corresponding to the volume of the object and a trained model corresponding to at least one of the shape and density of the object based on the attributes of the object, and the estimation unit estimates the weight of the object from the image using each selected trained model.

[0013] According to this, when using a trained model to estimate the weight of an object from an image including the object, it is possible to select and use a group of trained models corresponding to the object from among multiple trained models based on the attributes of the object. This eliminates the need to simultaneously use multiple trained models that do not correspond to the attributes of the object to perform an estimation process to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation process. Furthermore, because the weight of the object is estimated simultaneously using multiple trained models that correspond to the attributes of the object, it is possible to improve the estimation accuracy.

[0014] (3) One aspect of the present invention is an information processing device as described in (1) or (2), wherein the object or the storage unit is provided with a physical tag capable of reading attributes of the object, and the information processing device further has an attribute acquisition unit that acquires attributes of the object using the physical tag.

[0015] This makes it possible to acquire the attributes of an object using a physical tag attached to the object or the container it is stored in. This allows for the appropriate selection and use of a trained model group corresponding to the object.

[0016] (4) One aspect of the present invention is an information processing device described in (2), in which the plurality of trained models are set for each of the plurality of attributes of the object, and the selection unit selects a trained model corresponding to the attribute of the object from among the plurality of trained models.

[0017] This makes it possible to select and use a group of trained models corresponding to an object from among multiple trained models set for each attribute of the object. This eliminates the need to simultaneously use multiple trained models that do not correspond to attributes to perform an estimation process to estimate the weight of the object from an image containing the object, thereby reducing the amount of calculation required for the estimation process. Furthermore, because the weight of the object is estimated simultaneously using multiple trained models that correspond to the attributes of the object, it is possible to improve estimation accuracy.

[0018] (5) One aspect of the present invention is an information processing device described in (4), in which the selection unit selects a trained model corresponding to the capacity according to the attributes of the object and a trained model corresponding to the shape according to the attributes of the object.

[0019] This allows the weight of an object to be estimated simultaneously using a trained model corresponding to the capacity based on the attributes of the object and a trained model corresponding to the shape based on the attributes of the object, thereby improving estimation accuracy.

[0020] (6) One aspect of the present invention is an information processing device described in (4), in which the selection unit selects a trained model corresponding to the capacity according to the attributes of the object and a trained model corresponding to the density according to the attributes of the object.

[0021] This allows the weight of an object to be estimated simultaneously using a trained model corresponding to the capacity according to the attributes of the object and a trained model corresponding to the density according to the attributes of the object, thereby improving the estimation accuracy.

[0022] (7) One aspect of the present invention is an information processing device described in (4), in which the selection unit selects a trained model corresponding to the capacity according to the attributes of the object, a trained model corresponding to the shape according to the attributes of the object, and a trained model corresponding to the density according to the attributes of the object.

[0023] This allows the weight of an object to be estimated simultaneously using a trained model corresponding to the capacity based on the attributes of the object, a trained model corresponding to the shape based on the attributes of the object, and a trained model corresponding to the density based on the attributes of the object, thereby improving the estimation accuracy.

[0024] (8) One aspect of the present invention is an information processing device according to any one of (2), (4) to (7), wherein the image acquisition unit acquires the images from an imaging device that captures an image of the storage unit to generate the images, and the estimation unit estimates the weight of the object at the time of capturing each of the multiple images generated in chronological order by the imaging device, and estimates the latest weight of the object based on the estimation results for each of the images.

[0025] This allows the weight of the object at the time each image was captured to be estimated for each of multiple images generated in chronological order, and the latest weight of the object to be estimated based on the estimation results for each image, thereby improving estimation accuracy.

[0026] (9) One aspect of the present invention is an information processing device as described in (8), further comprising a detection unit that detects the target object of an atypical object in the storage unit, and when the target object of an atypical object is detected, the selection unit selects a trained model corresponding to the target object of an atypical object for the image generated at the time of the detection.

[0027] According to this, for images generated when detecting an atypical object, the weight of the object is estimated using a trained model corresponding to the atypical object, and the weight of the latest object is estimated based on the estimation result, thereby improving estimation accuracy.

[0028] (10) One aspect of the present invention is an information processing device described in any of (1) to (9), wherein the storage unit stores a plurality of the objects, the shape is the shape of each of the plurality of objects, the density is the density of the plurality of objects in the storage unit, and the capacity is the capacity of the plurality of objects in the storage unit.

[0029] This makes it possible to select and use a group of trained models that correspond to the attributes of the object stored in the storage unit, such as shape, density, and volume, which reduces the amount of calculation required for the estimation process and improves estimation accuracy.

[0030] (11) One aspect of the present invention is an information processing device according to any one of (1) to (10), wherein the object is at least one of waste, agricultural products, marine products, and industrial products.

[0031] This also makes it possible to reduce the amount of calculation required for the estimation process and improve estimation accuracy when using a trained model to estimate the weight of an object (at least one of waste, agricultural products, marine products, and industrial products) from an image containing the object.

[0032] (12) One aspect of the present invention is an information processing method including an image acquisition process for acquiring an image including a storage section for storing an object; a selection process for selecting, from among a plurality of image processes used when estimating the weight of the object, an image process corresponding to the volume of the object and an image process corresponding to at least one of the shape and density of the object based on the attributes of the object; and an estimation process for estimating the weight of the object from the image using each selected image process.

[0033] According to this, when estimating the weight of an object from an image including the object, it is possible to select and use image processing corresponding to the object from among multiple image processing methods based on the attributes of the object. This eliminates the need to simultaneously execute image processing that does not correspond to the attributes of the object to execute estimation processing to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation processing. Furthermore, because the weight of the object is estimated using multiple image processing methods corresponding to the attributes of the object simultaneously, it is possible to improve the estimation accuracy.

[0034] (13) One aspect of the present invention is a program that causes a computer to execute an image acquisition procedure for acquiring an image including a storage section that stores an object; a selection procedure for selecting, from among multiple image processes used when estimating the weight of the object, an image process that corresponds to the volume of the object and an image process that corresponds to at least one of the shape and density of the object based on the attributes of the object; and an estimation procedure for estimating the weight of the object from the image using each selected image process.

[0035] According to this, when estimating the weight of an object from an image including the object, it is possible to select and use image processing corresponding to the object from among multiple image processing methods based on the attributes of the object. This eliminates the need to simultaneously execute image processing that does not correspond to the attributes of the object to execute estimation processing to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation processing. Furthermore, because the weight of the object is estimated using multiple image processing methods corresponding to the attributes of the object simultaneously, it is possible to improve the estimation accuracy.

[0036] Hereinafter, embodiments will be described with reference to the accompanying drawings.

[0037] [Example of use of information processing system] Fig. 1 is a diagram showing an example of using the information processing system 1. Fig. 1 shows an example in which a plurality of types of waste are stored in a plurality of containers C1 to C5 at a waste collection site WP1. Fig. 1 also shows an example in which physical tags PT1 to PT5 are provided on each of the plurality of containers C1 to C5 to identify the waste stored in each container.

[0038] For example, an example is shown in which container C1 contains PET (Polyethyleneterephthalate), container C2 contains glass scraps, container C3 contains vinyl bags, container C4 contains hard plastic, and container C5 contains iron scraps. Note that containers C1 to C5 can be made of materials and have structures, etc., that correspond to the waste to be contained therein. Also, Figure 1 etc. shows an example in which containers C1 to C5 are containers whose contents can be seen from the outside. For example, they may be made of a metal material with a mesh structure that allows the contents to be seen from the outside, or they may be made of a transparent material with a box-shaped or cylindrical structure that allows the contents to be seen from the outside. Note that various known technologies can be used for the materials, structures, etc. of containers C1 to C5 that correspond to the waste to be contained therein, and therefore detailed explanations thereof will be omitted here.

[0039] In addition, in this embodiment, an example is shown in which waste stored in containers C1 to C5 is identified using physical tags PT1 to PT5 included in the imaging range IM1 of the imaging device 200. The physical tags PT1 to PT5 will be described in detail with reference to FIG. 2. Also, configuration examples of the information processing device 100 and the imaging device 200 will be described in detail with reference to FIG. 6. Also, the bounding box BB1 and the like will be described in detail with reference to FIGS. 11 to 14, etc.

[0040] [Physical tag configuration example] FIG. 2 is a top view showing an example of the configuration of the physical tag PT1. Here, only the physical tag PT1 is shown as a representative example, but the same applies to the other physical tags PT2 to PT5. Furthermore, this embodiment shows an example in which the physical tags PT1 to PT5 are used to acquire the attributes and positions of the target object (waste) (or the attributes and positions of the corresponding containers C1 to C5). Note that the physical tags PT1 to PT5 can be changed as appropriate depending on the positions at which they are attached to the containers. For example, when the physical tag PT5 is attached to the edge of the opening of the container C5, the physical tag PT5 can be configured along the circumferential edge.

[0041] FIG. 2 shows an example of a color code in which five rectangles of a predetermined color are arranged in a row as the physical tag PT1. This color code can be configured by coloring the rectangle R1 at the end of the arrangement direction black, and coloring the other four rectangles R2 to R5 other than the black rectangle R1 in a color other than black. In this case, it is possible to represent multiple pieces of information by changing the colors of the four rectangles R2 to R5 arranged in a row. For example, an arrangement in which rectangle R2 is blue, rectangle R3 is red, rectangle R4 is yellow, and rectangle R5 is green can be used as a color code representing "PET."

[0042] Note that, in addition to the color code shown in FIG. 2, other physical tags readable by the imaging device 200 may be used, or other wireless tags may be used. Examples of other physical tags readable by the imaging device 200 are shown in FIGS. 3 to 5. Ordinary barcodes and two-dimensional codes (e.g., QR Code (registered trademark)) may also be used as physical tags. Examples of wireless tags are shown in FIGS. 25 to 27. These physical tags are associated with unique identification information (tag information). For example, the information processing device 100 stores a tag DB (Data Base) 140 indicating the relationship between physical tags and tag information in the storage unit 130 (see FIG. 6), and can acquire tag information associated with the read physical tag using the tag DB 140. Note that the tag DB 140 may be stored in an external device and acquired from the external device for use.

[0043] For example, the physical tag PT1 can be a sheet-like member (not shown) that is attached to the object, and five rectangles R1 to R5, each colored a specific number, can be arranged in a line on this sheet-like member. The rectangles R1 to R5 have the same length L1 in the arrangement direction. The lengths CR1 to CR4 between the rectangles R1 to R5 in the arrangement direction are also the same. The lengths L1 in the arrangement direction may be different from each other, and the lengths CR1 to CR4 in the arrangement direction may also be different from each other. The lengths CR1 to CR4 in the arrangement direction may also be zero.

[0044] [Barcode configuration example] FIG. 3 is a top view showing an example of the configuration of a physical tag PT20 relating to a barcode.

[0045] FIG. 3 shows an example of a color code in which multiple rectangles are arranged in a row as the physical tag PT20. This color code can be configured by placing standard bars (rectangle R21 of length L11 and rectangle R26 of length L12) at both ends to indicate the direction of the arrangement, and arranging four bars of two different lengths (long bars, short bars) between them. Note that the arrangement may also be configured with three types of information: bars of two different lengths (long bars, short bars) and blank spaces where no bars are arranged. In this case, multiple pieces of information can be represented by changing the bars of two different lengths (long bars, short bars) and blank spaces arranged between the standard bars at both ends (rectangles R21, R26).

[0046] 3 shows an example of arranging a long bar (rectangle R22 of length L13), a short bar (rectangle R23 of length L14), a short bar (rectangle R24 of length L14), and a long bar (rectangle R25 of length L13). For example, this arrangement can be used as a barcode representing "PET." In this way, the physical tag PT20 can represent multiple pieces of information by varying the horizontal length of each rectangle.

[0047] [Example of shape code configuration] FIG. 4 is a top view showing an example of the configuration of a physical tag PT30 relating to a graphic code.

[0048] FIG. 4 shows an example of a graphic code in which four types of graphics (circle (oval), triangle, square, inverted triangle) are arranged in a row as the physical tag PT30. This graphic code can be configured by placing standard bars (rectangles R31, R36) at both ends to indicate the direction of the arrangement, and arranging four of the four types of graphics between them. The four types of graphics may also be colored. In this case, multiple pieces of information can be represented by changing the four types of graphics arranged between the standard bars (rectangles R31, R36) at both ends and the colors assigned to them.

[0049] Figure 4 shows an example of arranging a blue circle (oval) graphic R32, a red triangle graphic R33, a yellow square graphic R34, and a green inverted triangle graphic R35. For example, this arrangement can be used as a graphic code representing "PET." In this way, the physical tag PT30 can represent multiple pieces of information by combining multiple types of graphics and their colors.

[0050] [Example of shape code for changing vertical length] FIG. 5 is a top view showing an example of the configuration of a physical tag PT40 relating to a graphic code whose length in the up-down direction (vertical direction, height direction) is changed.

[0051] FIG. 5 shows an example of a graphic code as a physical tag PT50, in which four types of graphics (circle (oval), triangle, square, inverted triangle) whose lengths can be changed in two directions (vertical and height directions) are arranged in a row. This graphic code can be configured by placing standard bars (rectangles R41 and R46) at both ends to indicate the direction of the arrangement, and arranging four of the four types of graphics whose lengths can be changed in the vertical direction between them. Furthermore, each of the four types of graphics may be assigned a color. In this case, multiple pieces of information can be represented by changing the four types of graphics arranged between the standard bars (rectangles R41 and R46) at both ends, the colors assigned to them, and the vertical lengths.

[0052] 5 shows an example of arranging a circle (oval) R42 with a long vertical line in blue, a triangle R43 with a short vertical line in red, a square R44 with a short vertical line in yellow, and an inverted triangle R45 with a long vertical line in green. For example, this arrangement can be used as a graphic code representing "PET." In this way, the physical tag PT40 can represent multiple pieces of information by combining multiple types of shapes, their colors, and their vertical lengths.

[0053] [Example of information processing system configuration] FIG. 6 is a block diagram showing an example of the system configuration of the information processing system 1. As shown in FIG.

[0054] The information processing system 1 is composed of multiple devices that can be connected via a network N1. FIG. 6 shows an example of the information processing system 1 including an information processing device 100, an imaging device 200, and a user terminal 300. The information processing device 100, the imaging device 200, and the user terminal 300 are each connected to the network N1 by a communication method using wired communication or wireless communication. The network N1 is a network such as a public line network or the Internet. In this case, the wireless communication may be a mobile communication network (e.g., standards such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), and 6G (6th Generation)). Alternatively, at least one of wireless communication standards such as wireless LAN (e.g., Wi-Fi (Wireless Fidelity)), Bluetooth (registered trademark), and ZigBee (registered trademark) may be used. In addition, multiple frequency bands (e.g., UHF band and 2.4 GHz band) may be used in combination.

[0055] Note that the information processing device 100, the imaging device 200, and the user terminal 300 may be directly connected using wired or wireless communication without going through the network N1. Also, although only the imaging device 200 and the user terminal 300 are shown as representative examples in Fig. 6, other imaging devices and other user terminals installed at various locations may also constitute the information processing system 1.

[0056] [Configuration example of information processing device] The information processing device 100 includes a communication unit 110, a control unit 120, and a storage unit 130. The information processing device 100 can be, for example, a server realized by one or more devices.

[0057] The communication unit 110, under the control of the control unit 120, exchanges various types of information with other devices using wired or wireless communication.

[0058] The control unit 120 controls each unit of the information processing device 100 based on a control program stored in the storage unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Specifically, the control unit 120 includes an image acquisition unit 121, a tag information acquisition unit 122, a setting unit 123, a detection unit 124, a selection unit 125, an estimation unit 126, an output control unit 127, a vehicle dispatch processing unit 128, and a correction unit 129.

[0059] The image acquisition unit 121 acquires the captured image (including image data that has been subjected to image processing and its accompanying information) generated by the imaging device 200 via the communication unit 110. Then, the image acquisition unit 121 outputs the acquired captured image to the tag information acquisition unit 122, the setting unit 123, the detection unit 124, and the estimation unit 126. The image acquisition unit 121 is an example of a measurement data acquisition unit that acquires measurement data of an object (e.g., waste).

[0060] Here, the object shown in this embodiment includes an object in which a regular object and an irregular object are mixed. For example, when the object is PET (Polyethyleneterephthalate) resin, it is assumed that a regular object such as a PET bottle and an irregular object such as a roll or plate are mixed. Furthermore, the measurement data of the object shown in this embodiment also includes data measured in a state in which one or more regular or irregular objects are present. For example, when the object is PET resin, it is assumed that one or more regular objects such as a PET bottle are mixed with one or more irregular objects (e.g., roll or plate). Furthermore, it is assumed that the regular objects include a mixture of normal regular objects (e.g., uncrushed PET bottles) and compressed regular objects (e.g., crushed PET bottles).

[0061] The tag information acquisition unit 122 reads a physical tag from the captured image output from the image acquisition unit 121 and acquires tag information based on the read physical tag. Then, the tag information acquisition unit 122 associates the position of the read physical tag (the position in the captured image) with the acquired tag information and outputs them to the setting unit 123, the detection unit 124, and the selection unit 125. For example, when using the physical tag PT1 shown in FIG. 2, it is possible to read the physical tag PT1 from the captured image using a known image recognition technique (e.g., an object detection technique using pattern matching). Then, the tag information acquisition unit 122 acquires tag information associated with the read physical tag PT1 using the tag DB 140. As described above, the tag DB 140, which indicates the relationship between the physical tag and the tag information, is stored in the storage unit 130.

[0062] The setting unit 123 sets a predetermined image area from the captured image output from the image acquisition unit 121 based on the tag information acquired by the tag information acquisition unit 122, and outputs setting information about the set image area to the detection unit 124 and the estimation unit 126. This setting information includes the position, size, shape, etc. of the image area in the captured image. For example, assume a case where an image area corresponding to waste contained in a container C1 is set from a captured image corresponding to an imaging range IM1 (see FIG. 1). In this case, the setting unit 123 identifies the edge of the opening of the container C1 in the captured image based on information about the physical tag PT1 output from the tag information acquisition unit 122 (the position (base point), length, and direction (angle with respect to a specific direction) of the physical tag PT1 in the captured image) and the tag information output from the tag information acquisition unit 122 (tag information corresponding to the physical tag PT1).

[0063] Specifically, the setting unit 123 extracts image area information 154 corresponding to the tag information (tag information corresponding to physical tag PT1) output from the tag information acquisition unit 122 from the setting information DB 150 (see FIG. 7). Then, the setting unit 123 identifies the edge of the opening of the container C1 based on the extracted image area information and information related to the physical tag PT1 (the position, length, and angle described above). Next, the setting unit 123 identifies an image area including the identified edge of the opening of the container C1, and sets the image area in the captured image. This setting method will be described in detail with reference to FIGS. 11 to 14, etc.

[0064] The selection unit 125 selects a trained model stored in the trained model DB 160 based on the tag information acquired by the tag information acquisition unit 122, and outputs the selection result to the estimation unit 126. Specifically, the selection unit 125 selects trained model group identification information 153 corresponding to the tag information 151 acquired by the tag information acquisition unit 122 from the setting information DB 150 (see FIG. 7 ). Next, the selection unit 125 outputs the selection result to the estimation unit 126. That is, the selection unit 125 outputs the selected trained model group identification information to the estimation unit 126.

[0065] Here, the trained model is an AI (Artificial Intelligence) model that has been machine-learned using teacher data or training data. Examples of the trained model that can be used include SVM (Support Vector Machine), CNN (Convolutional Neural Network), ViT (Vision Transformer), and YOLO (You Only Look Once).

[0066] In addition, the learned model shown in this embodiment is learned using an image of waste contained in a container, and it is possible to detect the weight of the waste contained in the container. For example, it is possible to generate a learned model by learning a large amount of teacher data in which the relationship between an example and the corresponding correct answer is associated. Further, when new input data is input to the learned model, it is possible to output output data that is the correct answer based on the learning result of the example and the corresponding correct answer.

[0067] For example, when learning the weight of waste, a learned model can be generated by learning a large amount of teacher data in which the relationship between an image including the target waste (waste contained in a container) and the weight of the waste in this container is associated. For example, the weight when there is no waste in the container is set to 0, the weight when the waste in the container is a predetermined value is set to 1, and the weight of the waste in containers other than these is learned as a value between 0 and 1. In this case, a value between 0 and 1 is output as the output result of the learned model. That is, when there is no waste in the container, a weight of 0 is output, and when the waste in the container is a predetermined value, a weight of 1 is output. Also, when there is waste in the container and it is not a predetermined value, a numerical value t (0 < t < 1) corresponding to the weight is output.

[0068] In this embodiment, an example is shown in which the weight of waste is estimated using output values (output values of the capacity, shape, and density of the target waste, respectively) from a combination of three learned models (learned model (capacity), learned model (shape), learned model (density)) according to the attributes of the target waste. That is, it is possible to estimate the weight of waste based on the three output values. Note that these combinations of learned models are generated for each attribute of the waste.

[0069] The trained model (volume), for example, is a model that learns volume as a state of waste. Specifically, it is possible to generate the trained model (volume) by training a large amount of training data in which images including the target waste (waste contained in a container) are associated with the relationship between the weight and volume of the waste in the container. The trained model (shape), for example, is a model that learns shape (volume) as a state of waste. Specifically, it is possible to generate the trained model (shape) by training a large amount of training data in which images including the target waste (waste contained in a container) are associated with the relationship between the weight and shape of the waste in the container. The trained model (density), for example, is a model that learns density as a state of waste. Specifically, it is possible to generate the trained model (density) by training a large amount of training data in which images including the target waste (waste contained in a container) are associated with the relationship between the weight and density of the waste in the container.

[0070] When generating a trained model, a rectangular image area including one container (including the waste) containing the target waste can be cut out from a captured image of each waste contained in a plurality of containers, and the cut-out image can be used. For example, as shown in FIG. 1, if waste is contained in five containers, five images can be cut out for each container. Also, for example, as shown in FIG. 1, if five types of waste (PET, glass scraps, plastic bags, hard plastics, and iron scraps) are contained in five containers, trained models can be generated for the five types of waste. When estimating the weight of waste, a trained model corresponding to the type of waste to be estimated can be selected, and the weight of the waste to be estimated can be output using the selected trained model. This selection example will be described in detail with reference to FIG. 15.

[0071] The detection unit 124 detects a specific object from an image area (an area set in the captured image) specified based on the setting information output from the setting unit 123, and outputs the detection result to the estimation unit 126. Here, the specific object is, for example, non-standard waste, waste with a changed density, etc. For example, assume that PET is the waste. In this case, it is possible to set PET bottles as standard waste, and rolled or plate-shaped PET as non-standard waste. Also, assume that uncompressed PET bottles are contained in a container. In this case, if the PET bottles are compressed by some means (for example, manually), it is possible to set the compressed PET bottles as waste with a changed density. Note that a known image recognition technology (for example, pattern matching processing) can be used as a method for detecting the specific object.

[0072] The estimation unit 126 estimates the weight of waste contained in a container to which a physical tag corresponding to the tag information acquired by the tag information acquisition unit 122 is attached, and stores the estimation result in the estimation result DB 170 of the storage unit 130. Specifically, the estimation unit 126 estimates the weight of waste contained in a container included in the captured image output from the image acquisition unit 121 using a trained model group selected by the selection unit 125. For example, the estimation unit 126 can identify an image area set in the captured image based on the setting information output from the setting unit 123. Therefore, the estimation unit 126 inputs the captured image in which the image area is set by the setting unit 123 into the trained model group selected by the selection unit 125, and estimates the output result from the trained model group as the weight of waste contained in that image area. This waste estimation method will be described in detail with reference to FIG. 15 etc.

[0073] The output control unit 127 executes control to output various types of information from the user terminal 300. For example, in response to a request from the user terminal 300, the output control unit 127 provides the estimation results stored in the estimation result DB 170 of the storage unit 130 to the user terminal 300 and causes the results to be displayed on the display unit 306. An example of this display is shown in FIG.

[0074] The vehicle allocation processing unit 128 executes a process of arranging a transport vehicle for transporting the waste when the weight of the waste estimated by the estimation unit 126 reaches a threshold value. The vehicle allocation process will be described in detail with reference to FIG. 20.

[0075] The correction unit 129 corrects the criteria, calculation formula, etc. used in the estimation process by the estimation unit 126 based on the comparison result (difference value) between the weight of waste estimated by the estimation unit 126 and the weight of waste that was actually measured. Specifically, if the difference value is equal to or greater than a threshold value, the correction unit 129 corrects the criteria, calculation formula, etc. This correction process will be described in detail with reference to FIG. 21.

[0076] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (e.g., a control program, a tag DB 140, a setting information DB 150 (see FIG. 7), a trained model DB 160 (see FIG. 8), and an estimation result DB 170) that are required for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 110. The storage unit 130 can be, for example, a read-only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0077] [Configuration example of imaging device] The imaging device 200 includes a communication unit 201, a control unit 202, a storage unit 203, and an imaging unit 204. As the imaging device 200, for example, a remote camera that can be remotely controlled can be used.

[0078] The communication unit 201 exchanges various types of information with the information processing device 100 using wireless communication under the control of the control unit 202 .

[0079] The control unit 202 controls each unit of the imaging device 200 based on a control program stored in the storage unit 203. The control unit 202 is realized by a processing device such as a CPU or a GPU. For example, the control unit 202 executes control to associate the captured image generated by the imaging unit 204 with imaging device identification information and transmit the image to the information processing device 100. Based on the imaging device identification information, the information processing device 100 can identify that the captured image was generated at the waste collection site WP1.

[0080] The storage unit 203 is a storage medium that stores various types of information. For example, the storage unit 203 stores various types of information (for example, a control program, imaging device identification information for identifying the imaging device 200) that is required for the control unit 202 to perform various processes. The storage unit 203 also stores various types of information acquired via the communication unit 201. The storage unit 203 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0081] The imaging unit 204 captures an image of a subject under the control of the control unit 202 to generate an image (image data), and outputs the generated image to the control unit 202. The imaging unit 204 is configured, for example, with an imaging element (image sensor) that receives light from the subject collected by lenses (e.g., multiple lenses that collect light from the subject), and an image processing unit that performs predetermined image processing on the image data generated by the imaging element. As the imaging element, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type imaging element can be used. For example, the imaging unit 204 captures an image of a subject included in an imaging range IM1 (see FIG. 1 ) to generate a captured image, and outputs the captured image to the control unit 202. The imaging unit 204 may also generate still images periodically or irregularly, or may generate moving images.

[0082] [Example of user terminal configuration] The user terminal 300 includes a communication unit 301 , a control unit 302 , a storage unit 303 , an operation unit 304 , a sound output unit 305 , and a display unit 306 .

[0083] The communication unit 301 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 202.

[0084] The control unit 302 controls each unit based on various programs stored in the storage unit 303. The control unit 302 is realized by a processing device such as a CPU or a GPU. For example, the control unit 302 executes control to display weight information relating to the weight of an object provided from the information processing device 100 on the display unit 306.

[0085] The storage unit 303 is a storage medium that stores various types of information. For example, the storage unit 303 stores various types of information (for example, control programs and applications) required for the control unit 202 to perform various processes. The storage unit 303 also stores various types of information acquired via the communication unit 301. The storage unit 303 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0086] The operation unit 304 receives various operations from the user U1 and outputs the received operation details to the control unit 302. For example, when a user operation is received to cause the display unit 306 to display weight information relating to the weight of waste contained in each of the multiple containers C1 to C5 installed at the waste collection site WP1 (see FIG. 1), the operation unit 304 outputs operation information relating to the user operation to the control unit 302.

[0087] The sound output unit 305 outputs various sounds based on the control of the control unit 302. As the sound output unit 305, for example, one or more speakers can be used.

[0088] The display unit 306 is a display unit that displays various images based on the control of the control unit 302. For example, a display panel such as an organic EL (Electro Luminescence) panel or an LCD (Liquid Crystal Display) panel can be used as the display unit 306. An example of the display on the display unit 306 is shown in FIG. 19.

[0089] [Example of settings information DB content] 7 is a diagram schematically showing each piece of information stored in the setting information DB 150. The setting information DB 150 is a database for managing each piece of information used to estimate the weight of waste contained in the containers C1 to C5.

[0090] The setting information DB 150 stores tag information 151, attribute information 152, trained model group identification information 153, and image region information 154 in association with each other. Note that each of these pieces of information is an example, and other information may also be stored in the setting information DB 150.

[0091] The tag information 151 is tag information acquired based on the physical tags PT1 to PT5 attached to the containers C1 to C5. In Fig. 7, an example is shown in which the tag information corresponding to the physical tag PT1 attached to the container C1 is "TG001," the tag information corresponding to the physical tag PT2 attached to the container C2 is "TG002," and the tag information corresponding to the physical tag PT5 attached to the container C5 is "TG05." Note that other tag information is not shown in the figure.

[0092] The attribute information 152 is attribute information that indicates the attributes of the object (waste) that uses the trained model group whose identification information is stored in the trained model group identification information 153.

[0093] The trained model group identification information 153 is identification information that indicates the trained model group to be used for the image area corresponding to the tag information stored in the tag information 151 when that tag information is acquired. Note that the same trained model group is often used for waste with the same attributes. Therefore, the trained model group identification information 153 may also be the same for waste with the same attribute information 152.

[0094] The image area information 154 is information used when tag information stored in the tag information 151 is acquired and an image area corresponding to the tag information is set. This image area information will be described in detail with reference to FIGS. 9A, 9B, 11 to 14, etc. The image area to be set is sometimes called a bounding box. This bounding box refers to an image area that surrounds an object included in a captured image. When a wireless tag is used as a physical tag, each setting information such as the attribute information 152, the trained model group identification information 153, and the image area information 154 may be stored in the wireless tag. In this case, the information processing device 100 can acquire and use each setting information stored in the wireless tag.

[0095] [Example of contents of trained model DB] 8 is a diagram schematically illustrating each piece of information stored in the trained model DB 160. The trained model DB 160 is a database for managing a plurality of trained models used to estimate the weight of waste contained in the containers C1 to C5.

[0096] Trained model DB 160 stores trained model group identification information 161, trained model group 162, and attribute information 163 in association with each other. Note that each of these pieces of information is an example, and other information may also be stored in trained model DB 160.

[0097] Trained model group identification information 161 is identification information for identifying each trained model group stored in trained model group 162. Note that trained model group identification information 161 corresponds to trained model group identification information 153 (see FIG. 7).

[0098] The trained model group 162 is a combination of multiple trained models used to estimate the weight of waste contained in containers C1 to C5. In this embodiment, an example is shown in which multiple trained models corresponding to the attributes of the waste are selected and used from among the multiple trained models. Specifically, three trained models corresponding to the volume, shape, and density according to the attributes of the waste are selected and used. An example of this selection is shown in FIG. 15. Note that, although FIG. 8 shows an example in which trained models are stored in the trained model DB 160 and used, trained models stored in an external device may also be used.

[0099] The attribute information 163 is attribute information indicating the attributes of the object (waste) that uses the trained model group stored in the trained model group 162. The attribute information 163 corresponds to the attribute information 152 (see FIG. 7).

[0100] [Image area information] 9A and 9B are diagrams schematically showing image region information stored in image region information 154 (see FIG. 7). 9A and 9B show an example in which a figure for a bounding box is set in advance, and the bounding box is set based on this figure and information about the physical tag (for example, the position, length, and angle of the physical tag in the captured image).

[0101] 9A shows an example of constructing image area information based on the relationship between the shape of the edge SE1 of the opening of container C1 when viewed from above and the physical tag PT1 attached to the edge SE1. The same applies to the relationship between the other containers C2 to C4, whose edges SE1 when viewed from above are rectangular, and the physical tags PT2 to PT4.

[0102] 9B shows an example of image area information based on the shape of the edge SE5 of the opening of container C5 when viewed from above and the relationship with the physical tag PT5 attached to the edge SE5 of the opening of container C5. The same applies to the relationship between other containers and physical tags whose edge SE1 when viewed from above is circular.

[0103] 9A, the shape of the edge of the opening of the container C1 to which the physical tag PT1 is attached (the shape of the edge when viewed from above) can be identified based on the physical tag PT1. Here, when using a captured image (a captured image from directly above the container C1) generated by the imaging device 200 provided directly above the container C1, it is assumed that the shape of the edge of the opening of the container C1 included in the captured image matches the shape of the edge SE1 associated with the physical tag PT1. Therefore, the shape of the edge of the opening of the container C1 to which the physical tag PT1 is attached can be identified based on the physical tag PT1.

[0104] However, when using a captured image generated by an imaging device 200 disposed diagonally above the container C1 (a captured image captured from diagonally above the container C1), it is assumed that the shape of the edge of the opening of the container C1 included in the captured image does not match the shape of the edge SE1 associated with the physical tag PT1. Therefore, in this case, the shape of the edge SE1 associated with the physical tag PT1 can be transformed based on the state of the physical tag PT1 included in the captured image (e.g., the position, length, and angle of the physical tag in the captured image), and the transformed shape can be identified as the shape of the edge of the opening of the container C1. For example, an affine transformation can be used as this transformation. Examples of this are shown in FIGS. 10A to 10D.

[0105] Affine Transformation 10A to 10D are diagrams showing an example of transition when a square figure S1 is subjected to affine transformation. In Fig. 10A to 10D, figures S2 to S5 after affine transformation are shown by thick lines.

[0106] Here, when affine transforming each coordinate (x, y) in two-dimensional space to coordinate (x', y'), the following formula is used: Therefore, it is possible to affine transform each coordinate (x, y) in a plane image, which is two-dimensional space, to coordinate (x', y') using the following formula.

[0107]

number

[0108] 10A shows an example of a transition when translating a figure S1 into a square figure S2. When translating the figure S1 in this way, the following affine matrix is ​​used. Note that tx and ty are parameters that specify the distance when translating.

[0109]

number

[0110] Figure 10B shows an example of a transition when scaling the figure S1 to convert it into a square figure S3. When scaling the figure S1 in this way, the following affine matrix is ​​used. Note that sx and sy are parameters that specify the scaling ratio when scaling.

[0111]

number

[0112] 10C shows an example of a transition when rotating the figure S1 to convert it into a square figure S4. When rotating the figure S1 in this way, the following affine matrix is ​​used. Note that θ is a parameter that specifies the angle when rotating.

[0113]

number

[0114] FIG. 10D shows an example of the transition when shearing (skewing) the figure S1 to transform it into a parallelogram figure S5. When shearing and transforming the figure S1 in this way, the following affine matrix is ​​used. Note that sh x , sh y is a parameter that specifies the distance at which shearing occurs.

[0115]

number

[0116] [Bounding box setting example] FIG. 11 is a diagram schematically showing an example of a setting method for setting a bounding box that includes the object (PET) contained in the container C1 included in the imaging range IM1 (see FIG. 1).

[0117] As shown in FIG. 11(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. Specifically, the tag information acquisition unit 122 extracts physical tags PT1 to PT5 included in the captured image using a known image recognition technique as described above. Next, the tag information acquisition unit 122 acquires tag information corresponding to each of the physical tags PT1 to PT5 extracted from the captured image based on the arrangement order of the colors constituting the physical tags PT1 to PT5 as described above. As described above, the tag information corresponding to each of the physical tags PT1 to PT5 can be acquired using the tag DB 140 (see FIG. 6) that indicates the relationship between physical tags and tag information.

[0118] Next, the setting unit 123 (see FIG. 6) extracts, from the setting information DB 150, image area information 154 (see FIG. 7) corresponding to the tag information (corresponding to the physical tag PT1) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 9A, the shape of a rectangular edge SE1 having the physical tag PT1 provided on one side is extracted.

[0119] 11(B), based on the physical tag PT1 extracted from the captured image (corresponding to the imaging range IM1), the setting unit 123 fits the extracted image region information (the shape of the rectangular edge SE1) to the shape of the edge of the opening of the container C1 included in the captured image. For example, the setting unit 123 identifies both ends E1' and E2' of the physical tag PT1 attached to the container C1 included in the imaging range IM1. Then, the setting unit 123 performs affine transformation on the rectangular edge SE1 so that both ends E1 and E2 of the rectangular edge SE1 coincide with both ends E1' and E2' of the physical tag PT1 included in the captured image. This affine transformation will be described in detail with reference to FIG. 12.

[0120] 11(C) shows an example in which the rectangular edge SE1 is affine transformed so that both ends E1 and SE2 of the shape of the rectangular edge SE1 coincide with both ends E1' and E2' (see FIG. 1) of the physical tag PT1 included in the captured image (corresponding to the imaging range IM1). That is, the rectangular edge SE1 is affine transformed so that the rectangular edge SE1 coincides with the edge of the opening of the container C1 included in the captured image.

[0121] 11(D), the setting unit 123 fits the affine-transformed rectangular edge SE1 to the edge of the opening of the container C1 included in the imaging range IM1 based on the position (base point) of the physical tag PT1 in the captured image. Then, the setting unit 123 sets an image area of ​​a predetermined size including the fitted rectangular edge SE1 in the captured image (corresponding to the imaging range IM1). This image area can be used as a bounding box BB1.

[0122] The size of the bounding box BB1 can be set appropriately depending on the size of the fitted rectangular edge SE1. For example, the smallest size that can contain the fitted rectangular edge SE1 can be selected from a plurality of preset rectangular sizes. Alternatively, for example, the size of a rectangle that contains the maximum size (e.g., horizontal length) of the fitted rectangular edge SE1 can be used as a reference. While FIG. 11 shows an example of setting a rectangular bounding box BB1, other shapes may also be set as the bounding box. For example, the bounding box may be set based on the position and size of the physical tag PT1. For example, a rectangular bounding box can be set with its center at the position of the physical tag PT1 and its size corresponding to the length of the physical tag PT1.

[0123] In this way, the bounding box BB1 can be set based on the position, size, shape (degree of deformation), etc. of the physical tag PT1. In this case, as described above, the image area can be set to any shape, figure, etc.

[0124] [Affine transformation example] FIG. 12 is a diagram showing an example of transformation when affine transformation is performed on image region information (the shape of the rectangular edge SE1) so that it matches the edge of the opening of the container C1.

[0125] For example, assume that the position of container C1 and the position and orientation (imaging direction) of imaging device 200 are fixed at waste collection site WP1. In this case, the distance from imaging device 200 to container C1 can be represented by r, and the position of container C1 relative to imaging device 200 can be represented by depression angle θ and azimuth angle φ. Furthermore, the spherical coordinates (r, θ, φ) can be converted to Cartesian linear coordinates (x, y, z) using the following equations.

[0126]

number

[0127] Furthermore, the opening surface of container C1 (the surface including the edge of the opening of container C1, the surface including physical tag PT1) is assumed to be on plane P (a virtual plane parallel to the floor surface). The height from the floor surface at the installation location of container C1 to the edge of the opening of container C1 is assumed to be h. Furthermore, it is assumed that the setting unit 123 is capable of grasping in advance the relationship between the three-dimensional space (three-dimensional coordinates) at waste collection site WP1 and the two-dimensional coordinates corresponding to the imaging range IM1.

[0128] In this case, the setting unit 123 identifies both ends E1', E2' of the physical tag PT1 acquired by the tag information acquisition unit 122, and calculates the length D1 of both ends E1', E2'. Next, the setting unit 123 calculates the length D2 (the length of ends E1', E3') of one side of the container C1 on which the physical tag PT1 is provided, based on the ratio (T1 / T2) of the length T1 (see FIG. 9A) of the physical tag PT1 in the image region information (the shape of the rectangular edge SE1) to the length T2 (see FIG. 9A) of one side on which the physical tag PT1 is provided. That is, the length D2 (=D1·T2 / T1) of the ends E1' and E3' on the plane P is calculated. This allows the setting unit 123 to identify the end E3' of the physical tag PT1 in three-dimensional space.

[0129] Next, the setting unit 123 changes the size of the image region information (the shape of the rectangular edge SE1) (see FIG. 9A) so that the length D2 of the ends E1' and E3' on the plane P matches the length T2 of the ends E1 and E3 of the image region information (the shape of the rectangular edge SE1). For example, the size of the image region information (the shape of the rectangular edge SE1) can be changed by an affine transformation (a transformation that enlarges or reduces a figure) shown in FIG. 10B. Then, the setting unit 123 draws the image region information (the shape of the rectangular edge SE1) after the size conversion on the plane P. This allows the setting unit 123 to identify both ends E4' and E5' of the physical tag PT1 in three-dimensional space.

[0130] The setting unit 123 can also identify the ends E1', E3', E4', and E5' of the physical tag PT1 included in the captured image corresponding to the imaging range IM1 based on the relationship between the three-dimensional space (three-dimensional coordinates) at the waste collection site WP1 and the two-dimensional coordinates corresponding to the imaging range IM1. Therefore, the setting unit 123 can calculate the crushed (sheared) area of ​​the opening of the container C1 in the captured image based on the relationship between the line segment between the ends E1' and E3' and the line segment between the ends E4' and E5' of the physical tag PT1 included in the captured image. That is, the shearing amount of the rectangle corresponding to the opening of the container C1 in the captured image can be calculated. The scaling size can be calculated based on the relationship between the length D1 of the line segment between the ends E1' and E2' of the physical tag PT1 and the length T1 of the line segment between the ends E1 and E2 of the image area information (the shape of the rectangular edge SE1). The rotation angle can be calculated based on the angle of the line segment between the ends E1' and E2' of the physical tag PT1 relative to a specific direction (e.g., the horizontal direction in the captured image).

[0131] Using the shear amount, scaling size, and rotation angle thus determined, it is possible to perform affine transformation so that the image region information (the shape of the rectangular edge SE1) coincides with the edge of the opening of the container C1, as shown in Figure 11(C). Furthermore, the image region information (the shape of the rectangular edge SE1) thus affine transformed is placed in the captured image so that the reference points (e.g., end E1' of the physical tag PT1 and end E1 of the physical tag PT1) coincide, as shown in Figure 11(D). This allows the bounding box BB1 to be set.

[0132] In this way, if the position and imaging direction of the imaging device 200 are fixed and the size of the container C1 can be known in advance, the shear amount can be calculated based on the depression angle and azimuth angle.

[0133] The above example shows how affine transformation is performed to fit image area information (rectangular or circular edge shape) to each container based on the relationship between the three-dimensional space (three-dimensional coordinates) at waste collection point WP1 and the two-dimensional coordinates corresponding to imaging range IM1. It is also possible to measure container C1 included in imaging range IM1 in advance and calculate the shear amount of the opening (rectangular) edge of container C1 in advance based on the measurement results. In this case, the calculation results can be used to perform affine transformation.

[0134] Furthermore, for example, affine transformation can be performed based on the length (size) D1 and angle (e.g., angle relative to the horizontal direction) of both ends E1', E2' of the physical tag PT1. That is, affine transformation can be performed so that the length and angle of both ends E1, E2 of the rectangular edge SE1 coincide with the length and angle of both ends E1', E2' of the physical tag PT1. For example, using the difference value (e.g., angle, length) between end E2 and end E2' when end E1 and end E1' are coincident, an affine transformation can be performed by setting an affine matrix so that the difference value between end E2 and end E2' becomes 0. For each of these affine transformations, a known transformation method can be used.

[0135] The edge of the opening of the container C1 can be detected using a known image recognition technology (e.g., edge detection technology). Therefore, the angle between one of the four edges of the container C1, on which the physical tag PT1 is provided, and another adjacent edge can be obtained. The angle between these two adjacent edges can be used to estimate the shape (degree of deformation) of the rectangle corresponding to the edge of the opening of the container C1. For example, when using a captured image (a captured image from directly above the container C1) generated by the imaging device 200 provided directly above the container C1, the rectangle corresponding to the edge of the opening of the container C1 included in the captured image is rectangular. Therefore, a 90-degree angle can be obtained as the angle between the one side on which the physical tag PT1 is provided and the other adjacent side. On the other hand, when using a captured image (a captured image from diagonally above the container C1) generated by the imaging device 200 provided diagonally above the container C1, the rectangle corresponding to the edge of the opening of the container C1 included in the captured image has a sheared rectangular shape. In this case, a value less than 90 degrees is acquired as the angle between one side on which the physical tag PT1 is provided and another side adjacent thereto.

[0136] By providing physical tags on two or more of the four sides of the container C1, it is possible to calculate the shear amount by calculating the angles of the two or more physical tags without detecting the edge of the opening of the container C1. Examples of providing physical tags on two or more of the four sides of the container C1 are shown in Figures 13, 22A and 22B, 23A and 23B.

[0137] Furthermore, if the positions and orientations of the containers C1 to C5 and the position and orientation (imaging direction) of the imaging device 200 are fixed at the waste collection site WP1, the shapes of the edges of the openings of the containers C1 to C5 included in the imaging range IM1 can be acquired in advance by measurement or the like. Therefore, in such a case, the shapes of the edges of the openings of the containers C1 to C5 included in the imaging range IM1 (deformed rectangular shapes) can be measured and acquired in advance and stored in the image region information 154 (see FIG. 7) corresponding to the tag information 151. For example, the graphic information shown in FIG. 11(C) can be stored in the image region information 154 as the image region information corresponding to the container C1. This makes it possible to omit the above-mentioned processes such as affine transformation, and the graphic information stored in the image region information 154 can be used as is.

[0138] The above example illustrates the affine transformation of the rectangular edge SE1 based on the length and angle of both ends E1', E2' of the physical tag PT1. However, the affine transformation may also be performed based on other information. For example, the rectangular edge SE1 after the affine transformation may be compared with the edge of the opening of the container C1, and whether or not to use the affine-transformed rectangular edge SE1 may be determined based on the degree of similarity. For example, if the degree of similarity is equal to or greater than a threshold, the affine-transformed rectangular edge SE1 is used. If the degree of similarity is less than the threshold, the affine-transformed rectangular edge SE1 is again affine-transformed. The edge of the opening of the container C1 can be detected using known image recognition technology (e.g., edge detection technology). Furthermore, known image recognition technology (e.g., a determination technology based on brightness difference values) can be used to compare images.

[0139] [Example of physical tags on three sides] 12 shows an example of affine transformation in which one physical tag PT1 is attached to one side of one container C1. However, one container may be attached with multiple physical tags. This makes it possible to simplify the calculation of the affine transformation.

[0140] 13 is a diagram showing an example in which physical tags PT1A, PT1B, and PT1C are provided on three sides of the edge of a container C1. In this case, image region information (the shape of the rectangular edge SE1A) corresponding to the physical tags PT1A, PT1B, and PT1C provided on the container C1 is stored in the image region information 154 (see FIG. 7).

[0141] 13 shows a perspective view of the container C1 on the left side, and image area information (the shape of the rectangular edge SE1A) on the right side. In this case, similar to the example shown in FIG. 11(B), the setting unit 123 fits the extracted image area information (the shape of the rectangular edge SE1A) to the shape of the edge of the opening of the container C1 included in the captured image based on the physical tags PT1A, PT1B, and PT1C extracted from the captured image (corresponding to the imaging range IM1). In the example shown in FIG. 13, the physical tags PT1A, PT1B, and PT1C are provided on three sides of the container C1, so the setting unit 123 identifies the ends, lengths, and angles (e.g., the angles between the physical tags) of the physical tags PT1A, PT1B, and PT1C provided on the container C1 included in the imaging range IM1. The setting unit 123 then performs affine transformation on the image region information (the shape of the rectangular edge SE1A) so that the physical tags PT1A, PT1B, and PT1C on three sides of the image region information (the shape of the rectangular edge SE1A) match the physical tags PT1A, PT1B, and PT1C included in the captured image. A known affine transformation method can be used for this affine transformation. As shown in FIG. 13, by providing physical tags PT1A, PT1B, and PT1C on three sides of the container C1, it is possible to calculate the angle by which the rectangle of the container C1 is deformed, which makes it easier to calculate the shear amount.

[0142] [Example of setting a bounding box for a cylindrical container] FIG. 14 is a diagram schematically showing an example of a setting method for setting a bounding box that includes the object (scrap iron) contained in the container C5 included in the imaging range IM1 (see FIG. 1).

[0143] 14(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. The method for acquiring this tag information is the same as the example shown in FIG.

[0144] Next, the setting unit 123 (see FIG. 6) extracts, from the setting information DB 150, image area information 154 (see FIG. 7) corresponding to the tag information (corresponding to the physical tag PT5) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 9B, the shape of a circular edge SE5 on which the physical tag PT5 is provided is extracted.

[0145] 14(B), based on the physical tag PT5 extracted from the captured image (corresponding to the imaging range IM1), the setting unit 123 fits the extracted image region information (the shape of the rectangular edge SE5) to the shape of the edge of the opening of the container C5 included in the captured image. For example, the setting unit 123 identifies both end portions E5', E6' and a central portion M5' of the physical tag PT5 attached to the container C5 included in the imaging range IM1. Then, the setting unit 123 performs affine transformation on the circumferential edge SE5 so that both end portions E5, E6, and the central portion M5 of the circumferential edge SE5 coincide with both end portions E5', E6', and the central portion M5' of the physical tag PT5 included in the captured image. A known transformation method can be used for this affine transformation.

[0146] For example, it is possible to perform affine transformation based on the circumferential shape specified by the lengths (sizes) of both ends E5', E6' of the physical tag PT5 and the central portion M5'. That is, it is possible to perform affine transformation so that the circumferential shape specified by both ends E5, E6 and the central portion M5 of the circumferential edge SE5 matches the circumferential shape specified by both ends E5', E6' and the central portion M5' of the physical tag PT5.

[0147] 14(C), the circumferential edge SE5 is affine transformed so that both ends E5, E6 and a central portion M5 of the rectangular shape of the edge SE5 coincide with both ends E5', E6' and a central portion M5' of the physical tag PT5 included in the captured image (corresponding to the imaging range IM1). That is, the circumferential edge SE5 is affine transformed so that the circumferential edge SE5 coincides with the edge of the opening of the container C5 included in the captured image.

[0148] 14(D), the setting unit 123 fits the affine-transformed circumferential edge SE5 to the edge of the opening of the container C5 included in the imaging range IM1 based on the position (base point) of the physical tag PT5 in the captured image. Then, the setting unit 123 sets an image area of ​​a predetermined size including the fitted circumferential edge SE5 in the captured image (corresponding to the imaging range IM1). This image area can be used as a bounding box BB5.

[0149] 11, the circumferential edge SE5 after the affine transformation may be compared with the edge of the opening of the container C5, and whether or not to use the circumferential edge SE5 after the affine transformation may be determined based on the degree of similarity. Bounding boxes can also be set for edges of other shapes in a similar manner.

[0150] [Example of selecting a trained model] Figure 15 is a diagram schematically showing an example of a selection method for selecting a trained model to be used when estimating the weight of waste (PET) contained in a container C1 included in the imaging range IM1 (see Figure 1).

[0151] 15(A), the tag information acquisition unit 122 (see FIG. 6) acquires tag information included in the captured image (corresponding to the imaging range IM1) acquired by the image acquisition unit 121. The method for identifying this tag information is similar to the examples shown in FIGS. 11 to 14, etc.

[0152] Next, the selection unit 125 (see FIG. 6) extracts, from the setting information DB 150, the trained model group identification information 153 (see FIG. 7) corresponding to the tag information (corresponding to the physical tag PT1) acquired by the tag information acquisition unit 122. For example, as shown in FIG. 15(B), the trained model group identification information "LL4" corresponding to the tag information (corresponding to the physical tag PT1) is selected. In this case, the fourth trained model group LL4 corresponding to the trained model group identification information "LL4" is selected from the trained model DB 160 (see FIG. 8) and used. In the example shown in FIG. 15(B), the combination of a trained model corresponding to the volume of the waste (trained model (capacity) LL4a), a trained model corresponding to the shape of the waste (trained model (shape) LL4b), and a trained model corresponding to the density of the waste (trained model (density) LL4c) is used as the fourth trained model group LL4. How the fourth trained model group LL4 is used will be described in detail with reference to FIGS. 16 to 18.

[0153] Next, the estimation unit 126 uses the trained model group selected by the selection unit 125 to estimate the weight of waste contained in a container included in the captured image output from the image acquisition unit 121. For example, as shown in FIG. 15(C), the estimation unit 126 inputs the captured image in which the image area (bounding box BB1) is set by the setting unit 123 to the fourth trained model group LL4 selected by the selection unit 125, and acquires the output result from the fourth trained model group LL4. Next, the estimation unit 126 estimates the weight of the waste (PET) included in the image area (bounding box BB1) based on the output result. Then, the estimation unit 126 stores the estimation result in the estimation result DB 170 of the storage unit 130.

[0154] [Example of transition of waste state] FIG. 16 is a diagram showing the relationship between the transition of the waste (PET) contained in the container C1 and the transition of the estimation result (predicted weight) estimated by the estimation unit 126.

[0155] The upper part of Fig. 16 shows an example of the transition of a container C1 (waste) included in each captured image generated in time series by the imaging device 200. The lower part of Fig. 16 shows a line SL1 indicating the estimation result (predicted weight) estimated by the estimation unit 126. In the graph at the bottom of Fig. 16, the vertical axis indicates the estimation result (predicted weight) estimated by the estimation unit 126, and the horizontal axis indicates the time axis.

[0156] As shown in FIG. 16, the amount of waste (PET) stored in the container C1 often increases over time.

[0157] [Example of transition of waste status when irregular objects are detected] Fig. 17 is a diagram showing the relationship between the transition of waste (PET) when an atypical object is detected in container C1 and the transition of the estimation result (predicted weight). Note that the example shown in Fig. 17 is the same as the example shown in Fig. 16 except that an atypical object is detected by the detection unit 124 at time t1. For this reason, parts common to Fig. 16 are given the same reference numerals and their description will be omitted.

[0158] 16, when regular waste (e.g., crushed PET bottles) is stored in the container C1, the amount of waste (PET) stored in the container C1 often increases over time. However, when non-regular waste (e.g., roll-shaped or plate-shaped PET) is stored in the container C1, the appearance of the waste (PET) stored in the container C1 is expected to vary greatly depending on the shape of the non-regular waste.

[0159] For example, as shown in FIG. 17, when a plate-shaped PET is placed in container C1, the appearance of the multiple PETs placed in container C1 changes significantly due to the large size of the plate-shaped PET, resulting in a sudden increase in volume. Therefore, if the weight of the waste (PET) is estimated based solely on its volume, the weight of the waste increases rapidly in response to the large change in volume at time t1, as shown by line SL2. In this case, the weight of the waste (PET) may not be estimated appropriately. Therefore, in this embodiment, when an irregular object is detected in container C1, the weight of the waste (PET) placed in container C1 is estimated using a trained model corresponding to the shape of the waste (trained model (shape) LL4b). An example of this estimation will be described later.

[0160] [Example of transition of waste state when a change in density of a standard object is detected] Fig. 18 is a diagram showing the relationship between the transition of waste (PET) when a change in density of a regular object is detected in container C1 and the transition of the estimation result (predicted weight). Note that the example shown in Fig. 18 is the same as the example shown in Fig. 16 except that a change in density of a regular object is detected by the detection unit 124 at time t2. Therefore, parts common to Fig. 16 are given the same reference numerals and their description will be omitted.

[0161] 16, when standard waste (e.g., crushed PET bottles) is stored in container C1, the amount of waste (PET) stored in container C1 often increases over time. However, when standard waste is compressed (e.g., crushed), it is expected that the appearance of the waste (PET) stored in container C1 will vary greatly depending on the density of the standard waste after compression.

[0162] For example, as shown in FIG. 18, when standard-sized waste is compressed, the external appearance of the standard-sized waste becomes smaller due to the compression, resulting in a rapid decrease in volume. Therefore, if the weight of the waste (PET) is estimated based solely on the volume of the waste, as shown by line SL3, the weight of the waste will rapidly decrease in response to a large change in volume at time t2. In this case, there is a risk that the weight of the waste (PET) cannot be estimated appropriately. Therefore, in this embodiment, when a change in the density of standard-sized waste is detected in container C1, the weight of the waste (PET) contained in container C1 is estimated using a trained model (trained model (density) LL4c) corresponding to the shape of the waste. An example of this estimation will be described later.

[0163] [Example of using a trained model (capacity)] For example, it is possible to estimate the weight of waste using the output result (filling rate v) of the trained model (capacity) and the following equation 1. w v =f(v)=k1·v …Equation 1

[0164] Here, the filling rate v is output as a value between 0 and 1, where 1 is the value when the container for the target waste is full. Also, k1 is a coefficient set based on the weight at a filling rate of 1 (100%).

[0165] For example, suppose the weight of PET garbage is to be estimated as waste. In this case, the ratio of PET bottles of various sizes and the filling rate are taken into consideration, and it is assumed that the weight when full is 60 kg. In this case, it is possible to set k1 = 60. The coefficient k1 can be set appropriately based on experiments, simulations, etc.

[0166] [Example using a trained model (shape)] For example, it is possible to estimate the weight of irregular waste using the output result (volume s) of the trained model (shape) and the following equation 2. w s =f(s)=k2·s …Equation 2

[0167] Here, k2 is a coefficient set based on specific gravity.

[0168] For example, assume that the weight of PET waste is to be estimated. In this case, as described above, the detection unit 124 can detect PET waste (irregular waste) other than standard PET bottles (uncompressed and compressed shapes) from among the objects included in the captured image. For example, roll-shaped PET, plate-shaped PET, etc. can be detected as irregular waste.

[0169] For example, in the case of a standard PET board (thickness 5 mm), the volume s when the length h (cm) × width I (cm) is 0.05 × h × I (cm 3 In this case, if k2 = 1.4 (specific gravity), the weight w s is 1.4×s(g).

[0170] In this way, when irregular waste is detected, it is possible to add the weight of the irregular waste estimated using the trained model (shape).

[0171] [Example using a trained model (density)] For example, it is possible to estimate the weight of waste using the output result (density d) of the trained model (density) and the following equation 3. w d =f(d)=k1·d·Δv …Equation 3

[0172] Here, the density d is the reciprocal of the compression rate. For example, if the compression rate is 30%, then d = 1 / 0.3.

[0173] For example, consider the case where the weight of PET waste is to be estimated. In this case, as described above, the detection unit 124 can detect changes in the density of standard PET bottles from among the objects included in the captured image. Therefore, the captured images generated in chronological order are checked one after another, and the volume Δv and density d of the waste at the point in time when the density of the waste changes are measured. Then, the measurement results can be subtracted to estimate the weight of the waste.

[0174] Taking these into consideration, the weight w of the waste can be calculated using the following equation 4. Note that the following equation 4 is an example in which captured images generated in time series are used, and k is a value indicating the time axis.

[0175]

number

[0176] The first term of Equation 4 represents the estimated weight of standard waste. The second term of Equation 4 represents the estimated weight of non-standard waste. The third term of Equation 4 represents the estimated weight of compressed standard waste. That is, when non-standard waste is detected from each object included in the captured image, the volume at the time of detection is subtracted from the first term. When a change in density of compressed or other waste is detected from each object included in the captured image, the compressed volume at the time of detection is subtracted from the first term. Then, the weight based on the subtracted volume of the non-standard waste is calculated in the second term. The weight based on the volume of compressed or other waste is calculated in the third term. Note that Equation 4 is an example shown for ease of explanation, and other equations determined based on experiments, simulations, etc. may be used.

[0177] In this way, the information processing device 100 can estimate the weight of the waste using multiple trained models according to the attributes of the target waste (a trained model corresponding to the volume of the waste, a trained model corresponding to the shape of the waste, and a trained model corresponding to the density of the waste). This makes it possible to improve the accuracy of estimating the weight of the waste.

[0178] In the above, an example has been shown in which the weight of waste is estimated using three types of trained models (volume, shape, and density). However, this embodiment is not limited to this. For example, the weight of waste may be estimated using two types of trained models (volume and shape, or volume and density), or the weight of waste may be estimated using four or more types of trained models. The trained models used in this case can be set appropriately based on experiments, simulations, etc. Furthermore, the judgment criteria, calculation formulas, etc. used in these cases can also be set appropriately based on experiments, simulations, etc.

[0179] [Example of display on user terminal] Fig. 19 is a diagram showing an example of a display screen 310 displayed on the display unit 306 of the user terminal 300. Fig. 19 shows an example of displaying the estimation results when the weight of waste contained in the containers C1 to C5 installed at the waste collection site WP1 (see Fig. 1) is estimated.

[0180] A container information display area 311, a waste information display area 312, and a weight display area 313 are displayed in association with each other on the display screen 310. Each of these pieces of information can be displayed based on each piece of estimation result information stored in the estimation result DB 170 of the storage unit 130. Note that Fig. 19 shows an example in which only some (containers C1 to C4) of the containers C1 to C5 installed at the waste collection site WP1 are displayed. The other containers can be displayed using a scroll bar 314.

[0181] Container information for identifying each of the containers C1 to C5 installed at the waste collection point WP1 is displayed in the container information display area 311. Fig. 19 shows an example in which the container information corresponding to container C1 is displayed as "first container," the container information corresponding to container C2 is displayed as "second container," the container information corresponding to container C3 is displayed as "third container," and the container information corresponding to container C4 is displayed as "fourth container."

[0182] The waste information display area 312 displays the names of the waste materials to identify the waste materials contained in each of the containers C1 to C5 installed at the waste material collection site WP1.

[0183] The weight display area 313 displays the weight of each waste estimated by the estimation unit 126. When images generated by the imaging device 200 are acquired in real time and the weight of the waste is estimated sequentially, the display contents of the weight display area 313 can be changed each time the estimation process is performed.

[0184] Furthermore, each piece of information that can be displayed on the display screen 310 can be output from the sound output unit 305. For example, each piece of information for each object (for example, audio information SS1) can be output from the sound output unit 305.

[0185] [Example of operation of information processing device] Fig. 20 is a flowchart showing an example of estimation processing in the information processing device 100. This estimation processing is executed by the control unit 120 (see Fig. 6) based on a program stored in the storage unit 130 (see Fig. 6). Fig. 20 also shows an example of estimating the weight of each waste contained in containers C1 to C5 installed at the waste collection site WP1 (see Fig. 1). This estimation processing will be explained with appropriate reference to Figs. 1 to 19.

[0186] In step S501 , the image acquisition unit 121 acquires a captured image generated by the imaging device 200 .

[0187] In step S502, the tag information acquisition unit 122 reads the physical tags PT1 to PT5 attached to the containers C1 to C5, respectively, based on the captured images acquired in step S501. A known image recognition technique can be used to read these physical tags.

[0188] In step S503, the tag information acquisition unit 122 acquires tag information of the objects (waste) contained in each of the containers C1 to C5 based on each of the physical tags PT1 to PT5 read in step S502.

[0189] In step S504, the detection unit 124 detects non-standard waste, changes in density of standard waste, etc. based on the captured image acquired in step S501. This detection method can use a known image recognition technique.

[0190] In step S505, setting unit 123 sets an image area including the object based on the tag information of the object acquired in step S503. The method for setting this image area can be the same as the examples shown in Figs. 11 to 14, etc.

[0191] In step S506, the selection unit 125 selects a group of trained models to be used when estimating the weight of the object based on the tag information of the object acquired in step S503. The method for selecting this group of trained models can be the same as the example shown in FIG.

[0192] In step S507, the estimation unit 126 determines whether or not an irregular waste object was detected in step S504. For example, it is determined whether or not an irregular waste object was detected in an image area including the waste object to be estimated. If an irregular waste object was detected, the process proceeds to step S508. On the other hand, if an irregular waste object was not detected, the process proceeds to step S509.

[0193] In step S508, the estimation unit 126 estimates the weight of the object whose tag information was acquired in step S503, using the trained model (shape) included in the trained model group selected in step S506. Specifically, the estimation unit 126 inputs the captured image whose image area was set in step S505 into the selected trained model (shape), and stores the calculation result obtained by calculating the response result using the above-mentioned Equation 2.

[0194] In step S509, the estimation unit 126 determines whether a change in density of standardized waste was detected in step S504. For example, it determines whether a change in density of standardized waste was detected in the image area including the waste to be estimated. If a change in density of standardized waste was detected, the process proceeds to step S510. On the other hand, if no non-standardized waste was detected, the process proceeds to step S511.

[0195] In step S510, the estimation unit 126 estimates the weight of the object whose tag information was acquired in step S503, using the trained model (density) included in the trained model group selected in step S506. Specifically, the estimation unit 126 inputs the captured image in which the image area was set in step S505 into the selected trained model (density), and stores the calculation result obtained by calculating the response result using the above-mentioned Equation 3.

[0196] In step S511, the estimation unit 126 estimates the weight of the object whose tag information was acquired in step S503, using a trained model (capacity) included in the trained model group selected in step S505. Specifically, the estimation unit 126 inputs the captured image in which the image area was set in step S505 into the selected trained model (capacity), and stores the calculation result obtained by calculating the response result using the above-described Equation 1.

[0197] In step S512, the estimation unit 126 estimates the weight of the object whose tag information was acquired in step S503, using the values ​​stored in steps S508, S510, and S511. Specifically, the estimation unit 126 calculates the values ​​stored in steps S508, S510, and S511 using the above-described formula 4, and estimates the calculation result as the weight of the object.

[0198] These processes may be executed sequentially for each object, or may be executed in parallel for each object.

[0199] The estimation unit 126 sequentially stores the estimation results estimated in step S512 in the estimation result DB 170 of the storage unit 130. Furthermore, the output control unit 127 provides the estimation results stored in the estimation result DB 170 of the storage unit 130 to the user terminal 300 in response to a request from the user terminal 300. Furthermore, the control unit 302 of the user terminal 300 causes the display unit 306 to display the estimation results provided from the information processing device 100. For example, as shown in FIG. 19 , it is possible to cause the display unit 306 to display a display screen 310. Furthermore, the control unit 302 of the user terminal 300 causes the sound output unit 305 to output audio information related to the estimation results provided from the information processing device 100. For example, as shown in FIG. 19 , it is possible to cause the sound output unit 305 to output audio information SS1.

[0200] In step S513, the vehicle allocation processing unit 128 determines whether or not there is waste whose weight estimated in step S512 has reached a threshold value. If there is waste whose estimated weight has reached the threshold value, the process proceeds to step S514. On the other hand, if there is no waste whose estimated weight has reached the threshold value, the estimation processing operation ends. Note that the threshold value shown here is a value that serves as a criterion for determining whether or not to arrange for the allocation of a vehicle for waste whose weight has been estimated, and can be set appropriately based on experiments, simulations, etc.

[0201] In step S514, the vehicle allocation processing unit 128 executes a vehicle allocation process to arrange for the allocation of a vehicle for the waste whose estimated weight has been determined to have reached the threshold value in step S513. For example, a process is executed to transmit request information to a company that provides a vehicle to transport the waste whose estimated weight has been determined to have reached the threshold value. In this case, the estimated weight may be included in the request information.

[0202] [Example of operation of information processing device] Fig. 21 is a flowchart showing an example of correction processing in the information processing device 100. This correction processing is executed by the control unit 120 (see Fig. 6) based on a program stored in the storage unit 130 (see Fig. 6). Fig. 21 also shows an example in which the weight of each waste contained in containers C1 to C5 installed at the waste collection site WP1 (see Fig. 1) is measured, and each piece of information is corrected based on the measurement results. This correction processing will be explained with appropriate reference to Figs. 1 to 20.

[0203] In step S521, the correction unit 129 acquires the measurement result of the weight of the object to be corrected. For example, when waste (PET) contained in the container C1 is loaded onto a transport vehicle, the weight of the waste (PET) can be measured using some kind of measuring device. In this case, the measuring device transmits the measurement result to the information processing device 100, and the correction unit 129 can acquire the measurement result of the weight of the waste (PET).

[0204] For example, as shown in Fig. 20, when the weight of waste estimated by the estimation unit 126 reaches a threshold, a vehicle dispatch process is executed to transport the waste. When the transport vehicle dispatched by this vehicle dispatch process performs collection work to collect the waste, there is often an opportunity to measure the weight of the waste. For example, it is expected that the weight will be measured using a forklift scale, truck scale, floor scale, etc. In this case, it is possible to acquire a physical tag attached to the container to be collected during the collection work.

[0205] For example, if the physical tag is a color code or the like, it can be read by the imaging device 200 (see FIG. 1), and if the physical tag is a wireless tag, it can be read by the communication device 400 (see FIG. 25). Furthermore, using the imaging device 200, it is possible to track the movement of a container (container to be collected) identified by the physical tag, and to identify the location where the container to be collected was placed. This makes it possible to recognize the attributes of the waste contained in the container to be collected. Furthermore, when the weight of the waste contained in the container to be collected is measured, it is possible to link the tag information of the waste with the measured weight. For example, it is possible to link with a forklift scale, a truck scale, a floor scale, etc. Furthermore, as will be described later, it becomes possible to learn the measured weight of the waste.

[0206] In step S522, the correction unit 129 compares the measurement result acquired in step S521 with the estimation result estimated in step S512 (see FIG. 20), and calculates the difference therebetween.

[0207] In step S523, the correction unit 129 determines whether the difference value calculated in step S522 is equal to or greater than a threshold value. If the difference value is equal to or greater than the threshold value, the process proceeds to step S524. On the other hand, if the difference value is less than the threshold value, the correction process ends. Note that the threshold value shown here is a value that serves as a criterion for determining whether to correct the judgment criteria, calculation formula, etc. related to the waste whose weight has been measured, and can be set appropriately based on experiments, simulations, etc.

[0208] In step S524, the modification unit 129 modifies the determination criterion, the calculation formula, and the like based on the difference value calculated in step S522. For example, a modification process is executed to modify the above-described formulas 1 to 4. Note that FIG. 21 shows an example in which the determination criterion, the calculation formula, and the like are modified when the difference value calculated in step S522 is equal to or greater than a threshold value, but this is not limiting. For example, even when the difference value calculated in step S522 is less than the threshold value, the determination criterion, the calculation formula, and the like may be modified as appropriate. This makes it possible to modify the determination criterion, the calculation formula, and the like used in the estimation process as appropriate based on the measurement results, thereby improving the estimation accuracy.

[0209] In this way, when the waste contained in the container is measured, it is possible to determine the difference between the actual weight measured and the weight estimated by the estimation unit 126. If the difference exceeds the allowable range, it is possible to appropriately correct the judgment criteria, calculation formula, etc., and improve the estimation accuracy.

[0210] [Modification of physical tags] In Figure 1 and other figures, one physical tag is provided for one container, and in Figure 13, three physical tags are provided for one container. However, two, four, or more physical tags may be provided for one container. This makes it possible to improve the accuracy of setting the bounding box.

[0211] 22A and 22B are diagrams showing an example in which physical tags PT1a and PT1b are provided on two of the edges of a container C1. FIG. 22A shows a perspective view of the container C1, and FIG. 22B shows the shape of the edge SE1a corresponding to the container C1, which is stored in the image region information 154 (see FIG. 7). In this manner, image region information corresponding to the physical tags PT1a and PT1b provided on the container C1 is stored in the image region information 154. The setting unit 123 can set a bounding box using the positions, lengths, angles, etc. of the physical tags PT1a and PT1b provided on the two edges of the container C1. In this manner, providing two physical tags PT1a and PT1b makes it possible to obtain the angles of the two sides of the container C1, thereby facilitating the calculation of the shear amount.

[0212] 23A and 23B are diagrams illustrating an example in which physical tags PT5a and PT5b are provided at two opposing positions on the edge of a container C5. FIG. 23A illustrates a perspective view of the container C5, and FIG. 23B illustrates the shape of the edge SE5a corresponding to the container C5, which is stored in the image region information 154 (see FIG. 7). In this manner, image region information corresponding to the physical tags PT5a and PT5b provided on the container C5 is stored in the image region information 154 (see FIG. 7). The setting unit 123 can also set a bounding box using the positions, lengths, angles, and the like of the physical tags PT5a and PT5b provided at two opposing positions on the container C5. In this manner, providing two physical tags PT5a and PT5b makes it possible to acquire the positional relationship between the two opposing positions on the container C5, thereby facilitating the calculation of the shear amount.

[0213] [Container Modification] In the above, examples have been shown in which physical tags are provided on box-shaped and cylindrical containers. However, one or more physical tags may be provided on containers of other shapes. Figures 24A and 24B show examples of other shapes.

[0214] 24A and 24B are diagrams showing an example in which a physical tag PT6 is provided on one side of the edge of an L-shaped container C6 when viewed from above. FIG. 24A shows a perspective view of the container C6, and FIG. 24B shows the shape of the edge SE6 corresponding to the container C6, which is stored in the image area information 154 (see FIG. 7). In this way, the image area information corresponding to the physical tag PT6 provided on the container C6 is stored in the image area information 154 (see FIG. 7). Note that the method of setting the image area is the same as in the above-described example. Furthermore, as in FIGS. 22A, 22B, 23A, and 23B, multiple physical tags may be provided on the container C6.

[0215] [Modification of information processing system] The above describes an example in which tag information of the containers C1 to C5 is acquired based on captured images generated by the imaging device 200. Here, the tag information of the containers C1 to C5 may be acquired using wireless communication. Therefore, the following describes an example in which tag information of the containers C1 to C5 is acquired using wireless communication.

[0216] [Example of use of information processing system] Fig. 25 is a diagram showing an example of use of the information processing system 1a. The information processing system 1a shown in Fig. 25 shows an example in which a communication device 400 is added to the information processing system 1 shown in Fig. 1, and physical tags PTT1 to PTT5 capable of wireless communication are provided on the containers C1 to C5. Note that, since other parts are common to the information processing system 1, the same reference numerals as in Fig. 1 are used and their description will be omitted. In the following, the physical tag PTT1 will be mainly described as an example, but the same applies to the physical tags PTT2 to PTT5.

[0217] The physical tag PTT1 is an integrated circuit (IC) tag capable of wireless communication. For example, the physical tag PTT1 can be provided inside a sheet that can be attached to the container C1. The physical tag PTT1 is an example of RFID (Radio Frequency Identification) and is a device capable of low-power wireless communication. The physical tag PTT1 can be configured as a wireless communication tag (RFID tag) compatible with RFID technology. For example, the physical tag PTT1 can be configured as an IC tag that employs a communication method such as Bluetooth Low Energy (BLE), Bluetooth, ZigBee, Low Power Wide Area (LPWA), or Ultra Wide Band (UWB), which are low-power communication modes. The maximum communication distance of the physical tag PTT1 is not particularly limited, but can be, for example, in the range of several tens of centimeters to several meters. For example, if the physical tag PTT1 complies with the BLE standard, it broadcasts packets at predetermined intervals (for example, every 1 to 10 seconds). A packet transmitted by this physical tag PTT1 includes a unique ID (Identification) (tag ID) that is identification information of the IC tag. Figures 25 to 27 show an example in which this tag ID is used as identification information (tag information) of the container C1 (waste).

[0218] That is, by reading the tag ID of the physical tag PTT1 provided on the container C1, it is possible to obtain the identification information of the object (PET) contained in the container C1.

[0219] Note that identification information that allows tag information to be acquired from the captured image generated by the imaging device 200 may be provided on the surface of the sheet provided with the physical tag PTT1. For example, triangular identification information PTT1a (see FIG. 26) that allows identification of the base point, size, and direction may be provided. Note that color codes or the like shown in FIGS. 2 to 5 may also be provided.

[0220] [Physical tag configuration example] 26 is a diagram showing an example of identification information PTT1a provided on the surface of a sheet having a physical tag PT1a. Note that the identification information PTT1a can be colored to identify the attributes of the waste contained in the container C1.

[0221] The identification information PTT1a is a figure formed by an isosceles triangle. The center position of the base of the isosceles triangle is defined as a base point PP1, the length of a line segment PP2 from the base point PP1 to the apex angle of the isosceles triangle is defined as a size PPL1 of the identification information PTT1a, and the angle of the line segment PP2 with respect to the horizontal direction is defined as a direction θ1 of the identification information PTT1a. The base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be determined based on the identification information PTT1a included in the captured image. The base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be used to perform affine transformation on the image region information stored in the image region information 154 (see FIG. 7).

[0222] [Examples of using information processing equipment] Fig. 27 is a block diagram showing an example of the system configuration of the information processing system 1a. The information processing system 1a is a partial modification of the information processing system 1 shown in Fig. 6. Specifically, the information processing system 1a is provided with a communication device 400 for reading the physical tag PTT1. Note that apart from the provision of the communication device 400, the information processing system 1a has the same components as the information processing system 1. Therefore, the same reference numerals are used for the components common to the information processing system 1, and some of these components will not be illustrated or described. Furthermore, the components different from the information processing system 1 will be described below as appropriate.

[0223] The information processing system 1a is configured by a plurality of devices that can be connected via a network N1. Fig. 27 shows, as an example, the information processing system 1a including an information processing device 100, an imaging device 200, a user terminal 300, and a communication device 400. Each of the information processing device 100, the imaging device 200, the user terminal 300, and the communication device 400 is connected to the network N1 by a communication method using wired communication or wireless communication.

[0224] Note that the information processing device 100, the imaging device 200, the user terminal 300, and the communication device 400 may be connected directly using wired or wireless communication without going through the network N1. Also, while Fig. 27 shows only the imaging device 200, the user terminal 300, and the communication device 400 as representative examples, the information processing system 1a may also include other imaging devices, other user terminals, and other communication devices installed at each location. Furthermore, as will be described later, if the position of the physical tag PTT1 can be identified by the communication device 400, the installation of the imaging device 200 may be omitted.

[0225] The communication device 400 and the physical tag PTT1 are connected by a direct connection using wireless communication without going through the network N1. Also, in Fig. 27, only the physical tag PTT1 and the container C1 are shown as representative examples, but the other physical tags PTT2 to PTT5 and the other containers C2 to C5 also constitute the information processing system 1a. Furthermore, as will be described later, when a plurality of communication devices 400 are used, the information processing system 1a is constituted by the plurality of communication devices 400.

[0226] The communication device 400 includes a communication unit 401 , a reading unit 402 , a control unit 403 , and a storage unit 404 .

[0227] The communication unit 401 exchanges various types of information with the information processing device 100 using wireless communication under the control of the control unit 403 .

[0228] The reading unit 402 is a communication unit that receives radio waves from the physical tag PTT1 provided on the container C1 and exchanges various information with the physical tag PTT1. Note that the wireless communication used by the physical tag PTT1 described above can be adopted as the wireless communication exchanged between the reading unit 402 and the physical tag PTT1. For example, the reading unit 402 reads the tag ID of the physical tag PTT1 and outputs this tag ID to the control unit 403.

[0229] The control unit 403 controls each unit of the communication device 400 based on a control program stored in the storage unit 404. The control unit 403 is realized by a processing device such as a CPU or a GPU. For example, the control unit 403 executes control to transmit the tag ID read by the reading unit 402 to the information processing device 100.

[0230] The storage unit 404 is a storage medium that stores various types of information. For example, the storage unit 404 stores various types of information (for example, a control program, location identification information for identifying the location where the communication device 400 is installed) that is required for the control unit 403 to perform various processes. The storage unit 404 also stores various types of information acquired via the communication unit 401. The storage unit 404 can be, for example, a ROM, a RAM, an SRAM, an HDD, an SSD, or a combination thereof.

[0231] [Example of location measurement using wireless tags] Here, a description will be given of a measurement method for measuring the position of a container using a wireless tag as the physical tag PTT 1. For example, the position of the wireless tag (position of the container) can be measured using a known indoor positioning system or the like.

[0232] For example, it is possible to use a measurement method that estimates the position of a wireless tag based on radio waves (radio waves emitted by the wireless tag) received by multiple communication devices. For example, three or more receivers can be installed at a waste collection site WP1 (see Figure 25), and the radio waves (radio waves emitted by the wireless tag) received by these receivers can be acquired. The position of the wireless tag can then be estimated by triangulation (cross-azimuth method) using the radio wave intensities.

[0233] Furthermore, for example, a measurement method can be used in which angle information between one or more communication devices and a wireless tag is calculated based on communication between the communication devices and the wireless tag, and the position of the wireless tag is estimated based on the angle information. For example, a reception angle detection technique (AoA (Angle of Arrival)) or a radiation angle detection technique (AoD (Angle of Departure)) can be used to calculate the angle information. For example, assume that containers C1 to C5 are located within a one-floor facility (waste collection site WP1) and an AoA receiver is installed on the ceiling of that floor. In this case, it is possible to calculate the angle of incidence of the radio waves (radio waves emitted by the wireless tag) received by the receiver installed on the ceiling. Then, the position of each wireless tag located on the floor of the waste collection site WP1 can be estimated based on the angle of incidence. Alternatively, the position of each wireless tag may be estimated using a receiver that can detect the position of the wireless tag that emitted the radio waves based on the directionality and reception strength of the received radio waves.

[0234] Using these position estimation techniques, it is possible to estimate the position of the physical tag PTT1 in three-dimensional space (waste collection point WP1). The estimation result (tag information and position) is transmitted by the communication device 400 to the information processing device 100. Note that the communication device 400 may transmit measurement data of the received radio waves to the information processing device 100, and the information processing device 100 may estimate the position of the physical tag PTT1 using the above-mentioned position estimation method.

[0235] Furthermore, by fixing the imaging range IM1 of the imaging device 200, it is possible to associate a three-dimensional space (the waste collection point WP1) with a two-dimensional space (the captured image corresponding to the imaging range IM1). Therefore, the tag information acquisition unit 122 can estimate the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) based on the position of the physical tag PTT1 at the waste collection point WP1.

[0236] In this way, the tag information acquisition unit 122 acquires the tag ID (tag information) of the physical tag PTT1 acquired by the communication device 400 and the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) estimated by the above-mentioned estimation method. Then, the tag information acquisition unit 122 associates the position of the physical tag PTT1 (position in the captured image) with the acquired tag information and outputs them to the setting unit 123, the detection unit 124, and the selection unit 125.

[0237] The setting unit 123 sets a predetermined image area from the captured image output from the image acquisition unit 121 based on the position and tag information of the physical tag PTT1 acquired by the tag information acquisition unit 122. For example, it is possible to set an image area of ​​a predetermined range based on the position of the physical tag PTT1 from the captured image corresponding to the imaging range IM1 (see FIG. 1). For example, it is possible to set a rectangular image area with the position of the physical tag PTT1 as the center and a size according to the attribute corresponding to the tag information.

[0238] 26, the identification information PTT1a of the physical tag PTT1 can be read, and based on this identification information PTT1a, the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a can be identified. Therefore, the setting unit 123 can affine transform the image region information (corresponding to the tag information of the physical tag PTT1) stored in the image region information 154 (see FIG. 7) using the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a. In this case, the setting unit 123 identifies the edge of the opening of the container C1 in the captured image based on the information about the physical tag PTT1 (the base point PP1, the size PPL1 of the identification information PTT1a, and the direction θ1 of the identification information PTT1a) and the tag information output from the tag information acquisition unit 122 (tag information corresponding to the physical tag PTT1).

[0239] Specifically, the setting unit 123 extracts image area information 154 corresponding to the tag information (tag information corresponding to the physical tag PTT1) output from the tag information acquisition unit 122 from the setting information DB 150 (see FIG. 7). Then, the setting unit 123 identifies the edge of the opening of the container C1 based on the extracted image area information and information about the physical tag PT1 (base point PP1, size PPL1 of the identification information PTT1a, and direction θ1 of the identification information PTT1a). This identification method is the same as the identification method shown in FIGS. 11 to 14, etc. Next, the setting unit 123 identifies an image area including the edge of the opening of the identified container C1, and sets the image area in the captured image.

[0240] The selection unit 125 selects a trained model group stored in the trained model DB 160 based on the tag information acquired by the tag information acquisition unit 122. Specifically, the selection unit 125 selects trained model group identification information 153 corresponding to the tag information 151 acquired by the tag information acquisition unit 122 from the setting information DB 150 (see FIG. 7).

[0241] The estimation unit 126 uses the trained model group selected by the selection unit 125 to estimate the weight of waste contained in a container included in the captured image output from the image acquisition unit 121. Here, if it is possible to estimate the position of the physical tag PTT1 in the captured image (corresponding to the imaging range IM1) using wireless communication, it is possible to associate the trained model group with the image area input thereto based on the position. Also, if it is not possible to estimate the position of the physical tag PTT1 using wireless communication, it is possible to associate the trained model group corresponding to the identification information PTT1a of the physical tag PTT1 with the image area identified based on the color of the identification information PTT1a.

[0242] [Example of effect of this embodiment] In recent years, various efforts to realize a circular economy have been widely underway. For example, there is a demand to recycle waste as a resource. For this reason, it is important to visualize the entire process of waste collection, processing, recycled material production, etc., and to achieve more advanced waste treatment and recycling. However, at many waste treatment sites, it is difficult to grasp the timing and amount of waste generated, which makes it difficult to plan efficient collection and processing.

[0243] Therefore, in this embodiment, containers C1 to C5 that store waste are provided with physical tags PT1 to PT5 and PTT1 to PTT5 from which tag information can be read. An imaging device 200 (or a communication device 400) is also installed at the waste disposal site. The information processing device 100 acquires tag information identifying each waste based on an image including the containers C1 to C5 generated by the imaging device 200 (or wireless communication by the communication device 400), and estimates the weight of the waste using the tag information. Specifically, the information processing device 100 uses the tag information to set an image area including the waste to be estimated from the image (or wireless communication) including the containers C1 to C5, and selects at least one trained model group from multiple trained model groups. The information processing device 100 then inputs the captured image in which the image area has been set to the selected trained model group, and estimates the weight of the waste based on the output from the trained model group.

[0244] This eliminates the need to simultaneously use multiple trained models to perform an estimation process to estimate the weight of waste from an image containing the waste, thereby reducing the amount of calculation required for the estimation process and increasing processing speed. Furthermore, because the weight of waste is estimated using an image region containing the waste to be estimated and trained models corresponding to the attributes of the waste to be estimated, the accuracy of the waste weight estimation can be improved. This makes it easy to grasp the timing and amount of waste generated at many waste disposal sites, making it easier to plan efficient collection and disposal. In other words, since more accurate weight can be determined, it becomes easier to plan waste collection timing, optimal vehicle dispatch arrangements, and collection routes so as not to exceed the load capacity of the waste collection vehicle. This makes it possible to realize a circular economy.

[0245] For example, using only one trained model (volume) may result in a decrease in the accuracy of estimating the weight of waste. For example, if waste (PET) is estimated using one trained model (volume), there is a risk of a large error between the volume and weight of the waste between an uncompressed PET bottle and a compressed PET bottle. Furthermore, for example, if the waste contains a roll of PET film, a sheet of PET film, or the like, there is a risk of a large error between the volume and weight of the waste. Therefore, in this embodiment, trained models for volume, shape, and density are selected based on attributes acquired from a physical tag associated with the waste. Then, values ​​related to the weight of the waste are calculated from captured images generated in chronological order using the trained models for the three elements (volume, shape, and density), and the weight is estimated by comprehensively evaluating each of these values.

[0246] [Example of weight estimation using image processing other than a trained model] The above describes an example of estimating the weight of waste using a trained model (AI model). However, the weight of waste may also be estimated using other image processing that can estimate the weight of the waste based on an image containing the target waste. As this image processing, for example, known object recognition technology (e.g., image recognition processing such as pattern matching processing or feature matching processing) can be used to perform the weight estimation process. For example, the weight of waste can be estimated using a first image processing corresponding to the volume of the waste, a second image processing corresponding to the shape of the waste, and a third image processing corresponding to the density of the waste, and the weight of the waste can be estimated using each of these estimation results. For example, the weight of the waste can be calculated and estimated using the above-mentioned Equations 1 to 4 for each estimation result obtained using the first to third image processing. In this case, the selection unit 125 may select at least one of the first image processing, the second image processing, and the third image processing. In this case, the estimation unit 126 can estimate the weight of the target object from the captured image using the selected image processing.

[0247] [Examples of application to other objects] In the above, waste has been described as an example of an object contained in a container. That is, an example of estimating the weight of waste (object) contained in a container has been shown. However, this embodiment can also be applied to objects other than waste contained in a container. For example, this embodiment can also be applied to estimating the weight of agricultural products, marine products, industrial products, etc., as objects contained in a container. This makes it possible to easily grasp the time and amount of objects produced at the processing site of each object, making it easier to plan efficient transportation, processing, etc. In other words, it is possible to realize a circular economy.

[0248] In the above, an example of estimating the weight of an object contained in a container has been shown. However, this embodiment can also be applied to an object that is not contained in a container. In this case, for example, a physical tag can be attached to the object itself or its surroundings (for example, a sign behind it), and the weight of the object can be estimated using this physical tag.

[0249] [Example of executing processing on other devices or systems] Although the above describes an example in which the tag information acquisition process, selection process, setting process, estimation process, etc. are executed in the information processing device 100, all or part of each of these processes may be executed in other devices. In this case, an information processing system is configured by the devices that execute part of each of these processes. For example, at least part of each process can be executed using devices available to the user (e.g., smartphones, tablet terminals, personal computers), various information processing devices such as servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0250] Furthermore, a part (or all) of the information processing system capable of executing the functions of the information processing device 100 (or the information processing system 1, 1a) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0251] [Configuration example and effect example of this embodiment] The main effects of the information processing device 100, the information processing system 1, 1a, the information processing method, and the program according to the embodiments of the present invention will be described below.

[0252] The information processing device 100 includes an image acquisition unit 121 (an example of an acquisition unit) that acquires images including containers C1 to C5 (an example of a storage unit) that store waste (an example of a target object), a selection unit 125 that selects, from among multiple image processing methods used to estimate the weight of the waste, a first image processing method corresponding to the volume of the waste and an image processing method (a second image processing method, a third image processing method) corresponding to at least one of the shape and density of the waste based on the attributes of the waste, and an estimation unit 126 that estimates the weight of the waste from the image using the selected image processing methods. It is also possible that a waste collection site will have one or more containers for only one type of waste (e.g., PET). In this case, the attributes of the waste can be acquired without acquiring tag information for each container (waste).

[0253] According to this, when estimating the weight of an object from an image including the object, it is possible to select and use image processing corresponding to the object from among multiple image processing methods based on the attributes of the object. This eliminates the need to simultaneously execute image processing that does not correspond to the attributes of the object to execute estimation processing to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation processing. Furthermore, because the weight of the object is estimated using multiple image processing methods corresponding to the attributes of the object simultaneously, it is possible to improve the estimation accuracy.

[0254] The image processing is image processing using a trained model used to estimate the weight of waste (an example of a target object). The selection unit 125 selects, from among a plurality of trained models (a first trained model group LL1 to an N-th trained model group LLN (where N is a natural number)), a trained model corresponding to the volume of the waste and a trained model corresponding to at least one of the shape and density of the waste based on the attributes of the waste. The estimation unit 126 estimates the weight of the waste from the image using each of the selected trained models.

[0255] According to this, when estimating the weight of waste from an image containing the waste using a trained model, it is possible to select and use a group of trained models corresponding to the waste from among multiple trained models based on the attributes of the waste. This makes it possible to reduce the amount of calculation involved in the estimation process, since it is not necessary to simultaneously use multiple trained models that do not correspond to the attributes of the waste to perform an estimation process to estimate the weight of the waste from an image containing the waste. Furthermore, since the weight of the waste is estimated simultaneously using multiple trained models that correspond to the attributes of the waste, it is possible to improve the estimation accuracy.

[0256] Physical tags PT1-PT5 and PTT1-PTT5 that can read the attributes of the waste are provided on the waste (an example of an object) or on the containers C1-C5 (an example of a container) that store the waste. The information processing device 100 further includes a tag information acquisition unit 122 (an example of an attribute acquisition unit) that acquires the attributes of the waste using the physical tags PT1-PT5 and PTT1-PTT5.

[0257] This makes it possible to acquire the attributes of the waste using the physical tags PT1 to PT11 and PTT1 to PTT5 attached to the waste or containers C1 to C5, thereby making it possible to appropriately select and use a group of trained models corresponding to the waste.

[0258] The plurality of trained models are set for each of the attributes of the plurality of wastes (an example of a target object). The selection unit 125 selects a trained model corresponding to the attribute of the waste from among the plurality of trained models.

[0259] According to this, it is possible to select and use a group of trained models corresponding to the waste from among a plurality of trained models set for each attribute of the waste. As a result, it is not necessary to simultaneously use a plurality of trained models that do not correspond to the attributes to perform an estimation process to estimate the weight of the waste from an image containing the waste, and therefore it is possible to reduce the amount of calculation involved in the estimation process. Furthermore, since the weight of the waste is estimated simultaneously using a plurality of trained models that correspond to the attributes of the waste, it is possible to improve the estimation accuracy.

[0260] The selection unit 125 may select a trained model corresponding to a volume according to the attributes of waste (an example of a target object) and a trained model corresponding to a shape according to the attributes of the waste.

[0261] This allows the weight of waste to be estimated using simultaneously a trained model corresponding to the volume according to the attributes of the waste and a trained model corresponding to the shape according to the attributes of the waste, thereby improving the estimation accuracy.

[0262] The selection unit 125 may select a trained model corresponding to a volume according to the attributes of waste (an example of a target object) and a trained model corresponding to a density according to the attributes of waste.

[0263] This allows the weight of waste to be estimated simultaneously using a trained model corresponding to the volume according to the attributes of the waste and a trained model corresponding to the density according to the attributes of the waste, thereby improving the estimation accuracy.

[0264] The selection unit 125 selects a trained model corresponding to the volume according to the attributes of the waste (an example of a target object), a trained model corresponding to the shape according to the attributes of the waste, and a trained model corresponding to the density according to the attributes of the waste.

[0265] This allows the weight of waste to be estimated simultaneously using a trained model corresponding to the volume according to the attributes of the waste, a trained model corresponding to the shape according to the attributes of the waste, and a trained model corresponding to the density according to the attributes of the waste, thereby improving the estimation accuracy.

[0266] The image acquisition unit 121 acquires images from the imaging device 200, which captures images of the containers C1 to C5 (examples of storage units) and generates the images. The estimation unit 126 estimates the weight of the waste (an example of a target object) at the time of capturing each of the multiple images generated in time series by the imaging device 200, and estimates the latest weight of the waste based on the estimation results for each image.

[0267] This allows the weight of the object at the time each image was captured to be estimated for each of multiple images generated in chronological order, and the latest weight of the object to be estimated based on the estimation results for each image, thereby improving estimation accuracy.

[0268] The information processing device 100 further includes a detection unit 124 that detects atypical waste (an example of a target object) in containers C1 to C5 (an example of a storage unit). When atypical waste is detected, the selection unit 125 selects a trained model corresponding to the atypical waste for the image generated at the time of the detection. For example, it is possible to select a trained model corresponding to a shape according to the attributes of the waste.

[0269] According to this, for images generated when detecting non-standard waste, the weight of the waste is estimated using a trained model corresponding to the non-standard waste, and the weight of the latest target object is estimated based on the estimation result, thereby improving estimation accuracy.

[0270] Containers C1 to C5 (an example of a storage section) store a plurality of wastes (an example of a target object). For example, wastes such as PET are stored in containers C1 to C5 by type. The shape according to the attribute of the waste is the shape of each of the plurality of wastes. The density according to the attribute of the waste is the density of the plurality of wastes in containers C1 to C5. The capacity according to the attribute of the waste is the capacity of the plurality of wastes in containers C1 to C5.

[0271] This makes it possible to select and use a group of trained models that correspond to the shape, density, and volume according to the attributes of the waste stored in the containers C1 to C5 (an example of a storage unit). This makes it possible to reduce the amount of calculation required for the estimation process and improve the estimation accuracy.

[0272] In addition to waste, the weight of at least one of agricultural products, marine products, and industrial products may be estimated.

[0273] This also makes it possible to reduce the amount of calculation required for the estimation process and improve estimation accuracy when using a trained model to estimate the weight of an object (at least one of waste, agricultural products, marine products, and industrial products) from an image containing the object.

[0274] An information processing method according to an embodiment of the present invention includes an image acquisition process (step S501) for acquiring an image including containers C1 to C5 (examples of storage sections) that store waste (an example of a target object), a selection process (step S506) for selecting, from among a plurality of image processes used when estimating the weight of the waste, a first image process corresponding to the volume of the waste and an image process (second image process, third image process) corresponding to at least one of the shape and density of the waste based on the attributes of the waste, and an estimation process (steps S507 to S512) for estimating the weight of the waste from the image using each of the selected image processes.

[0275] According to this, when estimating the weight of an object from an image including the object, it is possible to select and use image processing corresponding to the object from among multiple image processing methods based on the attributes of the object. This eliminates the need to simultaneously execute image processing that does not correspond to the attributes of the object to execute estimation processing to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation processing. Furthermore, because the weight of the object is estimated using multiple image processing methods corresponding to the attributes of the object simultaneously, it is possible to improve the estimation accuracy.

[0276] A program according to an embodiment of the present invention causes a computer to execute an image acquisition procedure (step S501) for acquiring an image including containers C1 to C5 (examples of storage sections) that store waste (an example of a target object); a selection procedure (step S506) for selecting, from among a plurality of image processes used when estimating the weight of the waste, a first image process corresponding to the volume of the waste and an image process (second image process, third image process) corresponding to at least one of the shape and density of the waste based on the attributes of the waste; and an estimation procedure (steps S507 to S512) for estimating the weight of the waste from the image using each of the selected image processes.

[0277] According to this, when estimating the weight of an object from an image including the object, it is possible to select and use image processing corresponding to the object from among multiple image processing methods based on the attributes of the object. This eliminates the need to simultaneously execute image processing that does not correspond to the attributes of the object to execute estimation processing to estimate the weight of the object from an image including the object, thereby reducing the amount of calculation involved in the estimation processing. Furthermore, because the weight of the object is estimated using multiple image processing methods corresponding to the attributes of the object simultaneously, it is possible to improve the estimation accuracy.

[0278] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that makes it possible to realize this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0279] Each process described in this embodiment is executed based on a program that causes a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to an information processing device can store the program in the storage device of the information processing device. This makes it possible to cause the updated information processing device to execute each process described in this embodiment.

[0280] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments. [Explanation of symbols]

[0281] 1, 1a Information Processing System 100 Information processing device 110 Communications Department 120 control section 121 Image acquisition unit 122 Tag information acquisition unit 123 Settings 124 Detector 125 Selection Section 126 Estimation Department 127 Output control section 128 Vehicle Dispatch Processing Unit 129 Correction section 130 Storage section 140 Tag DB 150 Setting information DB 160 trained model DB 170 Estimation result DB 200 Imaging device 201 Communications Department 202 Control section 203 Storage section 204 Imaging unit 300 User Terminals 301 Communications Department 302 Control Unit 303 Storage section 304 Operation section 305 Sound output unit 306 Display section 400 Communication equipment 401 Communications Department 402 Reading unit 403 Control Unit 404 Storage section C1~C6 container N1 Network PT1~PT6, PT1a, PT5a, PT1A, PT1B, PT1C, PT20, PT30, PT40, PTT1~PTT5 Physical tags WP1 Waste collection point

Claims

1. an image acquisition unit that acquires an image including a storage unit that stores an object; a selection unit that selects, from among a plurality of image processing methods used when estimating the weight of the object, an image processing method corresponding to the volume of the object and an image processing method corresponding to at least one of the shape and density of the object based on the attributes of the object; an estimation unit that estimates the weight of the object from the image using each selected image processing; An information processing device having the above.

2. 2. The information processing device according to claim 1, the image processing is image processing using a trained model used when estimating the weight of the object, the selection unit selects, from the plurality of trained models, a trained model corresponding to a volume of the object and a trained model corresponding to at least one of a shape and a density of the object based on attributes of the object; The estimation unit estimates the weight of the object from the image using each selected trained model. Information processing device.

3. 3. The information processing device according to claim 1, a physical tag capable of reading attributes of the object is provided on the object or the container; an attribute acquisition unit that acquires attributes of the object using the physical tag; Information processing device.

4. 3. The information processing device according to claim 2, The plurality of trained models are set for each of the plurality of attributes of the objects, The selection unit selects a trained model according to an attribute of the object from among the plurality of trained models. Information processing device.

5. 5. The information processing device according to claim 4, the selection unit selects a trained model corresponding to the capacity according to the attribute of the object and a trained model corresponding to the shape according to the attribute of the object. Information processing device.

6. 5. The information processing device according to claim 4, the selection unit selects a trained model corresponding to the capacity according to the attribute of the object and a trained model corresponding to the density according to the attribute of the object. Information processing device.

7. 5. The information processing device according to claim 4, the selection unit selects a trained model corresponding to the capacity according to the attribute of the object, a trained model corresponding to the shape according to the attribute of the object, and a trained model corresponding to the density according to the attribute of the object. Information processing device.

8. 8. An information processing device according to claim 2, wherein: the image acquisition unit acquires the image from an imaging device that captures an image of the storage unit and generates the image; the estimation unit estimates a weight of the object at the time of capturing each of the plurality of images generated in time series by the imaging device, and estimates a latest weight of the object based on the estimation results of each of the images; Information processing device.

9. 9. The information processing device according to claim 8, a detection unit for detecting the target object of an irregular shape in the storage unit; When the target object is an atypical object, the selection unit selects a trained model corresponding to the atypical object for the image generated at the time of the detection. Information processing device.

10. 8. An information processing device according to claim 1, 2, 4 to 7, The storage unit stores a plurality of the objects, the shape is a shape of each of the plurality of objects; the density is a density of the plurality of objects in the storage section, The capacity is the capacity of the plurality of objects in the storage section. Information processing device.

11. 8. An information processing device according to claim 1, 2, 4 to 7, The object is at least one of waste, agricultural products, marine products, and industrial products. Information processing device.

12. an image acquisition process for acquiring an image including a storage section for storing an object; a selection process for selecting, from among a plurality of image processes used when estimating the weight of the object, an image process corresponding to the volume of the object and an image process corresponding to at least one of the shape and density of the object based on the attributes of the object; an estimation process for estimating the weight of the object from the image using each selected image processing; An information processing method including:

13. an image acquisition step of acquiring an image including a container that contains an object; a selection step of selecting, from among a plurality of image processing methods used when estimating the weight of the object, an image processing method corresponding to the volume of the object and an image processing method corresponding to at least one of the shape and density of the object based on the attributes of the object; an estimation procedure for estimating the weight of the object from the image using each selected image processing; A program that causes a computer to execute the following.

Images