IMAGE PROCESSING APPARATUS, PROGRAM, AND IMAGE PROCESSING METHOD

Abstract

To provide an image processing apparatus which can easily be installed instead of X-ray inspection device and can inspect hand baggage at an event site or the like in a non-contact fashion.SOLUTION: In a system for communication of an inspection facility having an information processing apparatus and a server through a network, a server 100 operating as an image processing apparatus has a captured image acquiring unit 106 for acquiring an image of an opened bag, a weight estimating unit 110 for recognizing a plurality of objects in the bag based on the bag image to estimate weight of the bag including the plurality of objects, a measured value acquiring unit 108 for acquiring a measured value obtained by measuring the weight of the bag, and a determination unit 112 for determining, based on difference between the estimated weight estimated by the weight estimating unit and the measured value, whether or not any dangerous thing is hidden in the bag. The weight estimating unit estimates the weight of the bag including the plurality of objects by inputting the bag image to a learning model which estimates, from an image, an object included in the image and is generated by using machine learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, a program, and an image processing method. [Background technology]

[0002] Patent Document 1 describes an X-ray baggage inspection device. [Prior art document] [Patent documents] [Patent Document 1] JP 2021-099246 A Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment of the present invention, there is provided an image processing device. The image processing device may include an image acquisition unit that acquires an image of an open bag. The image processing device may include a weight estimation unit that recognizes a plurality of objects in the bag based on the bag image and estimates a weight of the bag including the plurality of objects. The image processing device may include an actual measurement value acquisition unit that acquires an actual measurement value of the weight of the bag. The image processing device may include a determination unit that determines whether or not a dangerous object is concealed in the bag based on a difference between the estimated weight estimated by the weight estimation unit and the actual measurement value.

[0004] The captured image acquisition unit may acquire a bag image of the bag placed on a weighing scale, and the actual measurement value acquisition unit may acquire the actual measurement value measured by the weighing scale. The captured image acquisition unit may acquire a holder image of the bag holder who places the bag on the weighing scale, and the determination unit may determine whether or not a dangerous object is concealed in the bag based on the difference between the estimated weight and the actual measurement value and the holder image. The determination unit may determine whether or not a dangerous object is concealed in the bag based on the suspiciousness of the holder estimated based on the holder image and the difference between the estimated weight and the actual measurement value. The determination unit may determine that a dangerous object is not concealed in the bag when the difference between the estimated weight and the actual measurement value is smaller than a difference threshold corresponding to the suspiciousness of the holder, and may determine that a dangerous object is concealed in the bag when the difference is larger than the difference threshold. The determination unit may estimate the suspiciousness of the holder by inputting the holder image into a learning model that estimates the suspiciousness of a person from an image of the person. The determination unit may input the holder image into the learning model generated by machine learning using an image including a person as a subject and suspiciousness information of the person as teacher data.

[0005] The image processing device may further include a warning control unit configured to control the bag to output a warning when the determination unit determines that a dangerous object is hidden in the bag. The warning control unit may control the output of the warning with a higher warning level as the difference between the estimated weight and the actual measurement value increases. The weight estimation unit may estimate the weight of the bag including the multiple objects by inputting the bag image into a learning model that estimates objects included in an image from the image. The weight estimation unit may input the bag image into the learning model to recognize the multiple objects in the bag, and estimate the weight of the bag including the multiple objects by referring to weight data in which the weights of each of the multiple types of objects are registered. The weight estimation unit may input the bag image into the learning model generated by machine learning using an image including an object as a subject and object identification information capable of identifying the object as teacher data.

[0006] The determination unit may determine that a dangerous object is not concealed in the bag if the difference between the estimated weight and the actual measurement value is smaller than a predetermined difference threshold, and may determine that a dangerous object is concealed in the bag if the difference is larger than the difference threshold. The determination unit may determine that a dangerous object is not concealed in the bag if the difference between the estimated weight and the actual measurement value is smaller than the difference threshold based on the size of the bag, and may determine that a dangerous object is concealed in the bag if the difference is larger than the difference threshold. The determination unit may determine that a dangerous object is not concealed in the bag if the difference between the estimated weight and the actual measurement value is smaller than the difference threshold, which indicates a higher value as the size of the bag increases, and may determine that a dangerous object is concealed in the bag if the difference is larger than the difference threshold. The determination unit may determine that a dangerous object is not concealed in the bag when the difference between the estimated weight and the actual measurement value is smaller than the difference threshold value according to the shape of the bag, and may determine that a dangerous object is concealed in the bag when the difference is larger than the difference threshold value. The device may further include an instruction control unit that controls to output instruction information to a holder of the bag when the difference between the estimated weight and the actual measurement value is larger than an instruction threshold value that is higher than the difference threshold value.

[0007] According to one embodiment of the present invention, there is provided a system including the image processing device described above and an imaging device that captures an image of a bag.

[0008] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the image processing device.

[0009] According to one embodiment of the present invention, there is provided an image processing method executed by a computer. The information processing method may include an image acquisition step of acquiring a bag image of an open bag. The information processing method may include a weight estimation step of recognizing a plurality of objects in the bag based on the bag image and estimating a weight of the bag including the plurality of objects. The information processing method may include an actual measurement value acquisition step of acquiring an actual measurement value of the weight of the bag. The information processing method may include a determination step of determining whether or not a dangerous object is concealed in the bag based on a difference between the estimated weight estimated in the weight estimation step and the actual measurement value.

[0010] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0011] [Figure 1] An example of a system 10 is shown diagrammatically. [Diagram 2] 1 illustrates an example of a functional configuration of a server 100. [Diagram 3] 1 shows an example of a processing flow by the server 100. [Figure 4] 1 shows an example of a processing flow by the server 100. [Diagram 5] 1 shows an example of a processing flow by the server 100. [Figure 6] An example of the functional configuration of the information processing device 200 when the inspection is performed independently is shown in schematic form. [Figure 7]An example of a hardware configuration of a computer 1200 functioning as the server 100 or the information processing device 200 is illustrated in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In places with good facilities such as airports, there are inspection systems such as X-ray inspection devices, and users are fully educated. However, in places without sufficient facilities such as events, visual baggage inspection is performed. There is a limit to the education, and it is difficult to find hidden dangerous objects. The system 10 according to the present embodiment can be easily installed in place of, for example, an X-ray inspection device, and enables automatic, non-contact baggage inspection at event venues and the like. As a specific example, the system 10 performs object recognition in real time from an image of the inside of an open bag 30, and estimates the weight of each object. In addition, a weighing scale 320 is installed at the place where the bag 30 is placed, and the weight is compared with the actual weight. If the difference is the weight including the dangerous object, a warning is output and a check is performed by a security guard or the like. In addition, suspicious movements and the weight of the bag 30 are estimated from the movement of the holder 32 when placing the bag 30 on the weighing scale 320, and accuracy is improved.

[0013] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0014] FIG. 1 illustrates an example of a system 10. The system 10 may include a server 100. The server 100 may be an example of an image processing device. The system 10 may include an inspection equipment 300. The inspection equipment 300 may include an information processing device 200. The inspection equipment 300 may include a camera 310. The camera 310 may be an example of an imaging device that captures an image of the bag 30. The inspection equipment 300 may include a weighing scale 320. The inspection equipment 300 may include a display 330.

[0015] The information processing device 200 may receive an image captured by the camera 310 from the camera 310 and transmit the image to the server 100. For example, the information processing device 200 transmits a bag image of the bag 30 to the server 100. For example, the information processing device 200 transmits a carrier image of the carrier 32 to the server 100. The information processing device 200 and the camera 310 may be connected by wire. The information processing device 200 and the camera 310 may be connected wirelessly. The information processing device 200 may transmit the captured image to the server 100 via the network 20. Note that the camera 310 may transmit the captured image directly to the server 100.

[0016] The information processing device 200 may receive from the weighing scale 320 the actual measurement value of an object measured by the weighing scale 320, and transmit the measurement value to the server 100. For example, the information processing device 200 transmits the actual measurement value of the bag 30 to the server 100. The information processing device 200 and the weighing scale 320 may be connected by wire. The information processing device 200 and the weighing scale 320 may be connected wirelessly. The information processing device 200 may transmit the actual measurement value to the server 100 via the network 20. Note that the weighing scale 320 may transmit the actual measurement value directly to the server 100.

[0017] The information processing device 200 may display various information on the display 330 in accordance with an instruction from the server 100. For example, the information processing device 200 displays instruction information for the holder 32 on the display 330. For example, the information processing device 200 displays warning information on the display 330. The display 330 may include a speaker, and the information processing device 200 may output various information by voice on the display 330 in accordance with an instruction from the server 100. For example, the information processing device 200 outputs instruction information for the holder 32 by voice on the display 330. For example, the information processing device 200 outputs warning information by voice on the display 330. Note that the display 330 may directly display various information or output it by voice in accordance with an instruction from the server 100.

[0018] The network 20 may include a mobile communication network. The mobile communication network may be compliant with any of the following communication methods: 3G (3rd Generation), LTE (Long Term Evolution), 5G (5th Generation), and 6G (6th Generation) or later. The network 20 may include the Internet. The network 20 may include a LAN (Local Area Network).

[0019] The server 100 may be connected to the network 20 by wire. The server 100 may be connected to the network 20 wirelessly. The server 100 may be connected to the network 20 via a wireless base station. The server 100 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

[0020] The information processing device 200 may be wirelessly connected to the network 20. The information processing device 200 may be connected to the network 20 via a wireless base station. The information processing device 200 may be connected to the network 20 via a Wi-Fi access point. The information processing device 200 may be wired to the network 20. Similarly, the camera 310, the weighing scale 320, and the display 330 may be wirelessly connected to the network 20 or wired to the network 20.

[0021] If baggage inspection is performed using only image processing, it is difficult to detect hidden dangerous objects such as weapons. In contrast, the system 10 performs baggage inspection using image processing and a weight scale. Here, it is assumed that dangerous objects such as weapons have a certain weight.

[0022] Server 100 acquires a bag image of bag 30 captured by camera 310 with the bag open, recognizes multiple objects inside bag 30 based on the bag image, and estimates the weight of bag 30 including the multiple objects. Server 100 also acquires an actual measurement value indicating the weight of bag 30 measured by weighing scale 320, and calculates the difference between the estimated weight of bag 30 and the actual measurement value. If the difference is greater than a threshold, it is estimated that something heavy is hidden, and server 100 outputs a warning. Server 100 outputs a warning to, for example, an administrator who manages baggage inspection.

[0023] The method of outputting the warning may be any method. For example, the server 100 transmits warning information to a communication terminal owned by the administrator. For example, the server 100 displays the warning information on a display being viewed by the administrator. Also, for example, the server 100 displays the warning information on the display 330.

[0024] 2 shows an example of a schematic functional configuration of the server 100. The server 100 includes a storage unit 102, a registration unit 104, a captured image acquisition unit 106, an actual measurement value acquisition unit 108, a weight estimation unit 110, a determination unit 112, a warning control unit 114, an instruction control unit 116, and a model generation unit 118. Note that it is not essential that the server 100 includes all of these units.

[0025] The storage unit 102 stores various types of information. The storage unit 102 may store various types of information registered by the registration unit 104.

[0026] The registration unit 104 registers, for example, information input via a user interface of the server 100. The registration unit 104 registers, for example, information received from other devices. The registration unit 104 registers, for example, information received from other devices via the network 20.

[0027] The registration unit 104, for example, registers a learning model that estimates an object included in an image from the image. The registration unit 104, for example, registers weight data in which the weights of multiple types of objects are registered. The registration unit 104, for example, registers a learning model that estimates the suspiciousness of a person from an image of the person.

[0028] The captured image acquisition unit 106 acquires a captured image captured by the camera 310. The captured image acquisition unit 106 stores the acquired captured image in the storage unit 102. The captured image acquisition unit 106 acquires, for example, a bag image captured by the camera 310 of the bag 30 with its opening. The captured image acquisition unit 106 may acquire a bag image captured by the bag 30 placed on the weighing scale. The bag image may be a still image or a video.

[0029] The captured image acquisition unit 106 acquires, for example, a bearer image captured by the camera 310 of the bearer 32 of the bag 30. The captured image acquisition unit 106 may acquire a bearer image captured by the bearer 32 placing the bag 30 on the weighing scale 320. The bearer image may be a still image or a video. When analyzing the movement of the bearer, the bearer image may be a video. The bearer image may be used for face authentication of the bearer 32. When the bearer image is used for face authentication of the bearer 32, the camera 310 may be placed in a position where it can capture images of the contents of the bag 30 and the face of the bearer 32. Note that the captured image acquisition unit 106 may acquire an image of the bag 30 from the camera 310, and acquire an image of the face of the bearer 32 from a camera other than the camera 310. For example, the camera may be placed in a position in front of the bearer 32, or in a position where it can capture an image of the face of the bearer 32 placing the bag 30 on the weighing scale 320 from below.

[0030] The actual measurement value acquiring section 108 acquires an actual measurement value obtained by actually measuring the weight of the bag 30. The actual measurement value acquiring section 108 may acquire an actual measurement value measured by a weighing scale 320. The actual measurement value acquiring section 108 causes the memory section 102 to store the acquired actual measurement value.

[0031] The weight estimation unit 110 recognizes multiple objects in the bag 30 based on the bag image, and estimates the weight of the bag 30 including the multiple objects. The weight estimation unit 110 may estimate the weight of the bag 30 including the multiple objects by inputting the bag image to a learning model that is stored in the storage unit 102 and that estimates objects included in an image from the image.

[0032] Weight estimation unit 110, for example, recognizes multiple objects in bag 30 by inputting a bag image into the learning model, and estimates the weight of bag 30 including the multiple objects by referring to weight data stored in storage unit 102. Weight estimation unit 110 may estimate the weight of bag 30 including the multiple objects by extracting the weight of each of the multiple objects and the weight of bag 30 by referring to the weight data, and adding them up.

[0033] The learning model may be a learning model that estimates the weight of multiple objects included in an image from the image. In this case, the weight estimation unit 110 can estimate the weight of the bag 30 including multiple objects by inputting the bag image to the learning model.

[0034] The determination unit 112 determines whether or not a dangerous object is concealed in the bag 30 based on the difference between the estimated weight of the bag 30 estimated by the weight estimation unit 110 and the actual measurement value of the weight of the bag 30 measured by the actual measurement value acquisition unit 108. The determination unit 112 may determine whether or not a dangerous object is concealed in the bag 30 based on the difference between the estimated weight and the actual measurement value and the carrier image. The determination unit 112 may determine whether or not a dangerous object is concealed in the bag 30 based on the suspiciousness of the carrier 32 estimated based on the carrier image and the difference between the estimated weight and the actual measurement value.

[0035] For example, when the suspiciousness of bag 30 is lower than a predetermined suspiciousness threshold, and the difference between the estimated weight and the actual measurement value is smaller than a predetermined difference threshold, determination unit 112 determines that a dangerous object is not concealed in bag 30, and when the difference is greater than the difference threshold, determination unit 112 determines that a dangerous object is concealed in bag 30. Then, when the suspiciousness of bag 30 is higher than the predetermined suspiciousness threshold, determination unit 112 lowers the difference threshold, and when the difference between the estimated weight and the actual measurement value is smaller than the predetermined difference threshold, determines that a dangerous object is not concealed in bag 30, and when the difference is greater than the difference threshold, determines that a dangerous object is concealed in bag 30.

[0036] Also, for example, when the difference between the estimated weight and the actual measured value is smaller than the difference threshold corresponding to the suspiciousness level of the carrier 32, the determination unit 112 determines that a dangerous object is not concealed in the bag 30, and when the difference is larger than the difference threshold, the determination unit 112 determines that a dangerous object is concealed in the bag 30. The difference threshold corresponding to the suspiciousness level may indicate a lower value as the suspiciousness level increases.

[0037] For example, when inspecting the luggage of event participants, if the difference threshold is set low to prevent dangerous objects from being overlooked, the dangerous objects will be determined to be concealed during most inspections. In response to this, the efficiency of inspections can be improved by adjusting the difference threshold depending on the behavior of the holder 32, or by using a difference threshold that matches the behavior.

[0038] The determination unit 112 may estimate the suspiciousness of the holder 32 by analyzing the holder image. For example, when the determination unit 112 detects suspicious movements or a suspicious appearance of the holder 32 by analyzing the holder image, the determination unit 112 estimates that the suspiciousness is high. Examples of suspicious movements include large movements and looking around a lot. Whether the holder 32 looks around a lot can be determined based on whether the face and head rotation speed of the holder 32 exceed a threshold, whether the swing range of the face of the holder 32 exceeds a threshold, etc. Examples of suspicious appearances include the holder 32 looking heavy compared to the size of the bag 30, the holder 32 having a bad complexion, etc. Whether the holder 32 has a bad complexion can be determined based on whether the amount of sweat of the holder 32 exceeds a threshold, whether the blinking frequency of the holder 32 exceeds a threshold, etc.

[0039] The determination unit 112 may estimate the suspiciousness degree of the holder 32 by inputting the holder image to a learning model, which is stored in the storage unit 102 and which estimates the suspiciousness degree of a person from an image of the person.

[0040] The determination unit 112 may determine that no dangerous object is concealed in the bag 30 when the difference between the estimated weight and the actual measured value is smaller than a predetermined difference threshold, and may determine that a dangerous object is concealed in the bag 30 when the difference is larger than the difference threshold, but may use multiple difference thresholds. For example, a difference threshold based on information about the bag 30 identified by analyzing the bag image is used.

[0041] As a specific example, when the difference between the estimated weight and the actual measurement value is smaller than a difference threshold based on the size of bag 30, determination unit 112 determines that a dangerous object is not concealed in bag 30, and when the difference is larger than the difference threshold, determination unit 112 determines that a dangerous object is concealed in bag 30. When the difference between the estimated weight and the actual measurement value is smaller than a difference threshold that indicates a higher value the larger the size of bag 30, determination unit 112 may determine that a dangerous object is not concealed in bag 30, and when the difference is larger than the difference threshold, determination unit 112 may determine that a dangerous object is concealed in bag 30. This can improve the accuracy of determining dangerous objects.

[0042] As a specific example, when the difference between the estimated weight and the actual measured value is smaller than a difference threshold corresponding to the shape of bag 30, determination unit 112 determines that a dangerous object is not concealed in bag 30, and when the difference is larger than the difference threshold, determines that a dangerous object is concealed in bag 30. For example, the difference threshold corresponding to a shape that widens toward the bottom is set to a lower value than the difference threshold corresponding to a shape that does not widen toward the bottom. This makes it easier to determine that a dangerous object is concealed in a bag 30 that has a wide bottom and a shape that makes it easy to conceal a dangerous object, and improves the accuracy of determining dangerous objects.

[0043] The determination unit 112 may perform face authentication of the holder 32. The determination unit 112 performs face authentication of the holder 32 using the face image of the holder 32 acquired by the captured image acquisition unit 106. For example, when inspecting entry to a venue, the determination unit 112 determines whether the holder 32 is a person authorized to enter by face authentication. Such a determination can be realized, for example, by registering face images of people authorized to enter in advance. Also, for example, the determination unit 112 determines whether the holder 32 is a dangerous person by face authentication. Such a determination can be realized, for example, by registering face images of dangerous people in advance. By the determination unit 112 performing face authentication of the holder 32, baggage inspection and entrance inspection can be performed together, and smooth and secure entry can be realized.

[0044] The warning control unit 114 performs control so as to output a warning when the determination unit 112 determines that a dangerous object is concealed in the bag 30. The warning output method may be any method as described above. For example, the warning control unit 114 transmits warning information to communication terminals of the manager who manages the inspection equipment 300 and the person in charge of inspection at the inspection equipment 300, or causes the warning information to be displayed on a display viewed by the manager and the person in charge, or to be output as audio. Also, for example, the warning control unit 114 causes the display 330 to display the warning information.

[0045] The warning control unit 114 may perform control so that the greater the difference between the estimated weight and the actual measurement value, the higher the level of warning to be output. For example, the greater the difference between the estimated weight and the actual measurement value, the greater the level of emphasis of the warning information to be displayed. Also, for example, the greater the difference between the estimated weight and the actual measurement value, the greater the volume of the warning information to be output by the warning control unit 114. Also, for example, the greater the difference between the estimated weight and the actual measurement value, the greater the number of targets to which the warning information is output by the warning control unit 114.

[0046] The warning control unit 114 may perform control to output a warning in accordance with the authentication result when the determination unit 112 executes face authentication of the holder 32. For example, the warning control unit 114 performs control to output a warning when it is determined that the holder 32 does not have the authority to enter. For example, the warning control unit 114 performs control to output a warning when it is determined that the holder 32 is a dangerous person.

[0047] Instruction control unit 116 controls to output instruction information to bearer 32 of bag 30. Instruction control unit 116 causes display 330 to output instruction information instructing bearer 32 to open bag 30 and place it on weighing scale 320, for example.

[0048] The instruction control unit 116 may perform control to output instruction information based on the result of analyzing the bag image. For example, when the result of analyzing the bag image shows that an object whose contents are not visible, such as an opaque pouch, is contained in the bag 30, the instruction control unit 116 causes the display 330 to display instruction information instructing the user to open the object. At this time, the instruction control unit 116 may also cause the display 330 to display a cut-out image of the object cut out from the bag image.

[0049] The instruction control unit 116 may perform control to output instruction information according to the difference between the estimated weight and the actual measurement value. For example, when the difference between the estimated weight and the actual measurement value is greater than an instruction threshold value that is higher than the difference threshold value, the instruction control unit 116 performs control to output instruction information to the holder 32 of the bag 30. The instruction information includes, for example, an instruction to open the opening of the bag 30 wider so that the contents of the bag 30 can be easily seen. The instruction information also includes, for example, an instruction to take some objects in the bag 30 out of the bag 30. When the difference between the estimated weight and the actual measurement value is very large, there is a possibility that an object that cannot be seen from the open part is present at the bottom of the bag 30, but by outputting such instruction information, it is possible to make the object visible.

[0050] The model generation unit 118 generates a learning model. The model generation unit 118 stores the generated learning model in the storage unit 102.

[0051] For example, the model generation unit 118 generates a learning model that estimates an object included in an image from the image. The model generation unit 118 may generate the learning model by machine learning using an image including an object as a subject and object identification information capable of identifying the object as training data. Such training data may be registered by the registration unit 104.

[0052] The model generation unit 118 may generate a learning model that estimates the weight of an object included in an image from the image. The model generation unit 118 may generate the learning model by planning and estimating an image including an object as a subject and the weight of the object as training data. Such training data may be registered by the registration unit 104.

[0053] For example, the model generation unit 118 generates a learning model that estimates the suspiciousness of a person from an image of the person. The model generation unit 118 may generate the learning model by machine learning using an image including a person as a subject and suspiciousness information of the person as training data. Such training data may be registered by the registration unit 104.

[0054] 3 shows an example of a processing flow by the server 100. Here, a processing flow will be described in which the server 100 inspects the bags 30 of a plurality of carriers 32 in sequence.

[0055] In step (sometimes abbreviated to S) 102, the captured image acquisition unit 106 acquires a bag image of the bag 30 captured by the camera 310. In S104, the actual measurement acquisition unit 108 acquires the actual measurement value of the weight of the bag 30 measured by the weighing scale 320.

[0056] In S106, the weight estimation unit 110 recognizes a plurality of objects in the bag 30 based on the shoe image acquired in S102. In S108, the weight estimation unit 110 estimates the weight of the bag 30.

[0057] In S110, determination unit 112 determines whether the difference between the estimated weight of bag 30 estimated in S108 and the actual measurement value acquired in S104 is greater than a difference threshold value. If it is determined that it is greater, the process proceeds to S114, and if it is not determined that it is greater, the process proceeds to S116.

[0058] In S114, the warning control unit 114 outputs a warning. If inspection has not been completed for all of the multiple carriers 32 (NO in S318), the process returns to S102 to obtain a bag image of the bag 30 of the next carrier 32, and if inspection has been completed for all of the multiple carriers 32 (YES in S318), the process ends.

[0059] Fig. 4 shows an example of a processing flow by the server 100. Differences from Fig. 3 will be mainly described here.

[0060] In S202, the captured image acquisition unit 106 acquires a carrier image of the carrier 32 captured by the camera 310. In S204, the determination unit 112 estimates the suspiciousness degree of the carrier 32 based on the carrier image acquired in S202.

[0061] In S206, the captured image acquisition section 106 acquires a bag image of the bag 30 captured by the camera 310. In S208, the actual measurement value acquisition section 108 acquires the actual measurement value of the weight of the bag 30 measured by the weighing scale 320.

[0062] In S210, the weight estimation unit 110 recognizes a plurality of objects in the bag 30 based on the shoe image acquired in S206. In S212, the weight estimation unit 110 estimates the weight of the bag 30.

[0063] In S214, the determination unit 112 determines whether the suspiciousness of the holder 32 estimated in S204 is higher than a predetermined suspiciousness threshold. If it is determined to be higher, the process proceeds to S216, and if it is not determined to be higher, the process proceeds to S218. In S216, the determination unit 112 changes the difference threshold to a lower value.

[0064] In S218, the determination unit 112 determines whether the difference between the estimated weight of the bag 30 estimated in S212 and the actual measurement value acquired in S208 is greater than the difference threshold value. If it is determined that it is greater, the process proceeds to S220, and if it is not determined that it is greater, the process proceeds to S222.

[0065] In S220, the warning control unit 114 outputs a warning. If the inspection has not been completed for all of the multiple bearers 32 (NO in S318), the process returns to S202 to obtain a bearer image of the next bearer 32, and if the inspection has been completed for all of the multiple bearers 32 (YES in S318), the process ends.

[0066] Note that acquisition of the carrier image in S202 and acquisition of the actual measured value of the weight of the bag 30 in S208 may be performed simultaneously. Furthermore, the determination unit 112 may further perform face authentication of the carrier 32 on the carrier image acquired in S202.

[0067] Fig. 5 shows an example of a processing flow by the server 100. Differences from Fig. 3 will be mainly described here.

[0068] In S302, the captured image acquisition section 106 acquires a bag image of the bag 30 captured by the camera 310. In S304, the actual measurement value acquisition section 108 acquires the actual measurement value of the weight of the bag 30 measured by the weighing scale 320.

[0069] In S306, the weight estimation unit 110 recognizes a plurality of objects in the bag 30 based on the shoe image acquired in S302. In S308, the weight estimation unit 110 estimates the weight of the bag 30.

[0070] In S310, the instruction control unit 116 determines whether or not the difference between the estimated weight of the bag 30 estimated in S308 and the actual measurement value acquired in S304 is greater than the instruction threshold. If it is determined to be greater, the process proceeds to S312, and if it is not determined to be greater, the process proceeds to S314. In S312, the instruction control unit 116 outputs instruction information. After outputting the instruction information, the captured image acquisition unit 206 captures an image of the bag 30 again.

[0071] In S314, the determination unit 112 determines whether the difference between the estimated weight of the bag 30 estimated in S308 and the actual measurement value acquired in S304 is greater than the difference threshold value. If it is determined that it is greater, the process proceeds to S316, and if it is not determined that it is greater, the process proceeds to S318.

[0072] In S316, the warning control unit 114 outputs a warning. If inspection has not been completed for all of the multiple carriers 32 (NO in S318), the process returns to S302 to obtain a bag image of the bag 30 of the next carrier 32, and if inspection has been completed for all of the multiple carriers 32 (YES in S318), the process ends.

[0073] In the above embodiment, the server 100 mainly executes the inspection, but the present invention is not limited to this, and the information processing device 200 may mainly execute the inspection. In this case, the information processing device 200 may be an example of an image processing device.

[0074] 6 shows an example of a functional configuration of the information processing device 200 when the information processing device 200 proactively executes an inspection. The information processing device 200 includes a storage unit 202, a registration unit 204, a captured image acquisition unit 206, an actual measurement value acquisition unit 208, a weight estimation unit 210, a determination unit 212, a warning control unit 214, an instruction control unit 216, and a model generation unit 218. Note that it is not essential that the information processing device 200 includes all of these units.

[0075] The storage unit 202, the registration unit 204, the captured image acquisition unit 206, the actual measurement value acquisition unit 208, the weight estimation unit 210, the determination unit 212, the warning control unit 214, and the instruction control unit 216 may have the same functions as the storage unit 102, the registration unit 104, the captured image acquisition unit 106, the actual measurement value acquisition unit 108, the weight estimation unit 110, the determination unit 112, the warning control unit 114, and the instruction control unit 116, respectively. Here, differences from the storage unit 102, the registration unit 104, the captured image acquisition unit 106, the actual measurement value acquisition unit 108, the weight estimation unit 110, the determination unit 112, the warning control unit 114, and the instruction control unit 116 will be mainly described.

[0076] The storage unit 202 stores various information. The registration unit 204 registers, for example, information input via a user interface of the information processing device 200. The captured image acquisition unit 106 acquires an image captured by the camera 310 from the camera 310. The actual measurement value acquisition unit 108 acquires, from the weighing scale 320, an actual measurement value indicating the weight of the bag 30 actually measured by the weighing scale 320.

[0077] Weight estimation unit 210 recognizes multiple objects in bag 30 based on the bag image and estimates the weight of bag 30 including the multiple objects. Determination unit 212 determines whether or not a dangerous object is concealed in bag 30 based on the difference between the estimated weight of bag 30 estimated by weight estimation unit 210 and the actual value of the weight of bag 30 actually measured by actual measurement value acquisition unit 208.

[0078] Warning control unit 214 performs control to output a warning when determination unit 212 determines that a dangerous object is concealed in bag 30. Instruction control unit 216 performs control to output instruction information to carrier 32 of bag 30. Model generation unit 218 generates a learning model.

[0079] 7 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the server 100 or the information processing device 200. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations associated with the device according to the present embodiment or one or more "parts", and / or cause the computer 1200 to execute a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0080] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0081] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0082] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0083] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0084] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0085] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0086] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0087] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0088] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0089] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0090] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0091] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0092] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0093] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0094] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order.

[0095] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0096] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0097] 10 system, 20 network, 30 bag, 32 owner, 100 server, 102 memory unit, 104 registration unit, 106 captured image acquisition unit, 108 actual measurement value acquisition unit, 110 weight estimation unit, 112 judgment unit, 114 warning control unit, 116 instruction control unit, 118 model generation unit, 200 information processing device, 202 memory unit, 204 registration unit, 206 captured image acquisition unit, 208 actual measurement value acquisition unit, 210 weight estimation unit, 212 judgment unit, 214 warning control unit, 216 instruction control unit, 218 model generation unit, 310 camera, 320 weighing scale, 330 display, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 Communication interface, 1224 storage device, 1230 ROM, 1240 input / output chip

Claims

1. An image acquisition unit that acquires a bag image from an imaging device of a baggage inspection facility, the bag image being captured by the imaging device of the baggage inspection facility; a weight estimation unit that recognizes a plurality of objects in the bag based on the bag image and estimates a weight of the bag including the plurality of objects; an actual measurement value acquisition unit that acquires an actual measurement value obtained by measuring the weight of the bag using a weighing scale included in the inspection equipment from the weighing scale; a determination unit that determines whether or not a difference between the estimated weight estimated by the weight estimation unit and the actual measured value is greater than a predetermined difference threshold; a warning control unit that performs control so as not to output a warning when it is determined that the difference between the estimated weight and the actual measured value is equal to or smaller than the difference threshold value, and to output a warning when it is determined that the difference is larger than the difference threshold value; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the captured image acquisition section acquires, from the imaging device, a bag image obtained by imaging the bag placed on the weighing scale.

3. the captured image acquisition unit acquires an image of a person carrying the bag placing the bag on the weighing scale, The image processing device according to claim 2 , wherein the determination unit determines whether the difference between the estimated weight and the actual measured value is greater than the difference threshold value, the lower the value the higher the suspiciousness of the holder estimated based on the holder image.

4. The image processing device according to claim 3 , wherein the determination unit estimates the suspiciousness degree of the holder by inputting the holder image into a learning model that estimates the suspiciousness degree of a person from an image of the person.

5. The image processing device according to claim 4 , wherein the determination unit inputs the holder image to the learning model generated by machine learning using an image including a person as a subject and suspiciousness level information of the person as training data.

6. The image processing device according to claim 1 , wherein the warning control unit performs control so as to output the warning with a higher warning level as the difference between the estimated weight and the actual measured value increases.

7. 7. The image processing device according to claim 1, wherein the weight estimation unit recognizes the plurality of objects in the bag by inputting the bag image into a learning model that estimates objects contained in the image from the image, and estimates the weight of the bag including the plurality of objects by referring to weight data in which the weights of each of a plurality of types of objects are registered.

8. The image processing device according to claim 7 , wherein the weight estimation unit inputs the bag image to the learning model generated by machine learning using an image including an object as a subject and object identification information capable of identifying the object as training data.

9. The image processing device according to claim 1 , wherein the determination unit determines whether or not the difference between the estimated weight and the actual measured value is greater than the difference threshold value, the greater the value of the difference threshold value being, the greater the size of the bag.

10. the determination unit determines whether or not a difference between the estimated weight and the actual measured value is greater than the difference threshold value according to a shape of the bag; The image processing device according to claim 1 , wherein the difference threshold value corresponding to a shape that widens toward the bottom is a lower value than the difference threshold value corresponding to a shape that does not widen toward the bottom.

11. an instruction control unit that performs control so as to output instruction information to a user of the bag when a difference between the estimated weight and the actual measured value is greater than an instruction threshold that is higher than the difference threshold; The image processing device according to claim 1 , further comprising:

12. An image processing device according to any one of claims 1 to 11, the imaging device for imaging the bag; A system comprising:

13. A program for causing a computer to function as the image processing device according to any one of claims 1 to 11.

14. 1. A computer-implemented image processing method, comprising the steps of: an image acquisition step of acquiring, from an imaging device of the baggage inspection equipment, an image of the baggage that is opened by the imaging device; a weight estimation step of recognizing a plurality of objects in the bag based on the bag image and estimating a weight of the bag including the plurality of objects; an actual measurement value acquisition step of acquiring an actual measurement value of the weight of the bag from the weighing scale included in the inspection equipment; a determination step of determining whether or not a difference between the estimated weight estimated in the weight estimation step and the actual measured value is greater than a predetermined difference threshold value; a warning control step of controlling not to output a warning when it is determined that the difference between the estimated weight and the actual measured value is equal to or smaller than the difference threshold, and to output a warning when it is determined that the difference is larger than the difference threshold; An image processing method comprising:

Images