Image Processing Device

Abstract

In this image processing device, a registered object image storage unit associates one or more images of an object with an object identification number uniquely identifying the object, and stores the images as registered object images. An unknown object image acquisition unit acquires an unknown object image, which is an image of an unknown object. An object re-identification unit re-identifies the unknown object image using the registered object images stored in the registered object image storage unit. If an unknown object image that has been successfully re-identified is different from all registered object images that are stored in the registered object image storage unit and that are associated with the object identification number of the object in the unknown object image, then a registered object image update unit associates the unknown object image with the object identification number and adds the unknown object image to the registered object image storage unit as a new registered object image.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a recording medium. [Background technology]

[0002] A technology that uses multiple images of an object taken at different times as input and finds the same object among them is called a re-identification technology. Here, the object is a person or an object. The object includes a person's belongings such as a backpack or a bag, moving objects such as a car, a motorcycle, a boat, and an airplane, animals other than people such as dogs and cats, and other objects other than people. For example, Patent Document 1 describes the re-identification technology. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special table 2018-506788 Summary of the Invention [Problem to be solved by the invention]

[0004] Re-identification of an unknown object is performed using images of known objects (registered object images) registered in advance. For example, if features are extracted from both an image of an unknown object and an image of a known object and compared with each other, and if they match, the unknown object is determined to be the same object (same individual) as the known object. At this time, the unknown object is assigned an object identification number (object ID) that was assigned to the known object. If only one registered object image is saved for each known object, re-identification cannot be performed unless the image of the unknown object matches the only registered object image. On the other hand, if various registered object images are saved for each known object, the image of the unknown object only needs to match any one of the multiple registered object images, improving the accuracy of re-identification. However, if multiple registered object images are saved randomly for a known object, the number of registered object images increases but the variation of the registered object images does not increase, and the number of re-identifications increases, decreasing the processing speed of re-identification, but not improving the accuracy of re-identification.

[0005] The present invention provides an image processing device that solves the above-mentioned problems. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present invention includes: In the image processing device, a registered object image storage unit that stores one or more images of an object as registered object images in association with an object identification number that uniquely identifies the object; an unknown object image acquisition unit that acquires an unknown object image that is an image of an unknown object; an object re-identification unit that uses the registered object image stored in the registered object image storage unit to re-identify the acquired unknown object image; a registered object image updating unit that adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number if a condition is met that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified by the object re-identification unit are different from the unknown object image; The device is configured to include:

[0007] Further, an image processing method according to an aspect of the present invention includes the steps of: 1. An image processing method comprising: Obtain an unknown object image, which is an image of an unknown object; re-identifying the acquired unknown object image using a registered object image stored in a registered object image storage unit that stores one or more images of the object as registered object images in association with an object identification number that uniquely identifies the object; if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified are different from the unknown object image, the unknown object image is added to the registered object image storage unit as a new registered object image in association with the object identification number. It is structured as follows.

[0008] A computer-readable recording medium according to one embodiment of the present invention includes: On the computer, A process of acquiring an unknown object image, which is an image of an unknown object; a process of re-identifying the acquired unknown object image using a registered object image stored in a registered object image storage unit that stores one or more images of the object as registered object images in association with an object identification number that uniquely identifies the object; a process of adding the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified are different from the unknown object image; The recording medium is configured to record a program for causing the recording medium to perform the above steps. Effect of the Invention

[0009] By having the above-described configuration, the present invention can increase the variation of registered object images and improve the accuracy of re-identification without unnecessarily increasing the number of registered object images and therefore without reducing the processing speed of re-identification. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing an example of the operation of the image processing device according to the first embodiment of the present invention. [Diagram 3] 2 is a diagram showing an example of a format of information stored in a registered object image storage unit in the image processing device according to the first embodiment of the present invention. FIG. [Figure 4] 5 is a flowchart showing an example of processing by a registered object image update unit in the image processing device according to the first embodiment of the present invention. [Diagram 5] 4 is a diagram showing another example of the format of information stored in a registered object image storage unit in the image processing device according to the first embodiment of the present invention. FIG. [Figure 6] 5 is a flowchart showing an example of processing by a registered object image update unit in the image processing device according to the first embodiment of the present invention. [Figure 7] 4 is a schematic diagram for explaining an example of a method for selecting a registered object image to be used for re-identification by an object re-identification unit in the image processing device according to the first embodiment of the present invention. FIG. [Figure 8]1 is a schematic diagram showing an example of a registered object image stored in a registered object image storage unit in an image processing device according to a first embodiment of the present invention. FIG. [Figure 9] 1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Next, an embodiment of the present invention will be described in detail with reference to the drawings. [First embodiment] FIG. 1 is a block diagram of an image processing device 100 according to a first embodiment of the present invention. The image processing device 100 is configured to re-identify an object. Here, the object may be a person. When the object is a person, the image processing device 100 has a function to re-identify the person. In addition, the object may be an object other than a person. When the object is an object other than a person, the image processing device 100 has a function to re-identify the object other than a person. In addition, the object may include both a person and an object other than a person. When the object includes both a person and an object other than a person, the image processing device 100 has both a function to re-identify the person and a function to re-identify the object other than a person. Referring to FIG. 1, the image processing device 100 is configured to include a camera I / F (interface) unit 110, a communication I / F unit 120, an operation input unit 130, a screen display unit 140, a storage unit 150, and an arithmetic processing unit 160.

[0012] The camera I / F unit 110 is connected to the image server 170 by wire or wirelessly, and is configured to transmit and receive data between the image server 170 and the arithmetic processing unit 160. The image server 170 is connected to the camera 175 by wire or wirelessly, and is configured to accumulate a plurality of images taken by the camera 175 at different shooting times for a certain period of time in the past. The camera 175 may be, for example, a color camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor having a pixel capacity of about several million pixels. The camera 175 may be a camera installed on a street where many people and things pass by, indoors, etc., for the purpose of crime prevention and surveillance. The camera 175 may also be a camera that captures the same or different shooting areas from a fixed location in a fixed shooting direction. Alternatively, the camera 175 may be a camera that is mounted on a moving object such as a car and captures the same or different shooting areas while moving. The camera 175 is not limited to one camera, and may be a plurality of cameras that capture different shooting areas from different locations.

[0013] The communication I / F unit 120 is composed of a data communication circuit, and is configured to perform data communication with an external device (not shown) by wire or wirelessly. The operation input unit 130 is composed of an operation input device such as a keyboard or a mouse, and is configured to detect an operator's operation and output it to the arithmetic processing unit 160. The screen display unit 140 is composed of a screen display device such as an LCD (Liquid Crystal Display), and is configured to display various information on the screen in response to instructions from the arithmetic processing unit 160.

[0014] The storage unit 150 is composed of a storage device such as a hard disk or a memory, and is configured to store processing information and a program 1501 required for various processes in the arithmetic processing unit 160. The program 1501 is a program that is read into the arithmetic processing unit 160 and executed to realize various processing units, and is read in advance from an external device or recording medium (not shown) via a data input / output function such as the communication I / F unit 120 and stored in the storage unit 150. Main processing information stored in the storage unit 150 includes a registered object image storage unit 1502 and object re-identification information 1503.

[0015] The registered object image storage unit 1502 is configured to store one or more registered object images, which are images of an object (individual), in association with an object identification number that uniquely identifies the object.

[0016] The object re-identification information 1503 is information that indicates the result of object re-identification. The object re-identification information 1503 includes, for example, information that identifies a frame image (for example, a camera ID and a shooting time), an object image for each object detected from the frame image and its position information, and an object identification number for each re-identified object. The object re-identification information 1503 is also called an object re-identification information storage unit.

[0017] The arithmetic processing unit 160 has a processor such as an MPU and its peripheral circuits, and is configured to read and execute a program 1501 from the storage unit 150, thereby realizing various processing units by making the above hardware and the program 1501 work together. The main processing units realized by the arithmetic processing unit 160 include an unknown object image acquisition unit 1601, an object re-identification unit 1602, and a registered object image update unit 1603.

[0018] The unknown object image acquisition unit 1601 is configured to acquire frame images constituting a video captured by the camera 175 or frame images obtained by downsampling the frame images from the image server 170 via the camera I / F unit 110. The acquired frame images are assigned a camera ID and a shooting time. The shooting time of the frame images differs for each frame. Furthermore, the unknown object image acquisition unit 1601 is configured to detect an unknown object image from the acquired frame images. An unknown object image is, for example, an image inside a circumscribing rectangle of an image of the entire object present in a frame image. For example, in the case of a person, the unknown object image is an image inside a circumscribing rectangle of an image of the person's entire body. Also, in the case of an object, the unknown object image is an image inside a circumscribing rectangle of an image of the entire object.

[0019] The unknown object image acquisition unit 1601 is configured to, for example, input the frame image into a trained learning model that has undergone machine learning to estimate an object image from a frame image, and acquire an object image present in the frame image as an unknown object image from the learning model. The learning model can be generated in advance, for example, by machine learning using a machine learning algorithm such as a neural network using various frame images and various object images in the frame images as teacher data. However, the method of detecting an unknown object image from a frame image is not limited to the above, and a method such as pattern matching may also be used.

[0020] The unknown object image acquisition unit 1601 is configured to add, for example, a tentative object ID, a camera ID, a shooting time, and position information of the unknown object image on a frame image to the detected unknown object image, and transmit the added information to the object re-identification unit 1602. The position information of the unknown object image may be, for example, coordinate information that identifies a circumscribing rectangle of the unknown object image.

[0021] The object re-identification unit 1602 is configured to use the registered object image stored in the registered object image storage unit 1502 of the storage unit 150 to re-identify the unknown object image transmitted from the unknown object image acquisition unit 1601 .

[0022] The object re-identification unit 1602 is configured to input the unknown object image and one of the registered object images to a trained first learning model that has performed machine learning to estimate whether or not two object images are object images related to the same object, and acquire an estimation result with a probability of whether or not the object images are related to the same object from the first learning model. The first learning model can be generated in advance by machine learning using a machine learning algorithm such as a neural network, using various object image pairs related to the same object and various object image pairs related to different objects as teacher data. For example, if the probability is equal to or greater than a threshold probability set in advance, the object re-identification unit 1602 determines that the unknown object image is the same object as the registered object image. Also, if the probability is less than a threshold probability set in advance, the object re-identification unit 1602 determines that the unknown object image is not the same object as the registered object image. In this case, if another registered object image is stored in the registered object image storage unit 1502, the re-identification of the unknown object image is repeated again using the other registered object image. If an estimation result with a probability equal to or higher than the threshold value is not obtained even when all or a predetermined range of registered object images are used to re-identify the unknown object image, the object re-identification unit 1602 determines that the unknown object image is a newly appeared object, and assigns a new object identification number to the object. The object re-identification unit 1602 also stores the unknown object image in the registered object image storage unit 1502 as a registered object image of a newly appeared object in association with the object identification number.

[0023] In the above, the image is directly input to the learning model to perform object re-identification, but the feature amount suitable for object re-identification may be extracted from the image, and the feature amount may be input to the learning model to perform object re-identification. The method of performing object re-identification is not limited to the above, and any other method may be used, such as a method of determining whether or not the distance between feature vectors extracted from two object images is equal to or less than a predetermined distance. The object re-identification unit 1602 may calculate the above probability between each of one or more registered object images stored in association with the object identification number and the unknown object image, and calculate a statistical value (for example, an average value, a maximum value) for each object identification number, and if the statistical value is equal to or greater than a threshold value, determine that the unknown object image is the same object as the registered object image.

[0024] The object re-identification unit 1602 creates object re-identification information 1503 representing the result of object re-identification performed on the unknown object image, and stores it in the storage unit 150. The object re-identification information 1503 stored in the storage unit 150 may be used, for example, to select a registered object image to be used for re-identification when the object re-identification unit 1602 re-identifies an object image in a new frame image. Alternatively, the object re-identification information 1503 may be displayed on the screen display unit 140 and / or transmitted to an external device via the communication I / F unit 120. Alternatively, the object re-identification information 1503 may be input data for an application program that monitors the behavior of people and objects. Examples of behavior monitoring of people and objects include monitoring of suspicious people and suspicious objects (e.g., abandonment, taking away, switching, shoplifting).

[0025] Furthermore, when the object re-identification unit 1602 is successful in re-identifying an unknown object image, it is configured to transmit re-identification success information including the object identification number of the re-identified object and the unknown object image to the registered object image update unit 1603. Furthermore, when the object re-identification unit 1602 fails to re-identify an unknown object image, that is, when there is no registered object image that matches the unknown object image, it is configured to transmit re-identification failure information including the unknown object image that could not be re-identified to the registered object image update unit 1603.

[0026] The registered object image update unit 1603 updates the registered object image storage unit 1502 based on the re-identification failure information and re-identification success information received from the object re-identification unit 1602. Specifically, when the registered object image update unit 1603 receives re-identification failure information from the object re-identification unit 1602, it registers the unknown object image in the registered object image storage unit 1502 as a registered object image in association with the object identification number assigned to the object related to the unknown object image that could not be re-identified.

[0027] On the other hand, when the registered object image update unit 1603 receives the re-identification success information from the object re-identification unit 1602, first, it reads out all registered object images stored in the registered object image storage unit 1502 in association with the object identification number of the object related to the unknown object image that has been successfully re-identified. Next, the registered object image update unit 1603 checks whether or not the unknown object image and the registered object image match each other for each pair. For example, the registered object image update unit 1603 calculates the similarity between the unknown object image and the registered object image, and if the calculated similarity is equal to or less than a preset threshold, it determines that the unknown object image and the registered object image match each other, and otherwise determines that the unknown object image and the registered object image do not match each other. Next, if the registered object image update unit 1603 determines that the unknown object image and the registered object image do not match each other for all pairs of the unknown object image and the registered object image, it adds the unknown object image to the registered object image storage unit 1502 as a new registered object image in association with the object identification number of the object related to the unknown object image. At this time, the registered object images already registered in the registered object image storage unit 1502 are not deleted but are kept stored. This adds one more variation of the registered object image that can be used to re-identify the object.

[0028] Furthermore, when at least one pair of an unknown object image and a registered object image matches, the registered object image update unit 1603 performs the following process (A) or (B).

[0029] (A) The registered object image update unit 1603 deletes one of the registered object images that matches the unknown object image from the registered object image storage unit 1502, and instead adds the unknown object image as a new registered object image to the registered object image storage unit 1502. As a result, the variation of the registered object image that can be used to re-identify the object does not change, but the registered object image is replaced with a more up-to-date one. (B) The registered object image update unit 1603 neither adds an unknown object image to the registered object image storage unit 1502 nor deletes a registered object image from the registered object image storage unit 1502. This makes it possible to prevent the number of registered object images from increasing unnecessarily, as occurs when unknown object images are added to the registered object image storage unit 1502 at random.

[0030] Next, a description will be given of the operation of the image processing device 100. FIG.

[0031] Referring to FIG. 2, first, the unknown object image acquisition unit 1601 acquires a frame image captured by the camera 175 from the image server 170 via the camera I / F unit 110, and acquires an unknown object image from the frame image (step S1).

[0032] Next, the object re-identification unit 1602 uses one or more registered object images stored in the registered object image storage unit 1502 to re-identify the acquired unknown object image (step S2).

[0033] Next, the registered object image update unit 1603 updates the registered object image storage unit 1502 based on the result of re-identification of the unknown object image by the object re-identification unit 1602 (step S3). Specifically, if re-identification fails, the registered object image update unit 1603 registers the unknown object image as a registered object image in the registered object image storage unit 1502 in association with the object identification number assigned to the object related to the unknown object image that could not be re-identified. If object re-identification is successful, the registered object image update unit 1603 checks whether or not all pairs of registered object images and unknown object images stored in the registered object image storage unit 1502 in association with the object identification number of the object related to the unknown object image that was successfully re-identified match each other, and if they do not match for all pairs, adds the unknown object image to the registered object image storage unit 1502 as a new registered object image in association with the object identification number of the object related to the unknown object image. On the other hand, when at least one pair of an unknown person image and a registered object image matches, the registered object image update unit 1603 performs the above-mentioned process (A) or (B).

[0034] After completing the process of step S3, the image processing device 100 returns to step S1 and repeats the same process as described above. As a result, if another unknown object image exists in the frame image captured by the camera 175, the acquisition of the unknown object image (step S1), re-identification of the unknown object image (step S2), and update of the registered object image storage unit 1502 based on the re-identification result (step S3) are repeated for each unknown object image. Furthermore, when the processes of steps S1 to S3 are completed for all unknown object images existing in one frame image, the next frame image is acquired, and the same process as for the previous frame image is repeated.

[0035] In this way, according to the image processing device 100 of this embodiment, it is possible to increase the variation of registered object images and improve the accuracy of re-identification without unnecessarily increasing the number of registered object images and therefore without decreasing the processing speed of re-identification. The reason is that when object re-identification is successful, the registered object image update unit 1603 checks whether or not all pairs of registered object images and unknown object images stored in the registered object image storage unit 1502 in association with the object identification number of the object related to the unknown object image that has been successfully re-identified match each other, and when they do not match for all pairs, adds the unknown object image to the registered object image storage unit 1502 as a new registered object image in association with the object identification number of the object related to the unknown object image.

[0036] Next, the registered object image update unit 1603 will be described in more detail.

[0037] There are two possible methods for registering multiple different images of the same object. The first method is to determine the conditions for the multiple different images in advance, and to determine whether a newly acquired image meets one of the conditions that the already registered images do not meet. The second method is to register the first acquired image unconditionally, and to determine whether the second and subsequent acquired images are exceptional images that do not belong to any of one or more clusters obtained by clustering one or more already registered images using a one-class discrimination plane, and to decide whether or not to register the images according to the result of the determination.

[0038] Various conditions can be considered when the registered object image update unit 1603 uses the first method. For example, a plurality of different postures of the object may be used as the condition. Alternatively, the presence or absence or type of attachments (belongings) of the object may be used as the condition. For example, in the case of a person, a person with no belongings, a person with a backpack as belongings, a person with a handbag as belongings, etc. may be used as the condition for a plurality of different images.

[0039] Moreover, when the registered object image update unit 1603 uses the second method, it is possible to extract a feature amount determined in advance from the image and cluster the images according to the similarity of the extracted feature amount. Various feature amounts are possible. For example, it may be a feature amount representing the posture of the object, or a feature amount representing the presence or absence and type of an accessory of the object.

[0040] Next, a specific example of the registered object image updating unit 1603 will be described.

[0041] <Example 1 of the registered object image update unit 1603> 3 shows an example of the format of information stored in the registered object image storage unit 1502 in association with a registered object number that uniquely identifies an object. Referring to FIG. 3, the registered object image storage unit 1502 has pairs of object identification number 171 and registered object image information 172 for the number of registered objects. A registered object number that uniquely identifies a registered object is set in the object identification number 171 field. Information on a registered object image used for re-identifying a registered object identified by the registered object number 171 is set in the registered object image information 172 field. In the example shown in FIG. 3, the registered object image information 172 has m sets of a posture number 1721, a registered object image 1722, and a property 1723.

[0042] In the item of posture number 1721, a number that determines one posture when the posture of the object is classified into m types is set in advance. In the item of registered object image 1722, a registered object image that matches the posture determined by the posture number set in posture number 1721 is set. In the item of property 1723, a camera ID and a shooting time that were added to a frame image that included the registered object image set in registered object image 1722 are set. If there is no registered object image that corresponds to posture number 1721, the items of registered object image 1722 and property 1723 are, for example, NULL values. In this example, the number of registered object images registered per posture number is one. However, in another embodiment, the number of registered object images registered per posture number may be two or more.

[0043] There are various methods for detecting the orientation of an object from an object image. Below, an example of detecting the orientation of an object using skeletal information of the object will be described. This method can be applied when the object is a person. In addition, even if the object is not a person, it can be applied to living creatures with skeletons such as dogs and cats.

[0044] First, the registered object image update unit 1603 detects skeletal information of an object from an object image. For example, skeletal information of a person includes information indicating the positions of joints constituting a human body. The joints may include not only joints such as the neck and shoulders, but also facial parts such as the eyes and nose. The skeletal information may include the reliability (likelihood of feature recognition) of each joint in addition to the position of the joint. For example, the registered object image update unit 1603 may be realized using a system that estimates skeletal information of an object in an image by deep learning. Examples of such systems include OpenPose, HRNet, and AlphaPose. For example, OpenPose extracts the positions (x, y coordinates) and reliability of a total of 18 joints, including the nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, right ear, and left ear. However, the method of detecting skeletal information of a person from an object image is not limited to the above, and any method can be used.

[0045] Next, the registered object image update unit 1603 determines the orientation of the object from the detected skeleton information. The method of determining the orientation of the object from the skeleton information may be, for example, the following method.

[0046] First, the registered object image update unit 1603 acquires position information of a pair of joints that exist symmetrically in advance from the skeleton information. There are several pairs of joints that exist symmetrically in advance, but in the case of a person, it is desirable to acquire position information of a pair of joints of the left and right shoulders that have a large influence on the posture. Next, the registered object image update unit 1603 calculates the direction of a normal line of a vector from one of the pairs (for example, the left shoulder joint) to the other (for example, the right shoulder joint). Since there are two directions perpendicular to the vector, one of the directions that is determined in advance is set as the direction of the normal line. Next, the registered object image update unit 1603 determines which of the reference directions corresponding to m types of posture numbers that are determined in advance corresponds to the above-mentioned normal line direction. For example, when m=8, the reference directions corresponding to m types of posture numbers may be 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. Furthermore, the registered object image update unit 1603 calculates the angle of difference between the determined normal direction and each reference direction, and determines the orientation number corresponding to the reference direction in which the absolute value of the angle of difference is the smallest as the orientation of the object. For example, if the normal direction is 5°, it is determined that the orientation is the orientation number corresponding to the reference direction of 0°. However, the method of determining the orientation of the object from the skeleton information and the number of types of orientation are not limited to the above.

[0047] FIG. 4 is a flowchart showing an example of processing by the registered object image update unit 1603. Referring to FIG. 4, the registered object image update unit 1603 first determines the posture of the unknown object from the unknown object image included in the re-identification success information or re-identification failure information transmitted from the object re-identification unit 1602 (step S11). Next, the registered object image update unit 1603 determines whether the object re-identification has been successful or unsuccessful (step S12). Next, if the object re-identification has been unsuccessful, the registered object image update unit 1603 creates a set of a new object identification number 171 and registered object image information 172 for the current unknown object in the registered object image storage unit 1502 (step S13). Specifically, the registered object image update unit 1603 assigns a new object identification number to the current object, and sets the assigned object identification number in the item of the object identification number 171 newly secured in the registered object image storage unit 1502. Furthermore, the registered object image update unit 1603 sets the current unknown object image and its properties in the items of registered object image 1722 and property 1723 in the same set as the orientation number 1721 that matches the orientation detected from the object image in the newly acquired registered object image information 172. Then, the registered object image update unit 1603 ends the processing in FIG. 4.

[0048] On the other hand, if the object re-identification is successful, the registered object image update unit 1603 focuses on one of the registered object images 1722 registered in the registered object image storage unit 1502 in association with the re-identified object identification number (step S14). Next, the registered object image update unit 1603 determines the orientation of the registered object from the registered object image 1722 being focused on (step S15) and compares it with the orientation of the unknown object image determined in step S11 (step S16). Next, if the orientations of the two match (YES in step S17), the registered object image update unit 1603 overwrites the registered object image 1722 being focused on stored in the registered object image storage unit 1502 with the unknown object image (step S18). At this time, the corresponding property 1723 is overwritten with that of the unknown object image. Then, the registered object image update unit 1603 ends the process of FIG. 4.

[0049] Furthermore, if the postures of the two do not match (NO in step S17), the registered object image update unit 1603 shifts attention to the next registered object image 1722 associated with the respecified object identification number (step S19), returns to step S15 via step S20, and repeats the same process as described above for the registered object image 1722 to which attention has been shifted. Then, when all registered object images 1722 stored in the registered object image storage unit 1502 in association with the respecified object identification number have been processed (i.e., if the postures do not match with all registered object images) (YES in step S20), the registered object image update unit 1603 executes step S21. In step S21, the registered object image update unit 1603 registers a pair of the unknown object image and its property in the registered object image storage unit 1502 as a pair of the registered object image 1722 and property 1723 of the posture number corresponding to the posture. Then, the registered object image update unit 1603 ends the process of FIG. 4.

[0050] <Example 2 of the registered object image update unit 1603>

[0051] FIG. 5 shows another example of the format of information stored in the registered object image storage unit 1502 in association with a registered object number that uniquely identifies an object. Referring to FIG. 5, the registered object image storage unit 1502 has pairs of object identification number 173 and registered object image information 174 for the number of registered objects. In the item of object identification number 173, a registered object number that uniquely identifies a registered object is set. In the item of registered object image information 174, information of a registered object image used for re-identifying a registered object identified by the registered object number 173 is set. In the example shown in FIG. 5, the registered object image information 174 includes one or more pairs of a registered object image 1741 and a property 1742. Here, an upper limit may be set for the number of registered object images to be registered per one object identification number, or it may not be set.

[0052] An object image of a registered object is set in the item of registered object image 1741. In the item of property 1742, a camera ID and a shooting time etc. added to a frame image including the object image set in registered object image 1741 are set.

[0053] FIG. 6 is a flowchart showing an example of processing by the registered object image update unit 1603. Referring to FIG. 6, the registered object image update unit 1603 first detects skeleton information of an unknown object from an unknown object image included in the re-identification success information or re-identification failure information transmitted from the object re-identification unit 1602 (step S31). Next, the registered object image update unit 1603 determines whether the object re-identification has been successful or unsuccessful (step S32). Next, if the object re-identification has been unsuccessful, the registered object image update unit 1603 creates a set of a new object identification number 173 and registered object image information 174 for the current unknown object in the registered object image storage unit 1502 (step S33). Specifically, the registered object image update unit 1603 assigns a new object identification number to the current object, and sets the assigned object identification number in the item of the object identification number 173 newly secured in the registered object image storage unit 1502. Furthermore, the registered object image updating unit 1603 sets a pair of the registered object image 1741 in which the current unknown object image is set and its property 1742 in the newly acquired registered object image information 174. Then, the registered object image updating unit 1603 ends the processing in FIG.

[0054] On the other hand, if the object re-identification is successful, the registered object image update unit 1603 performs clustering on all registered object images 1741 registered in the registered object image storage unit 1502 in association with the re-identified object identification number based on the skeleton information extracted from them (step S34). Next, the registered object image update unit 1603 focuses on one of the clusters obtained by the above clustering (step S35), and creates a one-class discrimination plane for the cluster in focus (step S36). Next, the registered object image update unit 1603 uses the created one-class discrimination plane to determine whether the skeleton information of the unknown object image is of the same class as the cluster in focus (step S37). Next, if the registered object image update unit 1603 determines that they are of the same class (YES in step S38), it overwrites one of the registered object images 1741 belonging to the cluster in focus stored in the registered object image storage unit 1502 with the unknown object image (step S39). At this time, the corresponding property 1742 is overwritten with that of the unknown object image. Then, the registered object image updating unit 1603 ends the process in FIG.

[0055] If the registered object image update unit 1603 determines that they are not in the same class (NO in step S38), it shifts its attention to one of the other clusters generated by clustering (step S40), returns to step S36 via step S41, and repeats the same processing as described above for the cluster to which the attention has been shifted. Then, when the processing for all the generated clusters is completed (i.e., when the joint information of the unknown object image does not belong to any cluster) (YES in step S41), it executes step S42. In step S42, the registered object image update unit 1603 registers the pair of the unknown object image and its property in the registered object image storage unit 1502 as a pair of the registered object image 1741 and property 1742 corresponding to the re-specified object identification number 173. Then, the registered object image update unit 1603 ends the processing of FIG. 6.

[0056] Next, the object re-identification unit 1602 will be described in more detail.

[0057] FIG. 7 is a schematic diagram for explaining an example of a method for selecting a registered object image used by the object re-identification unit 1602 for re-identification. In FIG. 7, a frame image 181 is a frame image that the object re-identification unit 1602 will perform re-identification from now on, and includes three unknown object images X1 to X3 detected by the unknown object image acquisition unit 1601. Meanwhile, frame images 182, 183, 184, and 185 are frame images one, two, three, and four before the frame image 181. The frame images 182 to 185 include three object images, which are re-identified by the object re-identification unit 1602. That is, one of the three object images is re-identified as person A, another as person B, and the remaining one as person C. These re-identification information are included in the object re-identification information 1503. In FIG. 7, the images of person A in each frame are distinguished as A1, A2, A3, and A4. Similarly, the images of person B for each frame are distinguished as B1, B2, B3, and B4, and the images of person C for each frame are distinguished as C1, C2, C3, and C4, respectively.

[0058] 7, the angles added to each image represent the posture of the person in the image. The posture of image A1 of person A is 0°, and the postures of images A2, A3, and A4 are the same 90°. Therefore, images A4 and A1 are registered in the registered object image storage unit 1502 in association with the object identification number of person A as shown in FIG. 8, and images A2 and A3 are not registered. Similarly, images B4, B2, B1 and images C4 and C2 are registered in the registered object image storage unit 1502 in association with the object identification numbers of person B and person C as shown in FIG. 8, and images B3, C1, and C3 are not registered.

[0059] Now, when the object re-identification unit 1602 re-identifies the unknown object image X1 in the latest frame image 181, the object re-identification unit 1602 uses the registered object image shown in FIG. 8 stored in the registered object image storage unit 1502 in association with the object identification numbers of the persons A, B, and C who have been successfully re-identified in one or more of the most recent frame images 182 to 185 to re-identify the unknown object image X1. That is, the object re-identification unit 1602 calculates the probability that the unknown object image X1 is the same person as the image A4, for example, using a trained learning model, and if the probability is equal to or greater than a threshold, determines that the unknown object image X1 is person A. If the probability is less than the threshold, the object re-identification unit 1602 calculates the probability that the unknown object image X1 is the same person as the image A1, and if the probability is equal to or greater than a threshold, determines that the unknown object image X1 is person A. Furthermore, if the probability is less than the threshold, the remaining registered images B4, B2, B1, C4, and C2 are used in order to attempt to re-identify the unknown object image X1. In this way, when re-identifying one unknown object image X1 in the latest frame image 181 with a registered object image of a person re-identified in the most recent four frames, in this embodiment, it is possible to limit the number of registered object images to a total of seven (A1, A4, B1, B2, B4, C2, and C4). In contrast, in a configuration in which the image of the person re-identified in the most recent four frames is used as the registered object image as it is, the unknown object image X1 will be re-identified with a total of 12 registered object images, which will reduce the processing speed of re-identification.

[0060] Furthermore, the object re-identification unit 1602 may preferentially use a registered object image having a posture matching the posture of the unknown object detected from the unknown object image, among one or more registered object images stored in the registered object image storage unit 1502, to re-identify the unknown object image. That is, in the example of the unknown object image X1 described above, a posture of 0° may be detected from the unknown object image X1, and images A1, B4, and C2 having the same posture of 0° may be selected from the registered object image storage unit 1502, and these three images may first be used in order to re-identify the unknown object image X1. This is expected to reduce the number of re-identifications required to successfully re-identify the unknown object image X1, and to further increase the processing speed of re-identification.

[0061] [Second embodiment] Next, a second embodiment of the present invention will be described with reference to the drawings. Fig. 9 is a block diagram of an image processing device in this embodiment. In this embodiment, an outline of the image processing device of the present invention will be described.

[0062] 9, an image processing device 200 according to this embodiment includes a registered object image storage unit 201, an unknown object image acquisition unit 202, an object re-identification unit 203, and a registered object image update unit 204.

[0063] The registered object image storage unit 201 is configured to store one or more images of an object as a registered object image in association with an object identification number that uniquely identifies the object. Here, the object may be a person. Alternatively, the object may be something other than a person. The registered object image storage unit 201 may be configured similarly to, for example, the registered object image storage unit 1502 in FIG. 1, but is not limited thereto.

[0064] The unknown object image acquisition unit 202 is configured to acquire an unknown object image, which is an image of an unknown object. The source of the unknown object image may be, for example, a camera image. The unknown object image acquisition unit 202 may be configured in the same manner as, for example, the unknown object image acquisition unit 1601 in FIG. 1, but is not limited thereto.

[0065] The object re-identification unit 203 is configured to re-identify the unknown object image acquired by the unknown object image acquisition unit 202, using the registered object image stored in the registered object image storage unit 201. The object re-identification unit 203 can be configured similarly to, for example, the object re-identification unit 1602 in Fig. 1, but is not limited thereto.

[0066] The registered object image update unit 204 is configured to add the unknown object image to the registered object image storage unit 201 as a new registered object image in association with the object identification number if a condition is met that all registered object images stored in the registered object image storage unit 201 in association with the object identification number of an object related to an unknown object image successfully re-identified by the object re-identification unit 203 are different from the unknown object image. The registered object image update unit 204 can be configured in the same manner as the registered object image update unit 204 in Fig. 1, for example, but is not limited to this.

[0067] The image processing device 200 configured in this manner operates as follows. That is, the registered object image storage unit 201 stores one or more images of an object as registered object images in association with an object identification number that uniquely identifies the object. Under this condition, the unknown object image acquisition unit 202 acquires an unknown object image that is an image of an unknown object. Next, the object re-identification unit 203 uses the registered object image stored in the registered object image storage unit 201 to re-identify the unknown object image acquired by the unknown object image acquisition unit 202. Next, if a condition is established that all registered object images stored in the registered object image storage unit 201 in association with the object identification number of the object related to the unknown object image successfully re-identified by the object re-identification unit 203 are different from the unknown object image, the registered object image update unit 204 adds the unknown object image to the registered object image storage unit 201 as a new registered object image in association with the object identification number, if the condition is satisfied.

[0068] According to the image processing device 200 configured and operated as described above, it is possible to increase the variation of registered object images and improve the accuracy of re-identification without unnecessarily increasing the number of registered object images and therefore without decreasing the processing speed of re-identification. The reason is that, if a condition is established that all registered object images stored in the registered object image storage unit 201 in association with the object identification numbers of objects related to an unknown object image successfully re-identified by the object re-identification unit 203 are different from the unknown object image, the registered object image update unit 204 adds the unknown object image to the registered object image storage unit 201 as a new registered object image in association with the object identification number.

[0069] Although the present invention has been described with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Industrial Applicability]

[0070] The present invention can be used to re-identify a person and / or an object such as a person's belongings, and can also be used in technology to detect events such as a person leaving behind, taking away, or swapping belongings.

[0071] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. [Appendix 1] In the image processing device, a registered object image storage unit that stores one or more images of an object as registered object images in association with an object identification number that uniquely identifies the object; an unknown object image acquisition unit that acquires an unknown object image that is an image of an unknown object; an object re-identification unit that uses the registered object image stored in the registered object image storage unit to re-identify the acquired unknown object image; a registered object image updating unit that adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number if a condition is met that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified by the object re-identification unit are different from the unknown object image; An image processing device comprising: [Appendix 2] if the condition is not satisfied, the registered object image update unit deletes the registered object image that matches the unknown object image from the registered object image storage unit, and adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number. 2. An image processing device as described in claim 1. [Appendix 3] the registered object image update unit, in determining whether the condition is satisfied, compares a posture of the registered object detected from each of one or more of the registered object images with a posture of the unknown object detected from the unknown object image, and determines that the condition is satisfied if there is no posture of the registered object that matches the posture of the unknown object; 3. An image processing device according to claim 1 or 2. [Appendix 4] the registered object image update unit detects a posture of the registered object based on skeleton information detected from the registered object image, and detects a posture of the unknown object based on skeleton information detected from the unknown object image; 4. The image processing device according to claim 3. [Appendix 5] the object re-identification unit preferentially uses a registered object image having a posture that matches a posture of the unknown object detected from the unknown object image, among one or more registered object images stored in the registered object image storage unit, to re-identify the acquired unknown object image. 5. The image processing device according to claim 3 or 4. [Appendix 6] the registered object image update unit, in determining whether the condition is satisfied, clusters one or more of the registered object images based on a predetermined feature amount, creates a one-class discrimination plane for each cluster obtained by the clustering, and determines whether the unknown object image is in the same class as the cluster using the created one-class discrimination plane; and determines that the condition is satisfied if it is determined that the unknown object image is not in the same class as any of the clusters. 3. An image processing device according to claim 1 or 2. [Appendix 7] the registered object image update unit performs the clustering based on skeleton information detected from the registered object image. 7. The image processing device according to claim 6. [Appendix 8] further comprising an object re-identification information storage unit that stores an object identification number of an object related to the unknown object image in a frame image that has been successfully re-identified by the object re-identification unit in association with a shooting time of the frame image, When re-identifying the unknown object image in the latest frame image, the object re-identification unit re-identifies the unknown object image in the latest frame image by using the registered object image stored in the registered object image storage unit in association with the object identification number of the object that has been successfully re-identified in the most recent frame image stored in the object re-identification information storage unit. 8. An image processing device according to any one of claims 1 to 7. [Appendix 9] 1. An image processing method comprising: Obtain an unknown object image, which is an image of an unknown object; re-identifying the acquired unknown object image using a registered object image stored in a registered object image storage unit that stores one or more images of the object as registered object images in association with an object identification number that uniquely identifies the object; if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified are different from the unknown object image, the unknown object image is added to the registered object image storage unit as a new registered object image in association with the object identification number. Image processing methods. [Appendix 10] On the computer, A process of acquiring an unknown object image, which is an image of an unknown object; a process of re-identifying the acquired unknown object image using a registered object image stored in a registered object image storage unit that stores one or more images of the object as registered object images in association with an object identification number that uniquely identifies the object; a process of adding the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified are different from the unknown object image; A computer-readable recording medium having a program recorded thereon for causing a computer to carry out the above. [Explanation of symbols]

[0072] 100, 200 Image processing device 110 Camera I / F section 120 Communication I / F section 130 Operation input section 140 Screen display section 150 Storage section 160 Processing section 171, 173 Object Identification Number 172, 174 Registered object image information 175 Camera 181~185 Frame images 201 Registered object image storage unit 202 Unknown object image acquisition unit 203 Object re-identification part 204 Registered object image update unit 1501 Program 1502 Registered object image storage unit 1503 Object re-identification information 1601 Unknown object image acquisition unit 1602 Object re-identification part 1603 Registered object image update unit 1721 Posture Number 1722, 1741 Registered object images 1723, 1742 Properties

Claims

1. In the image processing device, a registered object image storage unit that stores one or more images of an object as registered object images in association with an object identification number that uniquely identifies the object; an unknown object image acquisition unit that acquires an unknown object image that is an image of an unknown object; an object re-identification unit that uses the registered object image stored in the registered object image storage unit to re-identify the acquired unknown object image; a registered object image updating unit which, if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of an object related to the unknown object image successfully re-identified by the object re-identification unit are different from the unknown object image, adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number; Equipped with The registered object image update unit, in determining whether the condition is met, clusters one or more of the registered object images based on a predetermined feature, creates a one-class discrimination plane for each cluster obtained by the clustering, and determines whether the unknown object image is of the same class as the cluster using the created one-class discrimination plane, and determines that the condition is met if it is determined that the unknown object image is not of the same class as any of the clusters.

2. if the condition is not satisfied, the registered object image update unit deletes the registered object image that matches the unknown object image from the registered object image storage unit, and adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number. The image processing device according to claim 1 .

3. the registered object image update unit performs the clustering based on skeleton information detected from the registered object image. The image processing device according to claim 1 .

4. further comprising an object re-identification information storage unit that stores an object identification number of an object related to the unknown object image in a frame image that has been successfully re-identified by the object re-identification unit in association with a shooting time of the frame image, When re-identifying the unknown object image in the latest frame image, the object re-identification unit re-identifies the unknown object image in the latest frame image by using the registered object image stored in the registered object image storage unit in association with the object identification number of the object that has been successfully re-identified in the most recent frame image stored in the object re-identification information storage unit.

4. An image processing device according to claim 1.

5. A method for image processing by a computer, comprising: The computer acquires an unknown object image, which is an image of an unknown object; The computer re-identifies the acquired unknown object image using the registered object image stored in a registered object image storage unit that stores one or more images of the object as registered object images in association with an object identification number that uniquely identifies the object; if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified are different from the unknown object image, the computer adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number; In determining whether the condition is satisfied, the computer performs clustering of one or more of the registered object images based on a predetermined feature amount; the computer creates a one-class discrimination plane for each cluster obtained by the clustering; The computer determines whether the unknown object image is in the same class as the cluster using the created one-class discrimination plane; If the computer determines that the unknown object image does not belong to the same class as any of the clusters, it determines that the condition is satisfied. Image processing methods.

6. On the computer, A process of acquiring an unknown object image, which is an image of an unknown object; a process of re-identifying the acquired unknown object image using a registered object image stored in a registered object image storage unit that stores one or more images of the object as registered object images in association with an object identification number that uniquely identifies the object; a process of adding the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number if a condition is satisfied that all of the registered object images stored in the registered object image storage unit in association with the object identification number of the object related to the unknown object image that has been successfully re-identified are different from the unknown object image; Let them do so, A program for determining whether or not the condition is met, which comprises clustering one or more of the registered object images based on predetermined features, creating a one-class discrimination plane for each cluster obtained by the clustering, determining whether or not the unknown object image is in the same class as the cluster using the created one-class discrimination plane, and determining that the condition is met if it is determined that the unknown object image is not in the same class as any of the clusters.

Images