Object recognition system and object recognition program

Abstract

An object recognition system and an object recognition program are provided that can reduce the amount of calculation as much as possible while maintaining accuracy. [Solution] The object recognition system comprises an object detection means for detecting two or more parts of a target object that is captured in a frame image input from a camera, a suitability calculation means for calculating the suitability of the target object as a target for recognition based on the positions and sizes of the two or more parts, a comparison means for comparing the suitability as a target for recognition with a predetermined reference value, and an object recognition means for recognizing only target objects that meet the reference value as a result of the comparison.

Description

[Technical Field]

[0001] The present invention relates to an object recognition system and an object recognition program that can reduce the amount of calculation as much as possible while maintaining accuracy. [Background technology]

[0002] Object recognition systems are used that detect objects such as people in images captured by a fixed camera and output the results of recognizing the attributes of the detected objects. Such object recognition systems use a trained model (hereinafter referred to as the "trained model") that uses a neural network (hereinafter referred to as the NN) that is trained to input target image data and output the detection results of objects that appear in the image data, and a trained model using the NN that is trained to output the attributes of the detected objects.

[0003] Thanks to improvements in processor computing power and hardware technology, it is now possible to process data using learning models even on edge devices with relatively limited computing resources, rather than on a server with abundant computing resources that collects data and performs image processing.

[0004] When deploying an object recognition system with pre-customized conditions for a variety of environments, time is required to confirm the condition settings. Therefore, Patent Document 1 discloses a system that enables remote adjustment of parameters such as the angle of view of images captured by a camera in the object recognition system. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-039504 Summary of the Invention [Problem to be solved by the invention]

[0006] It goes without saying that even servers with abundant computing resources are required to be able to perform detection and recognition with lighter processing and greater accuracy, and there is also a demand for object detection and recognition processing to be lighter and more accurate so that even edge devices with relatively limited computing resources can perform accurate real-time processing.

[0007] An object of the present invention is to provide an object recognition system and an object recognition program that can reduce the amount of calculation as much as possible while maintaining accuracy. [Means for solving the problem]

[0008] An object recognition system according to an embodiment of the present disclosure includes an object detection unit that detects two or more parts of a detection target object captured in a frame image input from a camera, a compatibility calculation unit that calculates the compatibility of the detection target object as a recognition target based on the positions and sizes of the two or more parts, a comparison unit that compares the compatibility as a recognition target with a predetermined reference value, and an object recognition unit that recognizes only the detection target object that satisfies the reference value as a result of the comparison. The two or more parts detected by the object detection means are a person's head and face, the object detection means detects the person's head and face by obtaining a rectangular bounding box for the person's head and face, the rectangular bounding box for the head encompasses the rectangular bounding box for the face, the compatibility calculation means obtains a distance between a predetermined vertex in the rectangular bounding box for the head and a vertex corresponding to the vertex in the rectangular bounding box for the face, the comparison means compares the distance with the reference value, and the object recognition means recognizes only the person to be detected whose distance satisfies the reference value as a result of the comparison. .

[0015] The object recognition system described above uses a rectangle that indicates the area in which an object is captured, which is a common technique used in object detection. Bounding box of Using R The object recognition system Human head and face The corresponding rectangles Bounding box of Depending on the position and size relationship between the people It can be confirmed that the part of the frame image from which the feature amount that can identify the attribute of the object can be calculated is sufficiently or clearly shown in the frame image. is, people A rectangle corresponding to the head and face of an object Bounding box of In the case of comrades, The relative positions of these bounding boxes are If one side (head) contains the other side (face) and the other side (face) is not too close to one side within the range of the other side, it can be confirmed that it is clearly visible. do.

[0026] In the object recognition system according to an embodiment of the present disclosure, an object captured in a frame image input from a camera is Two or more parts of the object to be detected Detect object a detection means; The apparatus includes a suitability calculation means for calculating suitability of the object to be detected as a recognition target based on the positions and sizes of the two or more parts, a comparison means for comparing the suitability as a recognition target with a predetermined reference value, and an object recognition means for recognizing only the object to be detected that satisfies the reference value as a result of the comparison, wherein the two or more parts detected by the object detection means are a human head and face, The aforementioned object a face direction detecting means for detecting the face direction of the face detected by the detecting means; The fitness calculation means further includes: In addition to the position and size of the head and face, the face of the detection target is determined based on the face direction detected by the face direction detection means. recognition Calculate the suitability as a target do.

[0027] An object recognition program according to an embodiment of the present disclosure causes a computer to function as an object detection means for detecting two or more parts of a detection target object captured in a frame image input from a camera, a suitability calculation means for calculating a suitability of the detection target object as a recognition target based on positions and sizes of the two or more parts, a comparison means for comparing the suitability as a recognition target with a predetermined reference value, and an object recognition means for recognizing only the detection target object that satisfies the reference value as a result of the comparison. The two or more parts detected by the object detection means are a person's head and face, the object detection means detects the person's head and face by obtaining a rectangular bounding box for the person's head and face, the rectangular bounding box for the head encompasses the rectangular bounding box for the face, the compatibility calculation means obtains a distance between a predetermined vertex in the rectangular bounding box for the head and a vertex corresponding to the vertex in the rectangular bounding box for the face, the comparison means compares the distance with the reference value, and the object recognition means recognizes only the person to be detected whose distance satisfies the reference value as a result of the comparison. .

[0028] The object recognition program according to an embodiment of the present disclosure is configured to cause a computer to recognize an object captured in a frame image input from a camera. Two or more parts of the object to be detected Detect object a detection means; The computer is configured to function as a suitability calculation means for calculating the suitability of the object to be detected as a recognition target based on the positions and sizes of the two or more parts, a comparison means for comparing the suitability as a recognition target with a predetermined reference value, and an object recognition means for recognizing only the object to be detected that satisfies the reference value as a result of the comparison, and the two or more parts detected by the object detection means are a human head and face, The aforementioned object a face direction detecting means for detecting the face direction of the face detected by the detecting means; and the fitness calculation means further functions as In addition to the position and size of the head and face, the face of the detection target is determined based on the face direction detected by the face direction detection means. recognition Calculate the suitability as a target do. [Effects of the Invention]

[0029] According to the present disclosure, by performing recognition processing only on frame images obtained from a camera in which the detection target and recognition target are clearly visible to the extent that recognition processing of characteristic parts is easy, the amount of calculation is reduced compared to performing recognition processing on all frame images, which requires a lot of computational resources, and high recognition accuracy is maintained. [Brief explanation of the drawings]

[0030] [Figure 1] 1 is a schematic diagram of an object recognition system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a configuration of an edge device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a cloud server. [Figure 4] FIG. 2 is a block diagram showing the configuration of a client; [Figure 5] 10 is a flowchart illustrating an example of a processing procedure for image recognition in an edge device. [Figure 6] FIG. 10 is an explanatory diagram of processing by an edge device. [Figure 7] FIG. 10 is a schematic diagram of an object recognition system according to a second embodiment. [Figure 8] 10 is a flowchart illustrating an example of a model setting process procedure in the object recognition system according to the second embodiment. [Figure 9] 10 is a flowchart illustrating an example of a model setting process procedure in an object recognition system according to a third embodiment. [Figure 10] FIG. 11 is an explanatory diagram of a process performed by an edge device according to the third embodiment. [Figure 11] 10 is a flowchart illustrating an example of a model setting processing procedure in an object recognition system according to a fourth embodiment. [Figure 12] FIG. 11 is an explanatory diagram of a process performed by an edge device according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be specifically described with reference to the drawings illustrating embodiments thereof. In the following embodiments, an object recognition system according to the present disclosure will be described.

[0032] [First embodiment] 1 is a schematic diagram of an object recognition system 100 according to a first embodiment. The object recognition system 100 includes a camera 2, an edge device 1 connected to the camera 2, a cloud server 3 that can connect to the edge device 1 via a network N, and a client 4 that can connect to the cloud server 3.

[0033] The edge device 1 extracts features from image data acquired from the camera 2, detects objects such as a person to be detected from the image based on the features, recognizes attributes of the objects shown in the image data based on the features, and outputs the recognition results, all of which are performed using a neural network-based learning model. The edge device 1 outputs the recognition results of the attributes of the detected objects in text format. In the following description, the edge device 1 is described as one computer for one camera 2, but the edge device 1 may be configured such that multiple computers for one camera 2 share processing for each process, or one or multiple computers may perform processing for multiple cameras 2.

[0034] The camera 2 outputs image data using an image element that is compatible with visible light and / or near-infrared light. The camera 2 outputs image data of frame images in time series at a rate of several fps to several tens of fps.

[0035] The edge device 1 and the camera 2 can be communicatively connected via a signal line or via a wireless or wired communication medium, for example, a coaxial cable, a USB (Universal Serial Bus), a serial bus, a wired LAN, a wireless LAN, or Bluetooth (registered trademark).

[0036] The cloud server 3 is connected to the edge device 1 via the network N and functions as a cloud manager that instructs the edge device 1 on the processing content to be performed. The cloud server 3 functions as a cloud manager for the edge device 1 that is connected to the cameras 2 installed in different spaces. The cloud server 3 performs a manager function that instructs the edge device 1 on the processing content to be performed, such as setting reference values ​​to be referenced in the processing to be described later that is performed by the edge device 1.

[0037] The cloud server 3 acquires the results (text) of the recognition process executed by the edge device 1 in each space for each space and stores them in the database 300 (see FIG. 3). The cloud server 3 may also execute analytical processes such as aggregation processes or statistical processes of attributes related to detected objects for each space and store the results in the database 300. The results of the recognition process stored in the cloud server 3 can be confirmed by an operator using the client 4 to specify data identifying the space or data identifying the edge device 1.

[0038] The network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network.

[0039] In the object recognition system 100 configured in this manner, in order to reduce the processing load executed by the edge device 1 while maintaining high detection accuracy and recognition accuracy in the edge device 1, recognition processing is omitted for frame images that are likely to have low recognition accuracy. Furthermore, the object recognition system 100 receives a reference value setting from the cloud server 3 in order to identify frame images that are likely to have low recognition accuracy.

[0040] Such an object recognition system 100 will be described in detail below.

[0041] 2 is a block diagram showing the configuration of the edge device 1. An edge computer is used as the edge device 1. In the following description, the edge device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0042] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The processing unit 10 includes a memory that is a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 includes a timer and can acquire time information at each point in time from data from the timer. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0043] The processing unit 10 causes the processor to execute image processing based on the image recognition program P1 stored in the storage unit 11 and the learning model deployed from the cloud server 3. The processing unit 10 and the image recognition program P1 mainly correspond to the "fitness calculation means" and the "comparison means" in the claims.

[0044] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk, a flash memory, etc. A part of the storage unit 11 may be removable.

[0045] The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, the results of processing by the processing unit 10, and reference setting data. The setting data includes identification data of the device itself. The program product includes an OS (Operating System) program, an image recognition program P1 that runs on the OS, a learning model group M1, and configuration data. The learning model group M1 will be described in detail later.

[0046] The image recognition program P1 stored in the storage unit 11 may be an image recognition program P9 stored in a computer-readable storage medium 9 that is read by the processing unit 10 and stored in the storage unit 11, or may be a program that is pre-stored at the time of shipment. The image recognition program P1 stored in the storage unit 11 may be an image recognition program P9 that the processing unit 10 downloads from the cloud server 3 or another download server via the second communication unit 13 and stores in the storage unit 11.

[0047] The learning model group M1 stored in the memory unit 11 includes detection models that are trained to detect, for an input image, whether an object is captured in the image and, if so, the range in the image in which the object is captured, based on features obtained from the image. The detection models vary depending on the object, such as a person detection model that detects whether a person is captured in the image and a vehicle detection model that detects whether a vehicle is captured in the image. The detection models included in the learning model group M1 are selected depending on the detection object.

[0048] The learning model group M1 includes two or more detection models that detect two or more parts of a person or object to be detected. These two or more detection models, the processing unit 10, and the image recognition program P1 correspond to the "object detection means" in the claims. When the detection target is a person, the learning model group M1 includes, for example, a head detection model that detects the head and a face detection model that detects the face. The head detection model, the processing unit 10, and the image recognition program P1 correspond to the "head detection means" in the claims, and the face detection model, the processing unit 10, and the image recognition program P1 correspond to the "face detection means" in the claims. When the detection target is a person, the learning model group M1 may, in another example, include a person detection model that detects the entire person and a foot detection model that detects the feet. When the detection target is a vehicle, the learning model group M1 may, for example, include a vehicle body detection model that detects the entire vehicle body and a plate detection model that detects the license plate. When the detection target is a vehicle, the learning model group M1 may include a vehicle body detection model that detects the entire vehicle body and a detection model that detects the front door or rear door part where the vehicle brand logo is attached. In addition, the detection part varies depending on the detection target.

[0049] The learning model group M1 stored in the memory unit 11 includes a recognition model that recognizes the attributes of a detected person or object. This recognition model, the processing unit 10, and the image recognition program P1 correspond to the "object recognition means" in the claims. The learning model group M1 includes attribute-specific recognition models that recognize a person's gender, age, etc. as attributes. The learning model group M1 also includes a face direction detection model that detects the face direction of a face detected by the face detection model. This face direction detection model, the processing unit 10, and the image recognition program P1 correspond to the "face direction detection means" in the claims. The learning model group M1 also includes a face recognition model that performs face recognition processing on (an image of) a face detected by the face detection model. This face recognition model, the processing unit 10, and the image recognition program P1 correspond to the "face recognition means" in the claims. The recognition models included in the learning model group M1 are selected and stored according to the recognition target. The learning model group M1 may include object-specific models that recognize the clothing and ornaments worn by the detected person, or may include models that recognize the color or pattern of the detected object.

[0050] The learning model group M1 stored in the memory unit 11 may be selected or set by the client 4 via the cloud server 3, may be selected by a function of the cloud server 3, or may be automatically selected by the processing unit 10.

[0051] The memory unit 11 stores configuration data corresponding to the selected learning model group M1 and the installation environment of the camera 2. The configuration data includes setting information for each model included in the learning model group M1, such as the size of the detection target area in the image or the size of the recognition target area. The configuration data stored in the memory unit 11 is selected according to the learning model group M1.

[0052] The setting data stored in the storage unit 11 includes a reference value that is referenced in the processing procedure described below. The reference value may be an initial value or may be a value that is changed by the client 4 via the cloud server 3 as described below.

[0053] The first communication unit 12 is a device for connecting to the camera 2. The first communication unit 12 may be an interface such as a USB (Universal Serial Bus) that is connected to the camera 2, or may be a coaxial cable or other serial bus interface. The first communication unit 12 may be a network card for a LAN or a CAN communication device. The first communication unit 12 may be a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices that support various types of cameras 2. The first communication unit 12 may be the same device as the second communication unit 13.

[0054] The second communication unit 13 is a communication device that realizes communication with the cloud server 3 via the network N. The second communication unit 13 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support encrypted communication with the cloud server 3 using SSL or the like. The second communication unit 13 may be an interface that realizes connection with the cloud server 3 via a dedicated line.

[0055] 3 is a block diagram showing the configuration of the cloud server 3. The cloud server 3 is configured to distribute processing among multiple server computers that are communicatively connected. The cloud server 3 includes a processing unit 30, a storage unit 31, and a communication unit 32. The cloud server 3 may be configured as a single server computer as long as it can be communicatively connected to the edge device 1 and the client 4 via the network N.

[0056] The processing unit 30 includes one or more processors such as a CPU, MPU, GPU, or NPU. The processing unit 30 includes a memory that is a temporary storage medium such as an SRAM or DRAM. The processing unit 30 mainly corresponds to the "modification means" in the claims. Also, the processing unit 30 mainly corresponds to the "text modification means" in the claims.

[0057] The storage unit 31 is a relatively large-capacity non-temporary storage medium such as a hard disk, flash memory, etc. The storage unit 31 stores programs (program products) and setting data required for the processing unit 30 to execute processing.

[0058] The program products stored in the storage unit 31 include a server program P3. The server program P3 includes a module that functions as a web server, and can accept data input on a web page displayed on the client 4 and display calculated data on the web page.

[0059] The server program P3 may be a server program P8 stored in a computer-readable storage medium 8 that is read by the processing unit 30 and stored in the memory unit 31, or it may be a program that the processing unit 30 downloads from another download server via the communication unit 32 and stores in the memory unit 31.

[0060] The communication unit 32 is a communication device that realizes a communication connection with the client 4 and the edge device 1 via the network N.

[0061] 4 is a block diagram showing the configuration of the client 4. The client 4 is a personal computer, a smartphone, or a tablet terminal. The client 4 may be used as an operator by the manager of the space where the camera 2 is installed, or may be used by an operator of the management company of the cloud server 3.

[0062] The client 4 includes a processing unit 40, a storage unit 41, a communication unit 42, a display unit 43, and an operation unit 44. The processing unit 40 includes one or more processors such as a CPU, an MPU, a GPU, or an NPU. The processing unit 40 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0063] The storage unit 41 is a memory of a non-temporary storage medium such as a hard disk or flash memory. The storage unit 41 stores a client program P4 for a web server provided by the cloud server 3. The client program P4 is, for example, a web browser program. The client program P4 may also be a program that causes the processing unit 40 to execute a process of displaying data provided by the cloud server 3 on a screen.

[0064] The communication unit 42 is a communication device that establishes a communication connection with the cloud server 3 via the network N. The communication unit 42 may be a communication device that establishes a communication connection with the cloud server 3 via a dedicated line. The communication unit 42 may be a communication device that establishes a direct communication connection with the second communication unit 13 of the edge device 1 via a wireless communication medium, a USB cable, or the like.

[0065] The display unit 43 may be a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. The display unit 43 displays a web page including text or images through processing based on the client program P4 of the processing unit 40. A touch panel built-in display may also be used for the display unit 43.

[0066] The operation unit 44 is a user interface such as a keyboard or a pointing device that accepts operations from an operator. The operation unit 44 may be a touch panel built into the display of the display unit 43, or may be physical buttons. The operation unit 44 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 44 can notify the processing unit 40 of operation information by the operator.

[0067] In the object recognition system 100 configured as above, the edge device 1 will now explain the processing procedure for performing object recognition by narrowing down frame images captured by the camera 2 to those in which the features of the object can be clearly captured and the object can be recognized. Figure 5 is a flowchart showing an example of the processing procedure for image recognition in the edge device 1. The processing unit 10 of the edge device 1 receives frame images from the camera 2 in chronological order, and executes the following processing each time a frame image is received.

[0068] The processing unit 10 acquires a frame image (step S101), inputs the acquired frame image to a first detection model corresponding to the detection target (step S102), and acquires a first detection result (step S103). The processing unit 10 inputs the acquired frame image to a second detection model (step S104), and acquires a second detection result (step S105). In step S104, the processing unit 10 may extract the range of the target detected in the first detection result from the frame image and input it to the second detection model.

[0069] The processing unit 10 calculates the suitability of the detected detection target as a recognition target based on the position and size of the first part of the detection target obtained as the first detection result and the position and size of the second part of the detection target obtained as the second detection result (step S106).

[0070] In step S106, if the first portion includes the second portion, the processing unit 10 calculates the distance between a specific position in the first portion and a specific position in the second portion. The processing unit 10 uses the calculated distance as the degree of matching. The processing unit 10 may calculate the degree of matching from the proportion of the range of the second portion in the first portion. The processing unit 10 may calculate the degree of matching from the ratio of the length of a specific portion of the first portion to the length of a specific portion of the second portion, or may use the distance between the center position (center of gravity) of the first portion and the center position (center of gravity) of the second portion as the degree of matching.

[0071] The processing unit 10 compares the degree of conformance calculated in step S106 with a predetermined reference value, and determines whether the degree of conformance satisfies the condition using the reference value as a result of the comparison (step S107). In step S107, the processing unit 10 determines whether the condition is satisfied, such as whether the distance is equal to or less than a predetermined reference value, whether the distance is equal to or greater than a predetermined reference value, or whether the distance is within a range of a predetermined reference value. The processing unit 10 may determine whether the condition is satisfied based on whether the ratio is equal to or greater than a predetermined ratio, whether the ratio is equal to or less than a predetermined ratio, or whether the ratio is within a predetermined ratio range. The processing unit 10 may determine whether the condition is satisfied based on whether the ratio is equal to or greater than a predetermined rate, whether the ratio is equal to or less than a predetermined rate, or whether the ratio is within a predetermined range. Note that in the claims, "satisfies the reference value" means that "the degree of conformance satisfies the condition using the reference value" in step S107.

[0072] When the processing unit 10 determines that the conformance satisfies the condition using the reference value (S107: YES), it inputs the frame image acquired in step S101 to a recognition model in the learning model group M1 (step S108). In step S108, the processing unit 10 may input to the recognition model a partial image obtained by extracting a first part in the first detection result from the frame image, or a partial image obtained by extracting a second part in the second detection result.

[0073] The processing unit 10 acquires the recognition result from the recognition model (step S109). The processing unit 10 stores the acquired recognition result and the degree of conformance calculated in step S106 in association with the identification data of the frame image (step S110), and ends the process. If there are multiple recognition targets (whose conformance satisfies the condition using the reference value), the processing unit 10 executes the processes of steps S108-S110 according to the number of recognition targets.

[0074] In step S107, if it is determined that the degree of conformance does not satisfy the condition using the reference value (S107: NO), the processing unit 10 stores the degree of conformance calculated in step S106 in association with the identification data of the frame image (step S111), and ends the processing. In this case, the processing unit 10 omits processing using the recognition model for the frame image acquired in step S101.

[0075] When the recognition results for each frame image stored in the memory unit 11 have been accumulated for a predetermined period or for a predetermined number of frame images, the processing unit 10 of the edge device 1 associates the identification data of the device itself (for identifying the target space) with the identification data of the frame image and transmits the recognition results and data on the degree of matching to the cloud server 3. This allows an operator to access the cloud server 3 using the client 4 and refer to the recognition results and degree of matching in the edge device 1 for each space. The degree of matching (as a recognition target for the detected object to be detected) can be used as the reliability of the recognition result for the detected object to be detected.

[0076] Furthermore, instead of associating the identification data of the frame image with the recognition result and matching data, storing it in the storage unit 11 of the edge device 1, and transmitting it to the cloud server 3 as described above, the frame image itself may be associated with the recognition result and matching data, and stored (saved) in the storage unit 11 of the edge device 1, or transmitted to the cloud server 3. This makes it possible to obtain a learning image (or an image for fine tuning) for the recognition model used in step S108.

[0077] The processing procedure shown in FIG. 5 will be described using a specific example. FIG. 6 is an explanatory diagram of processing by the edge device 1. In the example of FIG. 6, the edge device 1 uses a head detection model M11 that detects a person's head and a face detection model M12 that detects a face area to recognize the person's age or gender. The edge device 1 uses, for example, an age recognition model M13 that recognizes the age. The processing unit 10 calculates the degree of conformance from the position and size of the head obtained by inputting a frame image to the head detection model M11 and the position and size of the face obtained by inputting the frame image to the face detection model M12. In the example shown in FIG. 6, the degree of conformance is calculated using distances D1 and D2 from predetermined vertices (top left and bottom right in FIG. 6) of a rectangle detected as the head area to predetermined vertices (top left and bottom right in FIG. 6) of a rectangle detected as the face area. If the degree of conformance (distances D1, D2) meets the conditions using the reference value, the processing unit 10 inputs the first or second part of the frame image into the age recognition model M13 and stores the recognition result (age and reliability score) and the degree of conformance from the age recognition model M13. If the degree of conformance (distances D1, D2) does not meet the reference value, the processing unit 10 does not continue processing the frame image, stores the degree of conformance for the identification data of the frame image, and ends processing.

[0078] A specific example of a method for calculating the degree of matching is shown at the bottom of FIG. 6. FIG. 6 shows examples of detection results for Cases 1 to 3. In Case 1, the processing unit 10 determines that the head area detected from the frame image includes the face area, and further determines that the distance D1 between the upper left vertex of the rectangle R1 detected as the head area and the upper left vertex of the rectangle R2 detected as the face area is less than a first threshold reference value, and that the distance D2 between the lower right vertex of the rectangle R1 corresponding to the head and the lower right vertex of the rectangle R2 corresponding to the face is less than a second threshold reference value. As a result, the processing unit 10 determines that the distances D1 and D2 calculated as the degree of matching are smaller than the first and second threshold reference values, respectively, and that the conditions are met. In Case 1, the processing unit 10 inputs the target frame image (either the entire image or a portion of the rectangles R1 and R2) to the age recognition model M13 to obtain a recognition result. The processing unit 10 may store and output a reliability score (score) corresponding to the accuracy included in the recognition result output from the age recognition model M13 as the reliability of the object recognition system 100 for the frame image (reliability of the recognition result of the object to be detected included in the frame image). Also, the processing unit 10 may store and output the above-mentioned reliability score (of the object recognition result) and the suitability of the object to be detected as a recognition target as the reliability of the object recognition system 100 for the frame image (reliability of the recognition result of the object to be detected included in the frame image).

[0079] In Case 2 of the example shown in FIG. 6, the processing unit 10 acquires, as detection results, a rectangle R1 corresponding to the head and a rectangle R2 corresponding to the face from the frame image, as in Case 1. In Case 2, the processing unit 10 determines that the head rectangle R1 encompasses the face rectangle R2, but the distance D1 between the upper left vertex of the head rectangle R1 and the upper left vertex of the face rectangle R2 is equal to or greater than a first threshold included in the reference value, and therefore does not satisfy the condition. In Case 2, the processing unit 10 terminates the processing without inputting the target frame image to the age recognition model M13, i.e., without performing age recognition on the target frame image. The processing unit 10 may store and output the calculated compatibility (distance D1 or distance D2) as the reliability of the object recognition system 100 for the frame image (reliability of the recognition result of the object to be detected appearing in the frame image). In this case, the larger the distance D1 or distance D2 used as the compatibility, the lower the reliability output.

[0080] In Case 3 of the example shown in FIG. 6, the processing unit 10 acquires, as detection results, a rectangle R1 corresponding to the head and a rectangle R2 corresponding to the face from the frame image, as in Case 1. In Case 3, the processing unit 10 determines that the head rectangle R1 encompasses the face rectangle R2, but the distance D2 between the lower right vertex of the head rectangle R1 and the lower right vertex of the face rectangle R2 is equal to or greater than a second threshold included in the reference value, and therefore does not satisfy the condition. In Case 3, the processing unit 10 terminates the processing without inputting the target frame image to the age recognition model M13, i.e., without performing age recognition on the target frame image. The processing unit 10 may store and output the calculated compatibility (distance D1 or distance D2) as the reliability of the object recognition system 100 for the frame image (reliability of the recognition result of the object to be detected included in the frame image). In this case, too, the larger the distance D1 or distance D2 used as the compatibility, the lower the reliability output.

[0081] As shown in Figure 6, by determining whether to proceed to recognition processing under the conditions that the head rectangle R1 contains the face rectangle R2 and the distance between the vertices of the head rectangle R1 and the face rectangle R2 is less than a reference value, recognition processing can be performed by narrowing down the frame images in which the face is clearly visible enough to fully calculate facial feature amounts. Rather than allocating computational resources to low-accuracy recognition processing for frame images in which the characteristic features are unclear or hidden, it is possible to reduce the load on computational resources and increase accuracy by narrowing down recognition to frame images in which the characteristics can be clearly captured.

[0082] In the example shown in FIG. 6, the processing unit 10 uses distances D1 and D2 between the vertices of the rectangle R1 corresponding to the head and the vertices of the rectangle R2 corresponding to the face as the degree of matching to be compared with the reference value. However, the degree of matching may be calculated in other ways. The degree of matching is not limited to the distance between the vertices of the rectangle R1 and the vertices of the rectangle R2, but may also be calculated from the ratio of the area occupied by the rectangle R2 corresponding to the face to the rectangle R1 corresponding to the head. The processing unit 10 may also calculate the degree of matching from the ratio of the length of the long side of the rectangle R1 corresponding to the head to the length of the long side of the rectangle R2 corresponding to the face. The distance between the center position (center of gravity) of the rectangle R1 and the center position (center of gravity) of the rectangle R2 may also be used as the degree of matching to be compared with the reference value. In this case, the shorter the distance between the center positions, the higher the degree of matching as a recognition target is determined to be.

[0083] 6, the processing unit 10 uses the distances D1 and D2 between the vertices of the rectangle R1 corresponding to the head and the vertices of the rectangle R2 corresponding to the face as the degree of conformance to be compared with the reference value. However, if the recognition model is not the age recognition model M13 as described above but a face recognition model (a model that determines whether a detected face is the same as any of the faces stored (registered) in the storage unit 11, etc.), in addition to the distances D1 and D2, a face direction score calculated using a face direction detection model (a score indicating the likelihood that the face direction obtained by inputting a face image in which a face region detected by the face detection model M12 is extracted into the face direction detection model) may be used as the degree of conformance to be compared with the reference value. In this case, the processing unit 10 performs face recognition processing using the face recognition model only when the distances D1 and D2 calculated as the degree of conformance are smaller than the first and second thresholds of the reference values, respectively, and the face direction score using the face direction detection model is higher than a predetermined threshold (facing in a direction close to the front). The process of using the face direction score obtained using the face direction detection model in addition to the distances D1 and D2 as the suitability for comparison with the reference value is a specific example of the process of "calculating the suitability of the detected face as a face recognition target based on the face direction detected by the face direction detection means in addition to the positions and sizes of the head and face" in the claims.

[0084] 6, the head detection model M11 for detecting the head of a person in a frame image and the face detection model M12 for detecting the face area are described as outputting a detection result in which the area in which the head is captured is indicated by a rectangle. However, the head detection model M11 and the face detection model M12 may each output a bounding box that is not limited to a rectangle, such as a square or elliptical bounding box, as the detection result.

[0085] 6, in order to recognize a person's age, a head detection model M11, a face detection model M12, and an age recognition model M13 are adopted as a learning model group M1, a rectangle R1 and a rectangle R2 are detected, and the distance between the rectangles R1 and R2 is calculated as the degree of conformance. However, if the recognition target is different, the method of calculating the degree of conformance will differ, and of course the reference value will also differ. Therefore, when the learning model group M1 is selected and stored in the storage unit 11, a corresponding reference value may be selected by the cloud server 3 and stored together.

[0086] For example, when using a person detection model that detects the entire person and a foot detection model that detects the foot to recognize the type or color of shoes from the foot, the positional relationship between the rectangle enclosing the area where the detected person is captured and the rectangle enclosing the area where the foot is captured is preferably such that the foot is located at a biased position relative to the entire person, and both feet are detected. In this case, the distance between the vertices of the rectangle should be longer in the vertical direction but shorter in the horizontal direction. Therefore, the reference value is set to a value different from the first and second thresholds shown in FIG. 6. Furthermore, when the detection target is a vehicle and a vehicle body detection model that detects the entire vehicle body and a plate detection model that detects the license plate are used to recognize the vehicle license plate, it is preferable that the rectangle enclosing the area where the license plate is captured occupies a small area relative to the entire vehicle body, and the reference value can be set appropriately depending on such conditions. When the detection target is an item on a tray, such as a sorter in a logistics warehouse, and the purpose is to recognize the type of item, it is possible to determine whether the frame image captures a range in which feature quantities for identifying the item can be appropriately calculated by setting a reference value for compatibility.

[0087] [Second embodiment] In the second embodiment, the learning model group M1 used in the edge device 1 can be changed as appropriate from the model group held in the model database 310 accessible by the cloud server 3. Fig. 7 is a schematic diagram of an object recognition system 100 according to the second embodiment. The hardware configuration of the object recognition system 100 according to the second embodiment is similar to that of the object recognition system 100 according to the first embodiment, and therefore common components are denoted by the same reference numerals and detailed description thereof will be omitted.

[0088] In the object recognition system 100 of the second embodiment, the cloud server 3 stores a group of learning models used by the edge device 1 in a model database 310. The cloud server 3, as a cloud manager, selects a learning model from the model database 310 according to the detection target and recognition target in the edge device 1, and deploys it to the edge device 1. The selection may be performed by the client 4 via the cloud server 3, or may be performed by processing based on a predetermined algorithm in the cloud server 3.

[0089] The model database 310 may be constructed in the storage unit 31 or in an external storage device. A part of the model database 310 may include a model providing service used on the Web that is connected for communication via the network N. The model database 310 holds detection models, such as a person detection model and a vehicle detection model, that are trained to determine whether a specific person or object is photographed based on feature quantities obtained from an image. The model database 310 holds recognition models for multiple recognition targets so that they can be provided. The model database 310 holds attribute-specific models that recognize a person's gender and age as attributes.

[0090] 8 is a flowchart showing an example of a model setting process procedure in the object recognition system 100 of the second embodiment. When an operator uses the client 4 to access the cloud server 3, the processing unit 30 of the cloud server 3 starts the following process.

[0091] The processing unit 30 identifies the identification data of the edge devices 1 that are permitted to be accessed by the account of the operator using the client 4, or the identification data or names of the corresponding spaces (step S301). The processing unit 30 transmits a web page including a list of the identified identification data or names to the client 4 (step S302), and accepts selection of the target edge device 1 (space) from the list on the web page (step S303).

[0092] The processing unit 30 transmits a web page including a screen for accepting selection of a detection target and a recognition target to the client 4 (step S304), and accepts the selection of the detection target and the recognition target on the web page displayed on the client 4 (step S305). The processing unit 30 selects a detection model and a recognition model from the model database 310 according to the selected detection target and recognition target (step S306), and reads out the reference value settings corresponding to the selected detection model and recognition model from the data stored in the storage unit 31 (step S307).

[0093] The processing unit 30 transmits the detection model and recognition model selected in step S306 and the reference value settings read in step S307 to the edge device 1 selected in step S303 (step S308). The processing unit 30 deploys the selected detection model and recognition model and an executable file using them to the edge device 1 (step S309), and ends the setting process.

[0094] 8 can be executed by the client 4 at any time. It may be executed during the initial setup of the edge device 1, or may be executed when the placement of the camera 2 is changed in the space where the camera 2 is installed.

[0095] 5, using the detection model and recognition model deployed to the edge device 1 in step S309 above, and the reference value transmitted in step S308, the "process of determining whether the conformance of the detection target in each frame image satisfies the reference value" shown in step S107 of Fig. 5 may be performed, and as a result, only frame images whose conformance satisfies the reference value may be stored in the storage unit 11 of the edge device 1 or transferred to and stored in the cloud server 3. In this way, it is possible to obtain learning images (or images for fine-tuning) for detection models and recognition models of the same type as the detection model and recognition model deployed to the edge device 1 in step S309.

[0096] [Third embodiment] In the third embodiment, the learning model group M1 used in the edge device 1 includes a VLM (Vision-Language Model) as a recognition model, which can accept text in addition to image data and change the processing of the image data depending on the text. This eliminates the need for the edge device 1 to change the recognition model itself when changing or adding a recognition target. Furthermore, even when there are multiple recognition targets, recognition processing can be performed with a single VLM. The learning model group M1 may also include a multimodal model (Multimodal Language Model). In addition to the processing unit 10 and the image recognition program P1 (see FIG. 2), the function of the VLM itself corresponds to the "recognition processing change means" in the claims.

[0097] The hardware configuration of the object recognition system 100 of the third embodiment is similar to that of the object recognition system 100 of the first or second embodiment, so the common components are denoted by the same reference numerals and detailed description thereof will be omitted. In the third embodiment, a detection model is selected from the model database 310 via the cloud server 3 and deployed to the edge device 1, as in the second embodiment.

[0098] 9 is a flowchart showing an example of a model setting processing procedure in the object recognition system 100 of the third embodiment. When an operator uses the client 4 to access the cloud server 3, the processing unit 30 of the cloud server 3 starts the following processing. Of the processing procedures shown in FIG. 9, steps that are common to the processing procedures shown in FIG. 8 of the second embodiment are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0099] When the processing unit 30 receives a selection of a target edge device 1 (space) from the list on the web page (S303), it transmits a web page including a screen for receiving the selection of the detection target to the client 4 (step S314), and receives the selection of the detection target on the web page displayed on the client 4 (step S315). The processing unit 30 selects a detection model from the model database 310 according to the selected detection target (step S316), and reads out the reference value settings corresponding to the selected detection model from the data stored in the storage unit 31 (step S317).

[0100] The processing unit 30 accepts text to be input to the recognition model, which is a VLM, on a web page displayed on the client 4 (step S318). In step S318, the processing unit 30 accepts text such as "Age of the detected person" or "How old is the detected person?" in English or any other language.

[0101] The processing unit 30 transmits the detection model selected in step S316, the reference value settings read in step S317, and the text for the recognition model accepted in step S318 to the edge device 1 (step S319). The processing unit 30 deploys the selected detection model and an executable file using the same to the edge device 1 (step S320), and ends the setting process.

[0102] The text accepted by the client 4 and transmitted from the cloud server 3 in step S319 is received by the edge device 1 and stored in association with the recognition model. The processing unit 10 of the edge device 1 inputs the acquired frame image into a detection model of two or more parts, and if the compatibility calculated based on the two detection results satisfies the condition using the reference value, inputs the frame image into the recognition model, which is a VLM. The processing unit 10 of the edge device 1 inputs text specifying the recognition target accepted from the client 4 via the cloud server 3 into the recognition model, which is a VLM, and obtains the recognition result output from the recognition model. When there are multiple recognition targets, for example, when age and gender are the recognition targets, the processing unit 10 inputs the text "Output the age of the detected person" and the text "Output the gender of the detected person," and obtains the recognition result including the age, gender, and a confidence score (of the recognition result).

[0103] According to the processing procedure shown in FIG. 9, the recognition model used in the edge device 1 can change the recognition target by changing the text at any timing in response to an instruction from an operator received by the client 4.

[0104] FIG. 10 is an explanatory diagram of processing by the edge device 1 of the third embodiment. Similar to the processing content shown in FIG. 6, FIG. 10 illustrates an example in which the edge device 1 uses a head detection model M11 and a face detection model M12 to recognize the age and gender of a person. The edge device 1 of the third embodiment uses a VLM model M14 as a recognition model. If the compatibility (distances D1 and D2) satisfies the condition using the reference value, the processing unit 10 inputs the first or second part of the frame image to the model M14, and inputs text instructing the output of the age and text instructing the output of the gender to the model M14. The processing unit 10 acquires the recognition result (age, gender, and confidence score) output from the model M14 and stores it together with the compatibility. The processing unit 10 may transmit the recognition result to the cloud server 3 in association with the identification data of the frame image.

[0105] In the third embodiment, the recognition content can be changed depending on the text, so there is no need to replace the recognition model in response to a change in the recognition content.

[0106] [Fourth embodiment] In the fourth embodiment, the learning model group M1 used in the edge device 1 is divided into a learning model of the backbone part that extracts features from input image data and a learning model of the task head part that executes recognition processing based on the extracted features, for both detection models and recognition models. In the fourth embodiment, the learning model group M1 used in the edge device 1 can also be changed as appropriate from the model group held in the model database 310 accessible by the cloud server 3.

[0107] The hardware configuration of the object recognition system 100 of the fourth embodiment is similar to the hardware configuration of the object recognition system 100 of the first embodiment, and therefore the common configurations are denoted by the same reference numerals and detailed description thereof will be omitted. In the fourth embodiment, for both the (trained) detection model and the recognition model, the backbone parts are not swapped, and the task head parts of the detection model and the recognition model are selected from the model database 310 via the cloud server 3 and changed.

[0108] 11 is a flowchart showing an example of a model setting processing procedure in the object recognition system 100 of the fourth embodiment. When an operator uses the client 4 to access the cloud server 3, the processing unit 30 of the cloud server 3 starts the following processing. Of the processing procedures shown in FIG. 11, steps that are common to the processing procedures shown in FIG. 8 of the second embodiment are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0109] In the fourth embodiment, when the processing unit 30 receives the selection of the detection target and the recognition target (S305), it selects a learning model of the corresponding task head part according to each of the selected detection target and the selected recognition target (step S326).The processing unit 30 reads the reference value setting corresponding to the learning model of the selected task head part from the data stored in the storage unit 31 (step S327).

[0110] The processing unit 30 transmits the learning model of the task head part selected in step S326 and the reference value setting read in step S327 to the selected edge device 1 (step S328). The processing unit 30 deploys the learning model of the selected task head part and an executable file using these to the edge device 1 (step S329), and ends the setting process.

[0111] FIG. 12 is an explanatory diagram of processing by the edge device 1 of the fourth embodiment. Similar to the processing content shown in FIG. 6, FIG. 12 shows an example in which the edge device 1 uses a head detection model M11, a face detection model M12, and an age recognition model M13 to recognize the age and gender of a person. In the fourth embodiment, the head detection model M11 and the face detection model M12 are models of the task head portion. The head detection model M11 and the face detection model M12 are configured to perform head detection and face detection, respectively, using feature data obtained from the backbone portion model M11B. The age recognition model M13 is also a model of the task head portion, and outputs a recognition result using feature data obtained from the backbone portion model M13B.

[0112] In the fourth embodiment, the processing unit 30 inputs a frame image to the model M11B, and outputs a first detection result from the head detection model M11 using the feature amount calculated by the model M11B, and outputs a second detection result from the face detection model M12. Thereafter, the calculation of the goodness of fit using the first detection result and the second detection result is the same as in the first embodiment.

[0113] In the fourth embodiment, when an operator refers to the recognition result by the edge device 1 through the cloud server 3 at the client 4 and wishes to change the detection content and recognition content, the task head part can be swapped. In this case, as shown in the upper part of Fig. 12, the detection model can be changed to a person detection model M15 of the task head part that detects the entire person and a face detection model M16 that detects the face part from the entire person, and the recognition model can be changed to a gender recognition model M17.

[0114] In this way, in the fourth embodiment, the detection target and recognition target can be changed by replacing only the task head portion, so there is no need to replace the entire recognition model in response to changes in the recognition content. Depending on what the detection target is and what the content (object) to be recognized for the detection target is, it is possible to realize a recognition system that can meet a variety of conditions by replacing only the necessary parts as much as possible, without replacing everything.

[0115] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0116] 100 Object Recognition System 1. Edge Devices 10 Processing section 11 Storage section P1 Image Recognition Program M1 Learning Models M11 Head Detection Model M12 face detection model M13 Age Recognition Model M14 model 2 Cameras 3. Cloud Server 30 Processing section 31 Storage section 300,310 databases 4 Client 40 Processing section 43 Display section

Claims

1. an object detection means for detecting two or more parts of a detection target object captured in a frame image input from a camera; a matching calculation means for calculating a matching degree of the object to be detected as a recognition target based on the positions and sizes of the two or more portions; a comparison means for comparing the suitability as a recognition target with a predetermined reference value; an object recognition means for recognizing only the object to be detected that satisfies the reference value as a result of the comparison; the two or more parts detected by the object detection means are a human head and a human face, the object detection means detects the head and face of the person by determining a rectangular bounding box of the head and face of the person; the rectangular bounding box of the head encompasses the rectangular bounding box of the face; the matching calculation means calculates a distance between a predetermined vertex in a rectangular bounding box of the head and a corresponding vertex in a rectangular bounding box of the face; the comparison means compares the distance with the reference value; The object recognition means is an object recognition system that recognizes only the person to be detected whose distance satisfies the reference value as a result of the comparison.

2. An object detection means for detecting two or more parts of a detection target object captured in a frame image input from a camera; a matching calculation means for calculating a matching degree of the object to be detected as a recognition target based on the positions and sizes of the two or more portions; a comparison means for comparing the suitability as a recognition target with a predetermined reference value; an object recognition means for recognizing only the object to be detected that satisfies the reference value as a result of the comparison; the two or more parts detected by the object detection means are a human head and a human face, further comprising a face direction detection means for detecting a face direction of the face detected by the object detection means, The suitability calculation means calculates the suitability of the detected face as a recognition target based on the position and size of the head and face detected by the object detection means, as well as the facial direction detected by the facial direction detection means.

3. Computer, an object detection means for detecting two or more parts of a detection target object captured in a frame image input from a camera; a matching calculation means for calculating a matching degree of the object to be detected as a recognition target based on the positions and sizes of the two or more portions; a comparison means for comparing the suitability as a recognition target with a predetermined reference value; functioning as an object recognition means for recognizing only those detection target objects that satisfy the reference value as a result of the comparison; the two or more parts detected by the object detection means are a human head and a human face, the object detection means detects the head and face of the person by determining a rectangular bounding box of the head and face of the person; the rectangular bounding box of the head encompasses the rectangular bounding box of the face; the matching calculation means calculates a distance between a predetermined vertex in a rectangular bounding box of the head and a corresponding vertex in a rectangular bounding box of the face; the comparison means compares the distance with the reference value; The object recognition means is an object recognition program that recognizes only the person to be detected whose distance satisfies the reference value as a result of the comparison.

4. A computer, an object detection means for detecting two or more parts of a detection target object captured in a frame image input from a camera; a matching calculation means for calculating a matching degree of the object to be detected as a recognition target based on the positions and sizes of the two or more portions; a comparison means for comparing the suitability as a recognition target with a predetermined reference value; functioning as an object recognition means for recognizing only those detection target objects that satisfy the reference value as a result of the comparison; the two or more parts detected by the object detection means are a human head and a human face, causing the computer to further function as a face direction detection unit that detects a face direction of the face detected by the object detection unit; The suitability calculation means calculates the suitability of the detected face as a recognition target based on the position and size of the head and face detected by the object detection means, as well as the face direction detected by the face direction detection means.

Images