Personnel information extraction method and device based on image, equipment and storage medium

Abstract

The invention provides an image-based personnel information extraction method and device, equipment and a storage medium, relates to the technical field of image processing, and solves the technical problem that vulnerabilities exist in information analysis of a single object in the related art. According to the method, the correlation between the target persons can be identified more accurately and efficiently by using the correlation of the target persons in time and space, so that the corresponding person set can be determined, the accuracy of identifying the correlation behavior between the persons can be improved, and the information analysis of a single object is more reliable.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image-based personnel information extraction method, device, equipment and storage medium. Background Art

[0002] In public security control and crime detection work, surveillance cameras are used to monitor public places, specific areas and other surveillance locations. By utilizing facial recognition technology, the identity of each person can be accurately identified and confirmed based on captured images, and it can be determined whether key personnel have entered or exited the above-mentioned locations. The activity trajectory of key personnel can then be determined to achieve monitoring of key personnel.

[0003] However, although the related art provides solutions for using surveillance cameras to implement trajectory analysis, most of them focus on the trajectory recognition of a single object and cannot make a comprehensive judgment on the associated behaviors of multiple objects, resulting in loopholes in the information analysis of a single object. Summary of the invention

[0004] The present application provides an image-based personnel information extraction method, device, equipment and storage medium, which solves the technical problem of loopholes in the information analysis of a single object in the related technology. The present application can combine face recognition technology to find the connection between individuals and identify the corresponding personnel set, which helps to achieve efficient and reliable gang identification.

[0005] In a first aspect, the present application provides a method for extracting personnel information based on an image, comprising:

[0006] Obtaining location images provided by a number of surveillance cameras, and recording shooting time information of the location images and location information of the surveillance cameras;

[0007] Based on the facial information of several persons stored in a preset personnel information database, searching for a target person in the multiple acquired location images to determine the location image corresponding to the first capture of the target person as the first target image, wherein the target person is a person whose facial information is stored in the personnel information database;

[0008] When the first target image is determined, a time window is determined according to the shooting time information corresponding to the first target image and a preset time range to search for other target persons from the location images acquired within the time window, and a location image corresponding to the first shooting of the other target persons is determined as the second target image;

[0009] Based on the shooting time information and location information carried by each target image corresponding to a different target person, determining a spatiotemporal correlation value of the target person in each second target image relative to the target person in the first target image;

[0010] If the spatiotemporal correlation value is greater than or equal to a preset correlation threshold, the first target image and the second target image are added to the spatiotemporal cluster set, so as to determine that the target persons corresponding to the first target image and the second target image belong to the same personnel set with a preset correlation threshold spatiotemporal cluster set when the personnel density value corresponding to the spatiotemporal cluster set is greater than or equal to the preset density threshold.

[0011] In a second aspect, the present application also provides an image-based personnel information extraction device, which includes:

[0012] An image acquisition module is configured to acquire location images provided by a plurality of surveillance cameras and record shooting time information of the location images and location information of the locations where the surveillance cameras are located;

[0013] A first face recognition module is configured to search for a target person in the acquired multiple location images based on the face information of several persons stored in a preset personnel information database, so as to determine the location image corresponding to the first time the target person is photographed as the first target image, and the target person is a person whose face information is stored in the personnel information database;

[0014] The second face recognition module is configured to, when the first target image is determined, determine a time window according to the shooting time information corresponding to the first target image and a preset time range to search for other target persons from the location images acquired within the time window, and determine the location image corresponding to the first shooting of the other target persons as the second target image;

[0015] A similarity recognition module is configured to determine a spatiotemporal correlation value of a target person in each second target image relative to a target person in the first target image based on shooting time information and location information carried by each target image corresponding to a different target person;

[0016] The set discrimination module is configured to add the first target image and the second target image to the spatiotemporal cluster set if the spatiotemporal correlation value is greater than or equal to a preset correlation threshold, so as to determine that the target persons corresponding to the first target image and the second target image belong to the same person set when the person density value corresponding to the spatiotemporal cluster set is greater than or equal to the preset density threshold.

[0017] In a third aspect, the present application further provides an electronic device, comprising:

[0018] one or more processors;

[0019] a storage device for storing one or more programs,

[0020] When one or more programs are executed by one or more processors, the one or more processors implement the image-based personnel information extraction method of the present application.

[0021] In a fourth aspect, the present application also provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to execute the image-based personnel information extraction method of the present application.

[0022] The present application scheme performs facial recognition on the people in the image to obtain the time and place of the target person's appearance, and utilizes the temporal and spatial correlation of the target person to more accurately and efficiently identify the correlation between the target persons, so as to determine the corresponding set of persons, thereby helping to improve the accuracy of identifying the correlation behavior between persons and making the information analysis of a single object more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the steps of a method for extracting personnel information based on an image provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of the steps for determining a spatiotemporal correlation value provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of the structure of an image-based personnel information extraction device provided in one embodiment of the present application;

[0026] Figure 4 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only the parts related to the embodiments of the present application rather than all structures are shown in the accompanying drawings, and those skilled in the art should be able to think of it after reading the specification of this application that as long as the technical features do not contradict each other, any combination of the technical features can constitute an optional implementation method.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated before and after are in an "or" relationship. In the description of the present application, "multiple" means two or more, and "several" means one or more.

[0029] In public security control and crime investigation, surveillance cameras are used to find key personnel, such as using facial recognition technology to identify key personnel entering and leaving the surveillance site, or manually checking surveillance images one by one to find key personnel, so as to analyze the activity trajectory of key personnel. However, the inventors found that this method can only check the activity trajectory of a single key person, lacking in-depth analysis of the relationship between multiple key personnel, such as being unable to comprehensively judge the associated behaviors of multiple objects, resulting in loopholes in the information analysis of a single key person.

[0030] In actual application, the inventor found that the above method can only analyze the activity trajectory of key personnel who need to be paid attention to through the images taken by the surveillance camera, but cannot make a corresponding analysis of the correlation between the key personnel and other key personnel. For example, in the application scenario where thieves are key personnel, the information analysis made by the above method can only determine the activity trajectory of the personnel between some places, but it is easy to ignore the gang behavior between key personnel, and it is impossible to determine whether the personnel is related to other key personnel, resulting in loopholes in the information analysis of the personnel, which makes it limited in personnel monitoring and is not conducive to social security control. The above examples are exemplary, and key personnel can also be groups of personnel such as students and staff. Similarly, for these groups of personnel, the above method can only be used to analyze the trajectory of a single object, and it is impossible to analyze the correlation between two objects or even more objects. It is difficult to determine whether there is a correlation behavior between them, making it difficult to meet the needs of information analysis of a single object.

[0031] In this regard, the present application provides an image-based personnel information extraction method, which can identify key personnel in a monitored area based on time and space scope, thereby determining whether the person has related behavior with the personnel in the database, which helps to achieve more accurate information identification.

[0032] The image-based personnel information extraction method provided in the embodiment of the present application can be executed by an electronic device, which can be a server, a computer and other devices. For example, the electronic device includes a processor and a memory, and the processor and the memory can be connected via a bus or other means. The number of processors can be one or more, and optionally, the processor can include processing units such as an application processor (AP), a graphics processing unit (GPU) and a central processing unit (CPU).

[0033] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the image-based personnel information extraction method in the embodiment of the present application. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to the use of the electronic device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The electronic device may also include one or more communication interfaces to achieve communication with other devices through the communication interface.

[0034] On the basis of the aforementioned hardware structure, the electronic device is installed with at least one operating system. The operating system may be an operating system such as an Android system, a Windows system, or a Linux system. Currently, it is taken as an example that the electronic device is at least installed with a Linux system. On the basis of the installed operating system, the electronic device may install at least one application, which may be an application that comes with the operating system, or an application downloaded from a background server or a third-party device. In one embodiment, the electronic device is installed with at least an application for implementing the aforementioned image-based personnel information extraction method, and the image-based personnel information extraction method in the present application may be implemented during the operation of the application.

[0035] The personnel information database is used to store facial information. The facial information of several persons is pre-stored in the personnel information database to facilitate facial recognition. It is conceivable that the facial information stored in the personnel information database is the facial information of persons who need to be paid attention to. For this, the stored facial information can be updated according to actual application needs. It is conceivable that the personnel information database can be stored in the memory of the electronic device so that the electronic device can quickly obtain the information in the personnel information database; of course, the personnel information database can also be stored on an external device (such as a storage device of a server) so that the electronic device can obtain the facial information of the person from the external device through a wireless network, optical fiber, etc.

[0036] Figure 1 This is a schematic diagram of the steps of the image-based personnel information extraction method provided in one embodiment of the present application. In one embodiment, the electronic device can obtain the location images provided by several surveillance cameras, such as by transmitting the captured location images to the electronic device in real time by the surveillance camera in a wired or wireless manner. Then, the electronic device identifies the associated behaviors of the personnel in the personnel information database based on the obtained location images. The specific steps include steps S110 to S150:

[0037] Step S110: Acquire location images provided by a plurality of surveillance cameras, and record shooting time information of the location images and location information of the locations of the surveillance cameras.

[0038] When the surveillance camera captures the image of the place, it can record the shooting time, shutter duration, etc. of the image accordingly as the shooting time information, and the location of the surveillance camera can also be used as the location information accordingly, and then provide the above information to the electronic device to facilitate gang identification. For the location of the surveillance camera, longitude and latitude coordinates or coordinates in a coordinate system constructed with a specific center point can be used as location information to indicate the location of the surveillance camera. It is conceivable that when there are multiple cameras in the same place (larger places), such as surveillance cameras are set up at different entrances and exits in parks, the locations of different surveillance cameras can be identified by corresponding longitude and latitude coordinates.

[0039] It can be imagined that multiple surveillance cameras corresponding to received location images can constitute a coverage area, so by adding surveillance cameras, more location images can be obtained, thereby expanding the coverage area and making the coverage area more accurate. In this regard, in actual applications, more surveillance cameras can be added according to actual application requirements to obtain corresponding information, such as the above-mentioned location images, shooting time information, and location information of the location where the surveillance cameras are located.

[0040] Optionally, images captured by several surveillance cameras within a certain period of time may be selected to identify whether there is gang behavior among the personnel in the warehouse during the period of time.

[0041] It should be noted that the shooting time information of the place image and the location information of the place where the surveillance camera is located can be that the surveillance camera records the time and location according to the application program of the device itself when shooting the place image, and then provides the above information to the electronic device. Of course, it can also be that the control device records the corresponding shooting time and location when the surveillance camera shoots the place image, so as to feedback the shooting time information and location information to the electronic device.

[0042] Step S120: based on the facial information of several persons stored in the preset personnel information database, search for the target person in the multiple acquired place images to determine the place image corresponding to the first time the target person is photographed as the first target image, and the target person is the person whose facial information is stored in the personnel information database.

[0043] After acquiring the location image, the facial information of the people therein is extracted, and then the facial information recorded in the personnel information database is used as a reference, and the two are compared and identified to determine whether the person in the personnel information database exists in the acquired location image. When it is determined that the target person exists, the electronic device uses the location image where the target person is first captured as the first target image. It is conceivable that in the case where the target person stays in the location where the surveillance camera is located, there will also be multiple location images where the target person exists in the multiple images captured by the surveillance camera, and then the location image where the target person is first captured is selected as the first target image.

[0044] The facial information recorded in the personnel information database includes a feature vector corresponding to the facial features of the person. Optionally, in one embodiment, for the screening of the first target image, after the electronic device obtains the venue image, it extracts the feature vector corresponding to the facial information of the person in each venue image and records it as the first feature vector. For example, in some embodiments, the electronic device calls the algorithm for face recognition stored in the dlib library to detect the position of the face in the image, and then obtains the key points of the face and extracts the features to obtain the corresponding feature vector. Among them, the dlib library is an open source library containing a variety of machine learning, computer vision and image processing functions, which provides a variety of functions such as face detection, face recognition, posture estimation, image classification, image segmentation, etc., which facilitates developers to develop various machine learning and computer vision tasks. For example, in some embodiments, the electronic device can also detect the face in the image by loading a deep learning model, and then complete the feature extraction through the algorithm for face recognition in the dlib library.

[0045] Then, the first feature vector and the feature vector corresponding to each face information are compared to determine the first vector similarity between the first feature vector and the feature vector corresponding to each face information. For example, in some embodiments, the cosine similarity between the two feature vectors can be calculated and used as the first vector similarity. Optionally, in one embodiment, the Euclidean distance between the first feature vector and the feature vector corresponding to each face information can be calculated and used as the first vector similarity.

[0046] It is conceivable that each place image can determine the corresponding first vector similarity. When the first vector similarity satisfies the first preset threshold, that is, the first vector similarity is greater than or equal to the first preset threshold, the electronic device can determine that the target person exists in the place image. Then, according to the order of shooting time, the place image in which the target person is first photographed is selected as the first target image. For example, the same surveillance camera may capture multiple place images for the same target person. The electronic device can identify the target person in these images during face recognition. Therefore, the electronic device will be arranged in the order of shooting time (such as in chronological order), and then the place image in which the target person is first photographed is the first target image.

[0047] In this regard, by identifying people in facial recognition images, electronic devices can accurately check the acquired location images to quickly determine the time and place where the target person appears, which helps to better determine whether the target person is a member of the same gang.

[0048] Step S130, when the first target image is determined, a time window is determined based on the shooting time information corresponding to the first target image and a preset time range to search for other target persons from the place images acquired within the time window, and the place image corresponding to the first shooting of the other target persons is determined as the second target image.

[0049] The electronic device can determine the size of a time window based on a preset time range, and the time window is used to filter the place images, that is, select the place images whose shooting time is within the time window according to the time range corresponding to the time window. Therefore, in the case of determining the first target image, the electronic device can determine the specific time corresponding to the time window according to the shooting time information and time range corresponding to the first target image. Optionally, in one embodiment, the shooting time corresponding to the first target image is used as the center point of the time window, and the preset time range is combined as the size of the time window, so that the time window can be finally determined. For example, the shooting time of the first target image is 9:44:30, and the preset time range is two minutes. For this, the time interval corresponding to the time window determined by the electronic device is 9:43:30 to 9:45:30. It can be imagined that in the time dimension, the people who appear in the same time window are considered to be related in time, and the closer the time is, the higher the degree of association. Therefore, according to the time window, the place images are screened out, and the electronic device can perform preliminary screening of the place images in combination with the association in time, reducing the workload. Optionally, in another embodiment, the shooting time corresponding to the first target image may be used as the starting time point of the time window, and then the size of the time window is determined according to a preset time range.

[0050] The electronic device screens other place images according to the time span corresponding to the determined time window and the shooting time corresponding to the place image, and then searches for other target persons from the screened place images to determine the second target image. It should be noted that the time span of the time window is determined according to a preset time range. In this regard, in actual applications, the time range can be preset according to actual application requirements. In addition, the second target image and the first target image can also be the same place image, that is, two persons in the personnel information database appear in the same place image.

[0051] For example, in one embodiment, when a preset time range is determined, the shooting time information corresponding to the first target image is used as the time center, that is, the shooting time corresponding to the first target image is used as the center of the time window, and the time range is used as the size of the time window, so that the corresponding time window, such as the time span, starting point and end point of the time window in the time dimension, can be determined. In this regard, the electronic device filters the place images according to the time span corresponding to the time window, thereby filtering out the place images whose shooting time is within the time window, and extracting the second feature vector corresponding to the facial information of the person in the image from the filtered place images. It can be imagined that when the electronic device extracts the second feature vector, it can complete the feature extraction according to the algorithm for face recognition based on the dlib library in the above embodiment.

[0052] The search for other target persons is also a query of the personnel information database, that is, the second vector similarity between the second feature vector and the feature vector corresponding to the facial information of the person in the personnel information database is calculated. It can be imagined that the second vector similarity can be the cosine similarity or Euclidean distance between the above two feature vectors to determine the corresponding value. In this regard, when the electronic device determines whether each place image is the second target image, after the device calculates the corresponding second vector similarity for the person in the image, it compares the second vector similarity with the second preset threshold. When the second vector similarity is greater than or equal to the second preset threshold, it is determined that there are other target persons in the current place image, and then the place image in which the other target person is first photographed is the second target image. It can be imagined that when identifying the target person, it is the identification of other target persons, which is convenient for locking the place image where other target persons exist. For example, when the device calculates the second vector similarity, it calculates the vector similarity between the second feature vector and the feature vector corresponding to the facial information of the person other than the target person in the first target image.

[0053] Therefore, by filtering the place image through the time window, the device can filter other target persons based on the correlation in the time dimension, so as to determine the corresponding second target image, and then determine other target persons, time and place appearing within the time span corresponding to the time window, which helps to better judge whether there is a corresponding set of persons.

[0054] Step S140: determining the spatiotemporal correlation value of the target person in each second target image relative to the target person in the first target image based on the shooting time information and location information carried by each target image corresponding to a different target person.

[0055] After screening out the first target image and the second target image, it can be determined that each target image corresponds to at least one target person. Therefore, the electronic device obtains the shooting time information and location information carried by the first target image and the second target image, and then in the time dimension, the temporal correlation degree between the two target persons can be determined according to the shooting times corresponding to the first target image and the second target image; in addition, in the spatial dimension, the spatial correlation degree between the two target persons can also be determined according to the positions corresponding to the first target image and the second target image, so that the electronic device can combine the correlation degrees in time and space to determine the spatiotemporal correlation value of the target person in each second target image relative to the target person in the first target image.

[0056] Step S150: If the spatiotemporal correlation value is greater than or equal to a preset correlation threshold, the first target image and the second target image are added to the spatiotemporal cluster set, so as to determine that the target persons corresponding to the first target image and the second target image belong to the same personnel set when the personnel density value corresponding to the spatiotemporal cluster set meets the preset density threshold.

[0057] It is conceivable that the electronic device is correspondingly provided with a preset correlation threshold value, which is used to compare the spatiotemporal correlation value to screen the second target image, wherein the larger the spatiotemporal correlation value is. After comparing the spatiotemporal correlation value and the preset correlation threshold value, if the spatiotemporal correlation value is greater than or equal to the preset correlation threshold value, the device can determine that the target person in the second target image and the target person in the first target image meet the requirements of being members of the same person set, and then, the first target image and the second target image whose spatiotemporal correlation value is greater than or equal to the preset correlation threshold value are added to the spatiotemporal cluster set.

[0058] The spatiotemporal cluster set is a data set, and when the place image meets the above conditions, the place image is added to the set. Moreover, the personnel density value corresponding to the spatiotemporal cluster set corresponds to the number of place images added to the spatiotemporal cluster set. For example, if the sum of the first target image and the second target image added to the spatiotemporal cluster set is 4, it can be determined that the personnel density value corresponding to the spatiotemporal cluster set is 4. In this regard, the electronic device further determines whether the personnel density value corresponding to the spatiotemporal cluster set meets the preset density threshold. When the personnel density value corresponding to the spatiotemporal cluster set is greater than or equal to the preset density threshold, the electronic device determines that the personnel density value meets the preset density threshold, and then the target personnel corresponding to the first target image and the second target image belong to the same personnel set.

[0059] It is understandable that by setting a preset density threshold, the electronic device can determine a personnel set of 2 or more people, a personnel set of 3 or more people, etc., which has a minimum number of people, thereby achieving more flexible dynamic recognition, and it is conceivable that after the electronic device determines the members in the personnel set, it can report it to the user terminal (such as a host computer, server, etc.) as a recognition result to present the corresponding recognition result to the user. It should be noted that the above-mentioned spatiotemporal correlation value can be normalized, and accordingly, the preset correlation threshold is also normalized.

[0060] From the above scheme, it can be seen that this scheme can obtain the time and place of the target person's appearance by performing face recognition on the person in the image, and use the temporal and spatial association of the target person to more accurately and efficiently identify the association between the target persons, so as to determine the corresponding set of persons, thereby helping to improve the accuracy of identifying the association behavior between persons. In addition, the scheme also supports periodic updating of the data in the personnel information database, so as to maintain the real-time nature of the analysis results.

[0061] Figure 2 A schematic diagram of the steps for determining a spatiotemporal correlation value provided in an embodiment of the present application. In an embodiment, the electronic device can determine the degree of temporal and spatial correlation between two target persons in the first target image and the second target image in the time dimension and the space dimension according to the corresponding shooting time information and location information of the acquired first target image and the second target image, so as to determine the corresponding spatiotemporal correlation value. The specific steps include steps S210 to S230:

[0062] Step S210: Determine the time similarity between other target persons and the target person in the first target image based on the time difference between the shooting time information corresponding to the first target image and the shooting time information corresponding to the second target image.

[0063] Step S220: Determine the spatial similarity between other target persons and the target person in the first target image according to the location information of the location of the surveillance camera that photographed the first target image and the location information of the location of the surveillance camera that photographed the second target image.

[0064] Step S230: Determine the spatiotemporal correlation value according to the temporal similarity and the spatial similarity.

[0065] It can be understood that, for the degree of temporal association, the electronic device calculates the time difference between the first target image and the second target image in terms of shooting time, and then uses the time difference to determine the temporal similarity between other target persons and the target person in the first target image, so as to indicate the degree of temporal association between the two. It can be imagined that the smaller the time difference, the higher the degree of temporal association. Similarly, for the degree of spatial association, the electronic device obtains the location information of the location of the surveillance camera that shoots the first target image and the location information of the location of the surveillance camera that shoots the second target image, such as extracting the corresponding coordinates, so as to calculate the spatial distance between the two surveillance cameras, so as to indicate the spatial distance between other target persons and the target person in the first target image, and thus uses the spatial distance to determine the spatial similarity between the two. Similarly, the smaller the distance between the two, the higher the degree of spatial association.

[0066] Furthermore, the electronic device can calculate the corresponding spatiotemporal correlation value according to the determined temporal similarity and spatial similarity, for example, calculating the temporal similarity and spatial similarity according to preset weights, thereby determining the value of the spatiotemporal correlation value.

[0067] For another example, in one embodiment, for the calculation of the spatiotemporal correlation value, after determining the spatial similarity and the temporal similarity, the first parameter value and the second parameter value may be determined respectively in combination with the spatial radius and the time span corresponding to the time window, such as by using the following steps to determine the spatiotemporal correlation value:

[0068] Step S310: Determine a first parameter value based on the spatial similarity and a preset spatial radius.

[0069] Step S320: Determine a second parameter value based on the time similarity and the time span corresponding to the time window.

[0070] Step S330: Calculate the square of the first parameter value and the square of the second parameter value to the power of one half to serve as the spatiotemporal correlation value.

[0071] For example, the spatial similarity divided by the spatial radius is used as the first parameter value, and the temporal similarity divided by the time span is used as the second parameter value. Then, the sum of the squares of the first parameter value and the second parameter value is calculated, and the value of the square sum divided by the first power is used as the temporal-spatial correlation value D. st (i, j), the corresponding calculation formula is as follows:

[0072]

[0073] Among them, D(i, j) is the spatial similarity, T(i, j) is the temporal similarity, s is the spatial radius, t is the time span, and i and j represent target person i and target person j.

[0074] Therefore, by obtaining the degree of association between the target persons in time and space, that is, the above-mentioned temporal similarity and spatial similarity, this solution can more accurately and efficiently identify the association between the target persons, which helps to improve the accuracy of judging whether the target persons belong to the same set of persons.

[0075] In one embodiment, in the process of obtaining spatial similarity, the electronic device can define the similarity between the spatial positions of two target persons based on the principle of density clustering algorithm, that is, the above-mentioned spatial similarity, which can enable the scheme to achieve more accurate identification of the association between the target persons in spatial positions, thereby helping to improve the accuracy of identification.

[0076] The position of the target person is represented by the position of the surveillance camera that takes the corresponding location image. Specifically, the position of the target person in the first target image is taken as the core point, and the position information of the location of the surveillance camera corresponding to the first target image is used as the position of the core point, and the coordinates of the core point are determined. The coordinates are the coordinates on the preset coordinate system to represent the position of the core point on the preset coordinate system. Similarly, the position of the target person in the second target image is taken as the target point, and the position information of the location of the surveillance camera corresponding to the second target image is used as the position of the target point, and the coordinates of the target point are determined, and the coordinates are also the coordinates on the preset coordinate system to represent the position of the core point on the preset coordinate system. It can be understood that in order to avoid different surveillance cameras using different coordinate systems for positioning, in the process of calculating the spatial distance, the coordinates of the core point and the target point are converted to the same coordinate system, that is, the above-mentioned preset coordinate system. Of course, if all surveillance cameras use the same coordinate system, the preset coordinate system is the coordinate system used by the surveillance camera. In this regard, based on the position of the core point and the position of the target point, the spatial distance between the core point and each target point can be determined in the preset coordinate system, and then the electronic device can use the spatial distance as the spatial similarity to realize the recognition of the spatial correlation between the target persons.

[0077] After determining the first target image, the location image is screened according to the time window and a second target image containing the target person is determined therefrom. It is conceivable that the spatial similarity between the target person in the second target image and the target person in the first target image may not meet the preset correlation threshold, that is, the calculated spatial similarity is less than the preset correlation threshold, so the second target image is not added to the spatiotemporal cluster set. In one embodiment, when the above-mentioned second target image exists, that is, when there is a second target image that is not added to the spatiotemporal cluster set, the device selects one of these second target images as the new first target image, such as sorting these second target images based on their shooting time information to select the image with the earliest shooting time as the new first target image. In this regard, after determining the new first target image, the second target image is re-screened, such as repeating steps S130-S150 of the above embodiment, that is, the acquired place image is screened according to the corresponding time window and the shooting time information of the new first target image to determine the second target image therefrom, and then the corresponding spatiotemporal correlation value is calculated to facilitate determination of whether the target persons corresponding to the new first target image and the new second target image belong to another set of persons.

[0078] Therefore, the solution can further adapt to the situation where there are related behaviors between different personnel to form multiple groups of personnel by re-judging whether there is a personnel set among the target personnel outside the personnel set, which helps to improve the accuracy of identifying related behaviors between personnel, thereby performing more accurate and efficient identification.

[0079] It should be noted that, in the case where there is still a second target image that has not been added to the spatiotemporal cluster set, the device still needs to re-determine a new first target image in order to determine a new set of people.

[0080] Exemplarily, the scheme is further elaborated by taking the application scenario of gang behavior recognition for multiple objects as an example. For example, there are two subway lines passing through community Z. For example, both subway line 1 and subway line 2 have one station in the area of ​​community Z, and the stations of the two subway lines in community Z are located at different locations. For example, the station of subway line 1 in community Z is station L1, and station L1 includes entrance A1, entrance A2, exit B1 and exit B2; the station of subway line 2 in community Z is station L2, and station L2 includes entrance C1, entrance C2 and exit D1. In addition, surveillance cameras are set at each entrance and exit to capture the location image of the location.

[0081] In this regard, when person N in the personnel information database enters site L1 from entrance A1 at 9:44:30, the surveillance camera at the location captures the corresponding location image I1; when person M in the personnel information database enters site L1 from entrance A2 at 9:44:55, the surveillance camera at the location captures the corresponding location image I2; when person O in the personnel information database enters site L2 from entrance C1 at 9:45:20, the surveillance camera at the location captures the corresponding location image I3; when person P in the personnel information database enters site L1 from entrance A1 at 9:46:08, the surveillance camera at the location captures the corresponding location image I4. The electronic device can obtain the above-mentioned location image, shooting time information and location information.

[0082] Based on the facial information corresponding to different persons pre-stored in the personnel information database, the electronic device can determine the place image I1 as the image of the first person N captured by performing facial recognition on the place image, and then use the place image I1 as the first target image. Then, according to the shooting time information corresponding to the first target image and the preset time range, the time window is determined. For example, when the preset time range is two minutes and the shooting time corresponding to the first target image is used as the center point of the time window, the time interval corresponding to the determined time window is from 9:43:30 to 9:45:30. In this regard, according to this time window, other target persons (such as person M and person O) can be found in the place image I2 and the place image I3, and the place image I2 and the place image I3 are both used as the second target image.

[0083] Accordingly, the spatiotemporal correlation value K1 of person M in place image I2 relative to person N in place image I is calculated, and the spatiotemporal correlation value K2 of person O in place image I3 relative to person N in place image I is calculated. It can be imagined that the smaller the time difference between the images, the higher the temporal correlation degree, the smaller the distance between the two, and the higher the spatial correlation degree accordingly, and then the higher the temporal correlation degree and the spatial correlation degree, the larger the corresponding spatiotemporal correlation value. That is, the spatiotemporal correlation value K1 is greater than the spatiotemporal correlation value K2, and then by comparing with the preset correlation threshold, if the spatiotemporal correlation value K1 is greater than the preset correlation threshold, and the spatiotemporal correlation value K2 is less than the preset correlation threshold, then both place image I1 and place image I2 are added to the spatiotemporal cluster set, and when the preset density threshold is 2, the electronic device can consider that person N and person M belong to the same person set.

[0084] For the place image I3 that has not been added to the spatiotemporal cluster set, the device uses the place image I3 as the new first target image, and thereby redetermines a new second target image according to a time window centered on the shooting time of the place image I3. At this time, the place image I4 is located within the time window and there is a target person (such as person P) on it. For this, the place image I4 is used as the new second target place, and the corresponding spatiotemporal correlation value K3 is calculated. When the spatiotemporal correlation value K3 satisfies a preset correlation threshold, the place image I3 and the place image I4 are added to another spatiotemporal cluster set to facilitate determination of whether person O and person P belong to another person set.

[0085] It can be seen that in the identification of whether there is gang behavior between people, the corresponding target person is determined through face recognition based on the personnel information database and place image, and then the related behavior between the target persons is analyzed based on the degree of correlation between the target persons in time and space to determine whether there are people belonging to the same gang. This realizes the judgment of the related behavior of multiple objects, which helps to improve the reliability of information analysis of a single object.

[0086] Figure 3 The schematic diagram of the structure of the device for extracting personnel information based on an image provided by an embodiment of the present application is used to execute the method for extracting personnel information based on an image provided by the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method. As shown in the figure, the device includes an image acquisition module 301, a first face recognition module 302, a second face recognition module 303, a similarity recognition module 304 and a set discrimination module 305.

[0087] The image acquisition module 301 is configured to acquire location images provided by a plurality of surveillance cameras, and record shooting time information of the location images and location information of the locations where the surveillance cameras are located;

[0088] The first face recognition module 302 is configured to search for a target person in the acquired multiple location images based on the face information of several persons stored in a preset person information database, so as to determine the location image corresponding to the first time the target person is photographed as the first target image, and the target person is a person whose face information is stored in the person information database;

[0089] The second face recognition module 303 is configured to, when the first target image is determined, determine a time window according to the shooting time information corresponding to the first target image and a preset time range to search for other target persons from the location images acquired within the time window, and determine the location image corresponding to the first shooting of the other target persons as the second target image;

[0090] The similarity identification module 304 is configured to determine the spatiotemporal correlation value of the target person in each second target image relative to the target person in the first target image based on the shooting time information and location information carried by each target image corresponding to a different target person;

[0091] The set identification module 305 is configured to add the first target image and the second target image to the spatiotemporal cluster set if the spatiotemporal correlation value is greater than or equal to a preset correlation threshold, so as to determine that the target persons corresponding to the first target image and the second target image belong to the same personnel set when the personnel density value corresponding to the spatiotemporal cluster set is greater than or equal to the preset density threshold.

[0092] Based on the above embodiment, the first face recognition module 302 is specifically configured as follows:

[0093] Extracting the first feature vector corresponding to the facial information of the person in each location image;

[0094] Determine the first vector similarity between the first feature vector and feature vectors corresponding to a plurality of facial information of corresponding persons in the personnel information database;

[0095] When the first vector similarity is greater than a first preset threshold, it is determined that the target person exists in the current place image, and the place image in which the target person is captured for the first time is taken as the first target image.

[0096] Based on the above embodiment, the second face recognition module 303 is specifically configured as follows:

[0097] Based on a preset time range, taking the shooting time information corresponding to the first target image as the time center, and determining the corresponding time window;

[0098] Filtering the location images according to the time span corresponding to the time window, and extracting the second feature vector corresponding to the facial information of the person from the filtered location images;

[0099] Determine the second vector similarity between the second feature vector and feature vectors corresponding to a number of facial information of corresponding persons in the personnel information database;

[0100] When the second vector similarity is greater than the second preset threshold, it is determined that other target persons exist in the current place image, and the place image in which the other target persons are captured for the first time is taken as the second target image.

[0101] Based on the above embodiment, the similarity identification module 304 is specifically configured as follows:

[0102] Determine the time similarity between the other target persons and the target person in the first target image based on the time difference between the shooting time information corresponding to the first target image and the shooting time information corresponding to the second target image;

[0103] Determine the spatial similarity between the other target persons and the target person in the first target image according to the location information of the location where the surveillance camera that photographed the first target image is located and the location information of the location where the surveillance camera that photographed the second target image is located;

[0104] According to the temporal similarity and spatial similarity, the spatiotemporal correlation value is determined.

[0105] Based on the above embodiment, the similarity identification module 304 is further configured as follows:

[0106] Determine the position of the core point in a preset coordinate system according to the position information of the location of the surveillance camera corresponding to the first target image;

[0107] Determine the position of the target point in the preset coordinate system according to the position information of the location of the surveillance camera corresponding to the second target image;

[0108] Based on the position of the core point and the position of the target point, the spatial distance between the two points is determined to determine the spatial similarity.

[0109] Based on the above embodiment, the similarity identification module 304 is further configured as follows:

[0110] Determining a first parameter value based on the spatial similarity and a preset spatial radius;

[0111] Determining a second parameter value based on the time similarity and the time span corresponding to the time window;

[0112] The square of the first parameter value and the square of the second parameter value are calculated to the power of one half to serve as the spatiotemporal correlation value.

[0113] On the basis of the above embodiment, the device further includes a restart determination module, and the restart determination module is configured as follows:

[0114] In the case that there is a second target image that has not been added to the spatiotemporal cluster set, the image with the earliest shooting time is selected as the new first target image, and the new second target image is re-screened to determine whether the corresponding target person belongs to another person set.

[0115] It is worth noting that in the embodiment of the above-mentioned device, the modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0116] Figure 4 The structural diagram of an electronic device provided in an embodiment of the present application is used to execute the image-based personnel information extraction method provided in the above embodiment, and has the functional modules and beneficial effects corresponding to the execution method. As shown in the figure, the electronic industrial equipment includes a processor 401, a memory 402, an input device 403 and an output device 404. The number of processors 401 can be one or more, and one processor 401 is taken as an example in the figure; the processor 401, the memory 402, the input device 403 and the output device 404 can be connected by a bus or other means, and the figure takes the connection through the bus as an example. The memory 402, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the image-based personnel information extraction method in the embodiment of the present application. The processor 401 executes the corresponding various functional applications and data processing by running the software programs, instructions and modules stored in the memory 402, that is, the above-mentioned image-based personnel information extraction method is realized.

[0117] The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data recorded or created during use, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may further include a memory remotely arranged relative to the processor 401, and these remotely arranged memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0118] The input device 403 can be used to input corresponding digital or character information to the processor 401, and to generate key signal input related to the user settings and function control of the device; the output device 404 can be used to send or display key signal output related to the user settings and function control of the device.

[0119] An embodiment of the present application also provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to perform relevant operations in the image-based personnel information extraction method provided in any embodiment of the present application.

[0120] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for extracting personnel information based on an image, characterized in that: include: Acquire location images provided by a plurality of surveillance cameras, and record shooting time information of the location images and location information of the locations where the surveillance cameras are located; Based on the facial information of several persons stored in a preset personnel information database, searching for a target person in the plurality of acquired location images to determine a location image corresponding to the first capture of the target person as a first target image, wherein the target person is a person whose facial information is stored in the personnel information database; When the first target image is determined, a time window is determined according to the shooting time information corresponding to the first target image and a preset time range to search for other target persons from the location images acquired within the time window, and a location image corresponding to the first shooting of the other target persons is determined as the second target image; Determine, based on the shooting time information and location information carried by each target image corresponding to a different target person, a spatiotemporal correlation value of the target person in each second target image relative to the target person in the first target image; If the spatiotemporal correlation value is greater than or equal to a preset correlation threshold, the first target image and the second target image are added to the spatiotemporal cluster set to determine that the target persons corresponding to the first target image and the second target image belong to the same personnel set when the personnel density value corresponding to the spatiotemporal cluster set is greater than or equal to a preset density threshold.

2. The method for extracting personnel information based on an image according to claim 1, characterized in that: The method of searching for a target person in the plurality of acquired location images based on the facial information of a plurality of persons stored in a preset location information database, so as to determine a location image corresponding to the first capture of the target person as a first target image, wherein the target person is a person whose facial information is stored in the location information database, comprises: Extracting a first feature vector corresponding to facial information of a person in each of the location images; Determine a first vector similarity between the first feature vector and feature vectors corresponding to a plurality of facial information of corresponding persons in the personnel information database; When the first vector similarity is greater than a first preset threshold, it is determined that the target person exists in the current place image, and the place image in which the target person is captured for the first time is taken as the first target image.

3. The method for extracting personnel information based on an image according to claim 1 or 2, characterized in that: In the case where the first target image is determined, according to the shooting time information corresponding to the first target image and a preset time range, a time window is determined to search for other target persons from the place images acquired within the time window, and a place image corresponding to the first shooting of the other target persons is determined as the second target image, including: Based on a preset time range, taking the shooting time information corresponding to the first target image as the time center, and determining the corresponding time window; Filtering the location images according to the time span corresponding to the time window, and extracting the second feature vector corresponding to the facial information of the person in the filtered location images; Determine a second vector similarity between the second feature vector and feature vectors corresponding to a plurality of facial information of corresponding persons in the personnel information database; When the second vector similarity is greater than a second preset threshold, it is determined that other target persons exist in the current place image, and the place image in which the other target persons are captured for the first time is taken as the second target image.

4. The method for extracting personnel information based on an image according to claim 1, characterized in that: The determining, based on the shooting time information and the position information carried by each target image corresponding to a different target person, the spatiotemporal correlation value of the target person in each second target image relative to the target person in the first target image comprises: Determining the time similarity between other target persons and the target person in the first target image based on the time difference between the shooting time information corresponding to the first target image and the shooting time information corresponding to the second target image; Determine the spatial similarity between other target persons and the target person in the first target image according to the location information of the location of the surveillance camera that photographed the first target image and the location information of the location of the surveillance camera that photographed the second target image; The spatiotemporal correlation value is determined according to the temporal similarity and the spatial similarity.

5. The method for extracting personnel information based on an image according to claim 4, characterized in that: The determining of the spatial similarity between other target persons and the target person in the first target image according to the location information of the location of the surveillance camera that photographed the first target image and the location information of the location of the surveillance camera that photographed the second target image includes: Determine the position of the core point in a preset coordinate system according to the position information of the location of the surveillance camera corresponding to the first target image; Determine the position of the target point in a preset coordinate system according to the position information of the location of the surveillance camera corresponding to the second target image; Based on the position of the core point and the position of the target point, a spatial distance between the two points is determined to determine the spatial similarity.

6. The method for extracting personnel information based on an image according to claim 4 or 5, characterized in that: The step of determining the spatiotemporal correlation value according to the temporal similarity and the spatial similarity includes: Determining a first parameter value based on the spatial similarity and a preset spatial radius; Determining a second parameter value based on the time similarity and a time span corresponding to the time window; The square of the first parameter value and the square of the second parameter value are calculated to the power of half to serve as the spatiotemporal correlation value.

7. The method for extracting personnel information based on an image according to claim 1, characterized in that: If the spatiotemporal correlation value is greater than or equal to the preset correlation threshold, the first target image and the second target image are added to the spatiotemporal cluster set, so as to determine that the target persons corresponding to the first target image and the second target image belong to the same person set when the person density value corresponding to the spatiotemporal cluster set is greater than or equal to the preset density threshold, the method further includes: In the case that there is a second target image that has not been added to the spatiotemporal cluster set, the image with the earliest shooting time is selected as the new first target image, and the new second target image is re-screened to determine whether the corresponding target person belongs to another person set.

8. An image-based personnel information extraction device, characterized in that: include: An image acquisition module is configured to acquire location images provided by a plurality of surveillance cameras, and record shooting time information of the location images and location information of the locations where the surveillance cameras are located; A first face recognition module is configured to search for a target person in the plurality of acquired location images based on the face information of a plurality of persons stored in a preset personnel information database, so as to determine a location image corresponding to the first capture of the target person as a first target image, wherein the target person is a person whose face information is stored in the personnel information database; The second face recognition module is configured to, when the first target image is determined, determine a time window according to the shooting time information corresponding to the first target image and a preset time range to search for other target persons from the location images acquired within the time window, and determine the location image corresponding to the first shooting of the other target persons as the second target image; a similarity recognition module configured to determine a spatiotemporal correlation value of a target person in each of the second target images relative to a target person in the first target image based on shooting time information and location information carried by each target image corresponding to a different target person; The set identification module is configured to add the first target image and the second target image to the spatiotemporal cluster set if the spatiotemporal correlation value is greater than or equal to a preset correlation threshold, so as to determine that the target persons corresponding to the first target image and the second target image belong to the same person set when the person density value corresponding to the spatiotemporal cluster set is greater than or equal to a preset density threshold.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the image-based personnel information extraction method as described in any one of claims 1-7.

10. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, they are used to perform the image-based personnel information extraction method according to any one of claims 1 to 7.

Images