Companion identification method and apparatus based on image clustering, device, and medium

By using image clustering and quality evaluation technology in public transportation to identify peers, the problems of low efficiency and low accuracy of peer recognition in public transportation are solved, and more efficient and accurate peer recognition is achieved.

WO2024230009A1PCT designated stage expired Publication Date: 2024-11-14SHENZHEN INTELLIFUSION TECHNOLOGIES CO LTD

Patent Information

Application Number
PCT/CN2023/114662
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2023-08-24
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In public transportation, peer personnel identification efficiency and low accuracy are mainly due to processing a large amount of image data, personnel position, occlusion and other factors.

Method used

Using an image clustering method, by obtaining the video of getting on and off the car in the car, extracting face images and performing clustering processing, combining image quality evaluation, high-quality images are selected for marking and information determination, and quickly identifying peers.

Benefits of technology

It improves the efficiency and accuracy of peer personnel recognition, avoids repeated comparison and erroneous clustering results of images of the same passenger passenger, and enhances the comprehensiveness and accuracy of the recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023114662_14112024_PF_FP_ABST
    Figure CN2023114662_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of artificial intelligence, in particular to a companion identification method and apparatus based on image clustering, a device, and a medium. The method comprises: performing face extraction on image frames in getting-on and getting-off videos of each carriage to obtain face images, and clustering all the face images to obtain a cluster set; evaluating image quality of all the face images in the cluster set to obtain quality evaluation factors, using the face images having the quality evaluation factors greater than a threshold value as archived images, and marking each archived image with a carriage number; according to acquisition time points and carriage number marks of the archived images, determining getting-on and getting-off information of corresponding passengers in the cluster set; and, according to the similarity degree between the getting-on and getting-off information, determining companions of a target person. The present invention summarizes image information effectively by means of clustering processing, avoids repeated image comparison, and screens face images in combination with image quality, thus improving the reliability of archived images, and increasing the efficiency and accuracy for identifying companions.
Need to check novelty before this filing date? Find Prior Art

Description

Companion identification method, device, equipment and medium based on image clustering

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 9, 2023, with application number 202310518309.9 and invention name “Method, device, equipment and medium for peer identification based on image clustering”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for identifying peers based on image clustering. Background Art

[0003] At present, public transportation has gradually become the travel choice of most people due to its convenience and other characteristics. In order to ensure the safety of passengers, public transportation needs to be regulated, and the supervision tasks include the task of screening fellow passengers. The task of screening fellow passengers is intended to be applied to scenarios such as screening of personnel involved in infectious diseases and investigation of criminal collaborative personnel.

[0004] However, public transportation has the characteristics of large passenger flow and complex routes. In actual application, passenger information often needs to be extracted through image data collected by cameras. Obviously, the identification of fellow passengers requires processing a large amount of image data, resulting in low efficiency of fellow passenger identification. Moreover, when re-identifying people in different images, it will be affected by factors such as person posture and occlusion, resulting in low accuracy of fellow passenger identification. Therefore, how to improve the accuracy and efficiency of fellow passenger identification has become an urgent problem to be solved.

[0005] Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method, apparatus, device, and medium for identifying companions based on image clustering to solve the problem of low accuracy and efficiency in identifying companions.

[0007] In a first aspect, an embodiment of the present invention provides a method for identifying companions based on image clustering, the method comprising:

[0008] Obtain the target vehicle's stop times at each of N stations. Based on the stop times and the preset collection duration, determine the collection time period corresponding to each station. Obtain boarding and alighting videos of each carriage at each station within the corresponding collection time period, where N is an integer greater than one.

[0009] Performing face extraction on all image frames contained in all boarding and alighting videos to obtain facial images of the corresponding image frames, clustering all facial images based on similarity between the facial images to obtain at least one cluster set containing at least one facial image, with one cluster set corresponding to one passenger;

[0010] For any cluster set, perform image quality evaluation on all facial images in the cluster set to obtain a quality evaluation factor for the corresponding facial image. Take facial images with a quality evaluation factor greater than a set threshold as archived images, and mark each archived image according to the car number corresponding to the boarding and alighting video to which the archived image belongs, to obtain a car number label for each archived image in the cluster set.

[0011] Determine the boarding and alighting information of passengers corresponding to the cluster set according to the acquisition time point and carriage number mark of each archived image in the cluster set;

[0012] Obtain the boarding and alighting information corresponding to a target person in the target vehicle, and determine the target person's companions among the other passengers in the target vehicle based on the similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle.

[0013] In a second aspect, an embodiment of the present invention provides a companion identification device based on image clustering, the companion identification device comprising:

[0014] The video acquisition module is used to obtain the stop times corresponding to the target vehicle at N stations, determine the collection time period corresponding to each station based on the stop time and the preset collection time, and obtain the boarding and alighting videos of each carriage at each station within the corresponding collection time period, where N is an integer greater than one;

[0015] An image clustering module is used to extract faces from all image frames contained in all boarding and alighting videos to obtain facial images corresponding to the image frames. Based on the similarity between the facial images, all facial images are clustered to obtain at least one cluster set containing at least one facial image. One cluster set corresponds to one passenger.

[0016] An image labeling module is configured to perform image quality evaluation on all facial images in any cluster set, obtain a quality evaluation factor for the corresponding facial image, select facial images with a quality evaluation factor greater than a set threshold as archived images, and label each archived image according to the carriage number corresponding to the boarding and alighting video to which the archived image belongs, thereby obtaining a carriage number label for each archived image in the cluster set;

[0017] An information determination module, configured to determine the boarding and alighting information of passengers corresponding to the cluster set based on the acquisition time point and carriage number tag of each archived image in the cluster set;

[0018] The companion identification module is used to obtain the boarding and alighting information corresponding to a target person in the target vehicle, and to determine the companions of the target person among the other passengers in the target vehicle based on the degree of similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle.

[0019] In a third aspect, an embodiment of the present invention provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying companions as described in the first aspect is implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying companions as described in the first aspect is implemented.

[0021] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0022] Obtain the stop times corresponding to the target vehicle at N stations respectively, determine the collection time period corresponding to each station according to the stop time of each station and the preset collection time, obtain the boarding and exiting videos of each carriage at each station within the corresponding collection time period, N is an integer greater than one, perform face extraction on all image frames contained in all boarding and exiting videos respectively, obtain the face images of the corresponding image frames, perform clustering processing on all face images according to the similarity between face images, obtain at least one cluster set, each cluster set contains at least one face image, and one cluster set corresponds to one passenger, for any cluster set, perform image quality evaluation on all face images in the cluster set respectively, obtain the quality evaluation factor of the corresponding face image, take the face image with a quality evaluation factor greater than the set threshold as the archived image, and select the boarding and exiting video to which the archived image belongs. According to the carriage number of the carriage, each archived image is marked to obtain the carriage number mark of each archived image in the cluster set. According to the acquisition time point and carriage number mark of each archived image in the cluster set, the boarding and alighting information of the passengers corresponding to the cluster set is determined, and the boarding and alighting information corresponding to a target person in the target vehicle is obtained. Based on the similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle, the companions of the target person are determined. By clustering the face images, the image information of the same passenger is effectively aggregated, and repeated comparison of the images of the same passenger is avoided. In combination with the image quality evaluation, the cluster set is screened to avoid the hidden danger of retaining the wrong clustering results. The passenger information is determined based on the screened archived images, which improves the efficiency and accuracy of identifying companions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of an application environment of a method for identifying fellow travelers based on image clustering provided by a first embodiment of the present invention;

[0024] FIG2 is a flow chart of a method for identifying fellow passengers based on image clustering according to a first embodiment of the present invention;

[0025] FIG3 is a schematic structural diagram of a device for identifying companions based on image clustering according to a second embodiment of the present invention;

[0026] FIG4 is a schematic structural diagram of a computer device provided in a third embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0028] A method for identifying peers based on image clustering provided in a first embodiment of the present invention can be applied in an application environment such as that shown in FIG1 , wherein a client communicates with a server. The client includes but is not limited to a PDA, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud terminal device, a personal digital assistant (PDA), and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, a content delivery network (CDN), and big data and artificial intelligence platforms.

[0029] The client can be connected to at least one image acquisition terminal to obtain the collected boarding and alighting videos from the image acquisition terminal. The image acquisition terminal includes but is not limited to a camera, a video camera, a video recorder, a handheld photographic device, a mobile device with a photographic function, etc. The client can be used in traffic supervision scenarios to perform the task of identifying fellow passengers. The actual scenarios of the client application can be subway scenarios, train scenarios, etc. Among them, the image acquisition terminal can be deployed in the platform. In this case, one image acquisition terminal corresponds to a car in a single platform, that is, an image acquisition terminal is deployed at the stop position of each car to collect the boarding and alighting videos of the corresponding car. The image acquisition terminal can also be deployed on a vehicle. In this case, one image acquisition terminal corresponds to a car in the vehicle, that is, an image acquisition terminal is deployed at the passage of each car to collect the boarding and alighting videos of the corresponding car.

[0030] Referring to FIG2 , which is a flow chart of a method for identifying fellow passengers based on image clustering provided in the first embodiment of the present invention, the above-mentioned method for identifying fellow passengers can be applied to the client in FIG1 . The computer device corresponding to the client is connected to the image acquisition terminal to obtain the boarding and alighting videos of each carriage collected at each station from the image acquisition terminal. The computer device corresponding to the client communicates with the server to store the archived images in the server. When the client receives the fellow passenger identification task and target person sent by the supervisor, the archived images are read from the server. As shown in FIG2 , the method for identifying fellow passengers may include the following steps:

[0031] Step S201: Obtain the stop times corresponding to the target vehicle at N stations. Determine the collection time period corresponding to each station based on the stop time and the preset collection duration. Obtain the boarding and alighting video of each carriage at each station within the corresponding collection time period.

[0032] Among them, the target vehicle may refer to a vehicle that requires traffic supervision. In this embodiment, taking the subway as an example, N is an integer greater than one. The station may refer to the stop of the target vehicle, and the stop is used for passengers riding the target vehicle to get on and off. The stop time may refer to the starting time of the stay at the corresponding station. The acquisition duration may refer to the duration of the image acquisition end for acquisition. The acquisition time period includes several acquisition time points. The acquisition time point may be the moment when the image acquisition end performs image acquisition. The boarding and alighting video may refer to an image sequence composed of all images acquired by the image acquisition end within the acquisition time period.

[0033] Specifically, the preset collection time may be a fixed value. In this embodiment, the preset collection time is set to 3 minutes. Accordingly, the collection time periods are all 3 minutes.

[0034] In one embodiment, the implementer can set a different collection time for each carriage within each station to adapt to the passenger flow in different carriages. For example, the collection time corresponding to the carriage with a larger passenger flow is set to 5 minutes, and the collection time corresponding to the carriage with a smaller passenger flow is set to 3 minutes.

[0035] Optionally, based on the stop time of each station and the preset collection time, a collection time period corresponding to each station is determined, and the boarding and alighting videos of each carriage at each station within the corresponding collection time period are obtained, including:

[0036] For any station, add the stop time and collection duration of the station to obtain the collection time period corresponding to the station;

[0037] The cameras deployed at each carriage at the station are used to collect images during the corresponding collection time period of the station to obtain the boarding and alighting videos of the corresponding carriages within the station.

[0038] Traverse all stations and obtain the boarding and alighting videos of each carriage at each station during the corresponding collection time period.

[0039] The stop time and the collection duration are added together to obtain the collection end time. The stop time is used as the collection start time. The collection time period can be formed based on the collection start time and the collection end time. In this embodiment, a camera is used as the image collection end. The camera can be used to capture images of the boarding and alighting area of ​​the carriage. The position of the camera is usually fixed.

[0040] In this embodiment, multiple cameras are used to jointly collect boarding and alighting videos of each carriage to ensure the integrity of passenger information collection, avoid unclear image details when using wide-angle cameras, and improve the accuracy of subsequent identification of fellow passengers.

[0041] The above steps of obtaining the stop times corresponding to the target vehicle at N stations, determining the collection time period corresponding to each station based on the stop time and the preset collection time, and obtaining the boarding and alighting videos of each carriage at each station within the corresponding collection time period can provide effective video information for subsequent image clustering, so that the image clustering can be archived according to the passengers, thereby improving the completeness of the passenger image archiving.

[0042] Step S202 : performing face extraction on all image frames contained in all the boarding and alighting videos to obtain face images of the corresponding image frames, and performing clustering processing on all face images based on the similarity between the face images to obtain at least one cluster set.

[0043] Among them, the image frame can refer to a single frame image that constitutes the boarding and disembarking video, the face image can refer to an image including the face area, the similarity can be used to characterize the degree of difference between face images, the face cluster set contains at least one face image, and one cluster set corresponds to one passenger.

[0044] Specifically, the similarity can be measured using Euclidean distance, cosine similarity, etc. In this embodiment, cosine similarity is used to calculate the similarity. The value range of cosine similarity is [0, 1]. The closer the cosine similarity value is to 1, the greater the similarity between the face images. The closer the cosine similarity value is to 0, the smaller the similarity between the face images.

[0045] Optionally, face extraction is performed on all image frames contained in all the boarding and alighting videos to obtain face images of the corresponding image frames, including:

[0046] Input each image frame into the trained face detection model to detect the face area and output the bounding box of the corresponding face area;

[0047] For any bounding box, the bounding box is extracted from the image frame to which it belongs to obtain the face image of the bounding box, and all bounding boxes are traversed to obtain all face images.

[0048] Among them, the trained face detection model can adopt the trained target detection model, and the target detection model can adopt an SSD model, a YOLO model, a Faster-RCNN model, etc. The bounding box can be used to represent the location information of the face area. The bounding box can be represented by a coordinate pair, and the coordinate pair can include the coordinates of the upper left corner point and the lower right corner point of the bounding box.

[0049] Specifically, after obtaining the bounding box, the face area can be extracted by multiplying the mask image and the image frame point by point to obtain the face image. In the mask image, the pixel values ​​of the pixels contained in the bounding box area are 1, and the pixel values ​​of other pixels are 0. The mask image and the image frame are the same size.

[0050] It should be noted that a single image frame may include several face regions, and each face region may be extracted separately to obtain a face image of the corresponding face region.

[0051] In this embodiment, facial images are extracted so that each facial image contains only facial region information of a single person, which facilitates subsequent facial similarity calculation, isolates irrelevant conditions, and improves the accuracy of facial image similarity calculation.

[0052] Optionally, after obtaining the face image of the corresponding image frame, the method further includes:

[0053] Each image frame is input into the trained human body detection model to extract the human body region, and obtain the human body image corresponding to the initial image;

[0054] For any image frame, if the intersection-over-union ratio between any human image and any facial image in the image frame is greater than a preset intersection-over-union ratio threshold, it is determined that there is an association relationship between the human image and the facial image in the image frame;

[0055] Accordingly, all face images are clustered according to the similarity between them to obtain at least one cluster set, including:

[0056] Clustering is performed on all face images based on the similarities between face images and the similarities between human body images associated with the face images to obtain at least one cluster set.

[0057] Among them, the trained human body detection model can use the same model architecture as the trained face detection model, but the labels corresponding to the data sets used in the training of the two are different. The corresponding labels of the samples of the human body detection model are the reference human body areas, and the corresponding labels of the samples of the face detection model are the reference face areas.

[0058] The intersection-in-union (IoU) can be used to characterize the degree of overlap between the face region and the body region. The value range of the IoU is [0, 1]. The closer the IoU is to 1, the greater the degree of overlap between the face region and the body region. The closer the IoU is to 0, the smaller the degree of overlap between the face region and the body region.

[0059] The preset intersection-over-union threshold can be used to measure whether the degree of overlap between the face region and the body region is large enough, so as to consider that the face region and the body region belong to the same passenger, that is, there is an association between the face image and the body image.

[0060] Specifically, in this embodiment, the preset IoU threshold may be set to 0.5, and the implementer may adjust the preset IoU threshold according to actual conditions.

[0061] For any two face images, the similarity between the two face images is first calculated to obtain the first similarity. If both face images have associated human images, the similarity between the two human images is then calculated to obtain the second similarity. The weighted average of the first and second similarities is used as the final similarity for clustering. If the two face images have no associated human images, the first similarity is used as the final similarity for clustering.

[0062] Clustering can be performed using a DBSCAN clustering algorithm based on density clustering, thereby completing clustering when the number of cluster sets is unknown.

[0063] In this embodiment, the similarity of facial images is corrected by the similarity between body images, thereby improving the accuracy of the clustering process and making the clustering result more reliable.

[0064] The above-mentioned steps include performing face extraction on all image frames contained in all boarding and exiting videos respectively to obtain face images of corresponding image frames, clustering all face images according to the similarity between the face images, and obtaining at least one cluster set, so that all face images of a single passenger exist in the same cluster set, which facilitates the subsequent determination of the corresponding passenger's boarding information according to the cluster set, thereby improving the accuracy of the determination of the boarding information.

[0065] In step S203, for any cluster set, image quality evaluation is performed on all facial images in the cluster set to obtain a quality evaluation factor of the corresponding facial image. Facial images with a quality evaluation factor greater than a set threshold are used as archived images. Each archived image is marked according to the car number corresponding to the boarding and alighting video to which the archived image belongs, to obtain a car number mark for each archived image in the cluster set.

[0066] Among them, the quality evaluation factor can be used to characterize the clarity, occlusion, etc. of the facial image, and the set threshold can be used to measure whether the quality of the facial image is high enough, thereby serving as a factor in determining the riding information. The archived image can refer to the facial image retained after being screened by the quality evaluation factor, the car number can refer to the serial number identification of the car, and the car number mark can be used to indicate the car number where the archived image was collected.

[0067] Specifically, image quality evaluation can adopt a pre-trained image quality evaluation model. The image quality evaluation model can include an encoder and a fully connected layer. The encoder can be used to extract image features of the input image, and the fully connected layer can be used to map the image features to the output feature space. The task type performed by the fully connected layer is a regression task, and the output predicted quality evaluation factor has a value range of [0, 1]. The closer the quality evaluation factor is to 1, the greater the possibility that the face area belongs to the corresponding passenger. The closer the quality evaluation factor is to 0, the smaller the possibility that the face area belongs to the corresponding passenger.

[0068] Optionally, performing image quality evaluation on all face images in the cluster set to obtain quality evaluation factors of the corresponding face images, and taking face images with quality evaluation factors greater than a set threshold as archive images, including:

[0069] Perform image quality evaluation on all face images in the cluster set and obtain the quality evaluation factor of the corresponding face image;

[0070] Calculate the mean of all quality evaluation factors, multiply the mean by the preset coefficient, and determine the multiplication result as the set threshold;

[0071] Face images whose quality evaluation factor is greater than the set threshold are regarded as archive images.

[0072] The preset coefficient may represent a dynamic weight for determining a set threshold. In this embodiment, the preset coefficient may be set to 0.5.

[0073] In this embodiment, the threshold is dynamically determined based on the quality evaluation of all facial images in the cluster set, thereby avoiding the omission of subsequent identification of fellow passengers due to too few or even no archived images of some passengers due to setting a fixed threshold, thereby improving the comprehensiveness and accuracy of identification of fellow passengers.

[0074] The above steps are as follows: for any cluster set, image quality evaluation is performed on all facial images in the cluster set respectively to obtain the quality evaluation factor of the corresponding facial image, and the facial image with a quality evaluation factor greater than the set threshold is used as the archived image. Each archived image is marked according to the car number of the car corresponding to the boarding and alighting video to which the archived image belongs, and the car number marking of each archived image in the cluster set is obtained. The facial images of the same passenger are clustered to avoid repeated comparison of the facial images of the same passenger when subsequent fellow passengers are identified, thereby improving the efficiency of subsequent fellow passengers identification.

[0075] Step S204: Determine the boarding and alighting information of passengers corresponding to the cluster set according to the acquisition time point and carriage number tag of each archived image in the cluster set.

[0076] The acquisition time point may refer to the time point at which the archived image is acquired, and the boarding and alighting information may include the boarding car number, the alighting car number, the boarding time point, and the alighting time point.

[0077] Specifically, the implementer can also use the entry and exit videos of the station entrance and exit. In this case, the images included in the entry and exit videos are all front-facing images. Through the similarity between the front-facing images and the archived images in each cluster set, the passengers corresponding to the front-facing images can be determined, and then the entry and exit times of the passengers can be obtained.

[0078] Based on the passenger's entry and exit time, the passenger's identity information can be obtained from the real-time stored ticketing data. The identity information may include a boarding code, a financial card, a subway card with identity registration, etc., so as to facilitate subsequent identification and access to the companions.

[0079] Optionally, based on the acquisition time point and carriage number tag of each archived image in the cluster set, determining the boarding and alighting information of passengers corresponding to the cluster set includes:

[0080] Arrange all archived images in the cluster set according to the chronological order of acquisition time points to obtain an archived image sequence;

[0081] According to the carriage number mark corresponding to the archived image and the archived image sequence, a carriage number sequence is obtained, wherein the carriage number sequence includes a zero carriage number, and the zero carriage number corresponds to the passenger in the archived image not being in the carriage;

[0082] In the car number sequence, two car numbers with car number changes and continuous time sequence are formed into a car number group to obtain at least one car number group;

[0083] If any carriage number group contains carriage number zero and the carriage number zero is in the first place in the time sequence, the non-zero carriage number in the carriage number group is determined to be the boarding carriage number, and the acquisition time point of the archived image corresponding to the boarding carriage number is used as the boarding time point;

[0084] If any car number group contains car number zero and the car number zero is at the end in time sequence, then the non-zero car number in the car number group is determined to be the getting-off car number, and the acquisition time point of the archived image corresponding to the getting-off car number is taken as the getting-off time point, and the getting-off car number, getting-off car number, getting-on time point and getting-off time point are used as the getting-off information.

[0085] The archived image sequence may refer to a sequence formed by all archived images in a single cluster set in the order of acquisition time, and the car number sequence may refer to a sequence formed by the car numbers corresponding to each archived image in the archived image sequence.

[0086] The car number group may include two car numbers, the collection time points corresponding to the car numbers included in the car number group are continuous, and the car numbers included in the car number group are different.

[0087] Specifically, when the archived image changes from a zero car number to a non-zero car number for the first time, it means that a passenger has boarded the train at this time, and when the archived image changes from a non-zero car number to a zero car number for the first time, it means that a passenger has gotten off the train at this time. By obtaining the time and car corresponding to the boarding operation, as well as the time and car corresponding to the getting off operation, the passenger's getting on and off information can be obtained.

[0088] In this embodiment, by performing time series analysis on all archived images in the cluster set, the boarding and alighting information of passengers can be determined quickly and accurately, thereby improving the accuracy of determining the boarding and alighting information of passengers.

[0089] The above steps of determining the boarding and alighting information of the passengers corresponding to the cluster set based on the acquisition time point and carriage number mark of each archived image in the cluster set provide the boarding and alighting information of each passenger for subsequent identification of fellow passengers, so that the boarding and alighting information can be directly compared to determine the passenger of the target person, thereby improving the accuracy and efficiency of identification of fellow passengers.

[0090] Step S205, obtain the boarding and alighting information corresponding to a target person in the target vehicle, and determine the target person's companions among the other passengers in the target vehicle based on the similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle.

[0091] Among them, the target person may refer to a passenger who needs to be identified as a fellow passenger. The target person may be a virus carrier, a criminal suspect, etc. The degree of similarity can be used to characterize the degree of difference in the riding information between the two passengers. The preset threshold can be used to measure whether the riding information between the two passengers satisfies a fellow passenger relationship.

[0092] Specifically, in this embodiment, the preset threshold may be set to 0.6, and the implementer may adjust the preset threshold according to actual conditions.

[0093] Optionally, obtaining boarding and alighting information corresponding to a target person in the target vehicle, and determining the target person's companions from among the other passengers in the target vehicle based on the similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle includes:

[0094] Determine the travel time interval of each passenger based on the boarding time point and alighting time point of each passenger, and determine the travel interval of each passenger based on the boarding car number and alighting car number of each passenger;

[0095] Perform intersection and union calculations on the travel time intervals corresponding to each other passenger in the target vehicle and the travel time interval corresponding to the target person, to obtain a first overlap result for each other passenger in the target vehicle;

[0096] Calculate the intersection and union ratio of the passenger movement interval corresponding to each other passenger in the target vehicle and the passenger movement interval corresponding to the target person, and obtain a second overlap result for each other passenger in the target vehicle;

[0097] The weighted average of the first overlap result and the second overlap result of each other passenger is used as the similarity between each other passenger and the target person, and all other passengers corresponding to the similarity greater than the preset threshold are determined as fellow passengers.

[0098] Among them, the riding time interval may refer to the riding time period of the passengers, and the riding movement interval may refer to the range of movement of the carriage of the passengers during the riding process.

[0099] The first overlap result can represent the overlap between the target person and the corresponding other passengers in the riding time period, and the first overlap result can represent the overlap between the target person and the corresponding other passengers in the moving range of the riding compartment.

[0100] Specifically, the weights corresponding to the first coincidence result and the second coincidence result when calculating the weighted mean may both be set to 0.5.

[0101] In one embodiment, for the screening of companions with high real-time requirements, such as the identification of companions of a criminal suspect, the product of the first overlap result and the second overlap result can be used as the degree of similarity between the corresponding other passengers and the target person. At this time, the passengers' time and space conditions must all overlap to be considered as having a companion relationship.

[0102] In this embodiment, by constructing intervals and performing intersection-over-union calculations, the overlap of passengers in time and space is obtained, thereby accurately determining whether there is a traveling relationship between passengers, thereby improving the accuracy of traveling companion identification.

[0103] The above-mentioned step of obtaining the boarding and alighting information corresponding to a target person in a target vehicle, and determining the target person's companions among other passengers in the target vehicle based on the degree of similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle, can quickly and accurately determine the companions who have a traveling relationship with the target person based on the direct comparison of the boarding and alighting information.

[0104] In this embodiment, by clustering facial images, the image information of the same passenger is effectively aggregated, avoiding repeated comparison of the same passenger's images, and combining with image quality evaluation, the cluster set is screened to avoid the hidden danger of retaining erroneous clustering results. The passenger information is determined based on the screened archived images, thereby improving the efficiency and accuracy of identifying fellow passengers.

[0105] Corresponding to the companion identification method based on image clustering in the above embodiment, Figure 3 shows a structural block diagram of the companion identification device based on image clustering provided in the second embodiment of the present invention. The above-mentioned companion identification device is applied to the client, and the computer device corresponding to the client is connected to the image acquisition end to obtain the boarding and getting-on videos of each carriage collected at each station from the image acquisition end. The computer device corresponding to the client communicates with the server to store the archived images in the server. When the client receives the companion identification task and target person sent by the supervisor, the archived image is read from the server. For the sake of convenience, only the part related to the embodiment of the present invention is shown.

[0106] Referring to FIG3 , the companion identification device includes:

[0107] The video acquisition module 31 is used to obtain the stop times corresponding to the target vehicle at N stations, determine the collection time period corresponding to each station based on the stop time and the preset collection time period, and obtain the boarding and alighting videos of each carriage at each station within the corresponding collection time period, where N is an integer greater than one;

[0108] An image clustering module 32 is configured to extract faces from all image frames contained in all boarding and alighting videos to obtain facial images corresponding to the image frames, and cluster all facial images based on similarity between the facial images to obtain at least one cluster set containing at least one facial image, with one cluster set corresponding to one passenger.

[0109] An image labeling module 33 is configured to perform image quality evaluation on all facial images in any cluster set, obtain a quality evaluation factor for the corresponding facial image, select facial images with a quality evaluation factor greater than a set threshold as archived images, and label each archived image according to the car number corresponding to the boarding and alighting video to which the archived image belongs, thereby obtaining a car number label for each archived image in the cluster set;

[0110] An information determination module 34 is used to determine the boarding and alighting information of passengers corresponding to the cluster set based on the acquisition time point and carriage number tag of each archived image in the cluster set;

[0111] The companion identification module 35 is used to obtain the boarding and alighting information corresponding to a target person in the target vehicle, and to determine the companions of the target person among the other passengers in the target vehicle based on the similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle.

[0112] Optionally, the video acquisition module 31 includes:

[0113] A time period determination unit is used to add the stop time and the collection duration of any station to obtain the collection time period of the corresponding station;

[0114] An image acquisition unit is used to acquire images using cameras deployed at each carriage at the station during a corresponding acquisition period at the station to obtain boarding and alighting videos of the corresponding carriage at the station;

[0115] The station traversal unit is used to traverse all stations and obtain the boarding and alighting videos of each carriage at each station within the corresponding collection time period.

[0116] Optionally, the image clustering module 32 includes:

[0117] A face detection unit is used to input each image frame into a trained face detection model to detect the face area and output a bounding box corresponding to the face area;

[0118] The face extraction unit is used to extract any bounding box from the image frame to which it belongs, obtain the face image of the bounding box, and traverse all bounding boxes to obtain all face images.

[0119] Optionally, the image clustering module 32 further includes:

[0120] A human body extraction unit is used to input each image frame into the trained human body detection model to extract the human body region and obtain a human body image corresponding to the initial image;

[0121] an image association unit, configured to determine, for any image frame, that an association relationship exists between the human image and the facial image in the image frame if an intersection-over-union ratio (IoU) between any human image and any facial image in the image frame is greater than a preset IoU threshold;

[0122] Accordingly, the image clustering module 32 includes:

[0123] The clustering processing unit is used to perform clustering processing on all face images according to the similarity between face images and the similarity between human body images associated with the face images to obtain at least one cluster set.

[0124] Optionally, the image marking module 33 includes:

[0125] An image evaluation unit is used to evaluate the image quality of all face images in the cluster set and obtain a quality evaluation factor of the corresponding face image;

[0126] A threshold determination unit, configured to calculate the mean of all quality evaluation factors, multiply the mean by a preset coefficient, and determine the multiplication result as a set threshold;

[0127] The threshold comparison unit is used to take the face image whose quality evaluation factor is greater than the set threshold as the archive image.

[0128] Optionally, the information determination module 34 includes:

[0129] An image arrangement unit, configured to arrange all archived images in the cluster set in chronological order of acquisition time points to obtain an archived image sequence;

[0130] The carriage number arrangement unit is used to obtain a carriage number sequence based on the carriage number mark corresponding to the archived image and the archived image sequence, wherein the carriage number sequence includes a zero carriage number, which indicates that the passenger in the archived image is not in the carriage;

[0131] a car number grouping unit, configured to form a car number group from two car numbers that have car number changes and are consecutive in time in the car number sequence, thereby obtaining at least one car number group;

[0132] a boarding information determining unit, configured to determine, if any car number group contains a car number zero and the car number zero is at the front in time sequence, that the non-zero car number in the car number group is the boarding car number, and to use the acquisition time point of the archived image corresponding to the boarding car number as the boarding time point;

[0133] The getting-off information determination unit is used to determine the non-zero car number in the car number group as the getting-off car number if any car number group contains the zero car number and the zero car number is at the end in time sequence, and to use the acquisition time point of the archived image corresponding to the getting-off car number as the getting-off time point, and the getting-off car number, getting-off car number, getting-on time point and getting-off time point as the getting-off information.

[0134] Optionally, the peer identification module 35 includes:

[0135] An interval determination unit is used to determine the travel time interval of each passenger based on the boarding time point and the alighting time point corresponding to each passenger, and to determine the travel interval of each passenger based on the boarding car number and the alighting car number corresponding to each passenger;

[0136] A first coincidence comparison unit is configured to perform intersection and union calculation on the travel time interval corresponding to each other passenger in the target vehicle and the travel time interval corresponding to the target passenger, to obtain a first coincidence result for each other passenger;

[0137] A second overlap comparison unit is configured to perform intersection and union calculation on the travel interval corresponding to each other passenger in the target vehicle and the travel interval corresponding to the target passenger, to obtain a second overlap result for each other passenger;

[0138] The companion determination unit is used to use the weighted average of the first overlap result and the second overlap result of each other passenger as the similarity between each other passenger and the target person, and to determine all other passengers corresponding to the similarity greater than a preset threshold as companions.

[0139] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0140] FIG4 is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. As shown in FIG4 , the computer device of this embodiment includes: at least one processor (only one is shown in FIG4 ), a memory, and a computer program stored in the memory and executable by the at least one processor. When the processor executes the computer program, the steps of any of the above-mentioned embodiments of the method for identifying a fellow traveler are implemented.

[0141] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that FIG4 is merely an example of a computer device and does not limit the computer device. The computer device may include more or fewer components than shown, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0142] The processor may be a CPU, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or any conventional processor, etc.

[0143] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the computer device, and in other embodiments, it can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory can also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or is about to be output.

[0144] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0145] The present invention may implement all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0146] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for identifying peers based on image clustering, characterized in that: The method for identifying fellow travelers comprises: Obtain the stop times of the target vehicle at N stations respectively, determine the collection time period corresponding to each station according to the stop time of each station and the preset collection time, and obtain the boarding and alighting videos of each carriage at each station within the corresponding collection time period, where N is an integer greater than one; Performing face extraction on all image frames contained in all boarding and alighting videos to obtain face images of corresponding image frames, and clustering all face images according to the similarity between face images to obtain at least one cluster set, wherein the cluster set contains at least one face image, and one cluster set corresponds to one passenger; For any cluster set, image quality evaluation is performed on all face images in the cluster set to obtain quality evaluation factors of the corresponding face images, and face images with quality evaluation factors greater than a set threshold are used as archived images. Each archived image is marked according to the car number of the car corresponding to the boarding and alighting video to which the archived image belongs, to obtain the car number mark of each archived image in the cluster set; Determine the boarding and alighting information of passengers corresponding to the cluster set according to the acquisition time point and carriage number mark of each archived image in the cluster set; Obtain boarding and alighting information corresponding to a target person in the target vehicle, and determine companions of the target person among the other passengers in the target vehicle based on the degree of similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle.

2. The method for identifying companions according to claim 1, characterized in that: The method of determining the collection time period corresponding to each station according to the stop time of each station and the preset collection time period, and obtaining the boarding and alighting videos of each carriage in the corresponding collection time period at each station includes: For any station, the stop time of the station and the collection duration are added to obtain the collection time period corresponding to the station; The cameras deployed at the station in each carriage are used to collect images during the collection time period corresponding to the station to obtain the boarding and alighting video of the corresponding carriage in the station; Traverse all stations and obtain the boarding and alighting videos of each carriage in the corresponding collection time period at each station.

3. The method for identifying companions according to claim 1, characterized in that: The face extraction is performed on all image frames contained in all the boarding and alighting videos to obtain the face images of the corresponding image frames, including: Input each image frame into the trained face detection model to detect the face area and output the bounding box of the corresponding face area; For any bounding box, the bounding box is used to extract from the image frame to which it belongs to obtain the face image of the bounding box, and all bounding boxes are traversed to obtain all face images.

4. The method for identifying companions according to claim 1, characterized in that: After obtaining the face image corresponding to the image frame, the method further includes: Input each image frame into the trained human body detection model to extract the human body region, and obtain the human body image corresponding to the initial image; For any image frame, if the intersection-and-union ratio between any human image and any facial image in the image frame is greater than a preset intersection-and-union ratio threshold, it is determined that there is an association relationship between the human image and the facial image in the image frame; Accordingly, the method performs clustering processing on all face images according to the similarity between the face images to obtain at least one cluster set, including: According to the similarity between the face images and the similarity between the human body images associated with the face images, clustering processing is performed on all the face images to obtain at least one cluster set.

5. The method for identifying companions according to claim 1, characterized in that: The step of performing image quality evaluation on all face images in the cluster set to obtain quality evaluation factors of corresponding face images, and taking face images with quality evaluation factors greater than a set threshold as archived images, comprises: Performing image quality evaluation on all face images in the cluster set to obtain quality evaluation factors of the corresponding face images; Calculating the mean of all quality evaluation factors, multiplying the mean by a preset coefficient, and determining the multiplication result as the set threshold; The face image whose quality evaluation factor is greater than the set threshold is used as the archived image.

6. The method for identifying a companion according to any one of claims 1 to 5, characterized in that: Determining the boarding and alighting information of passengers corresponding to the cluster set according to the acquisition time point and the carriage number mark of each archived image in the cluster set includes: Sort all archived images in the cluster set according to the chronological order of the acquisition time points. Column, get the archive image sequence; According to the carriage number mark corresponding to the archived image and the archived image sequence, a carriage number sequence is obtained, wherein the carriage number sequence includes a zero carriage number, and the zero carriage number indicates that the passenger in the archived image is not in the carriage; In the car number sequence, two car numbers with car number changes and continuous in time sequence are formed into a car number group to obtain at least one car number group; If any car number group contains the zero car number and the zero car number is in the front in time sequence, then the non-zero car number in the car number group is determined as the boarding car number, and the acquisition time point of the archived image corresponding to the boarding car number is taken as the boarding time point; If any car number group contains the zero car number and the zero car number is at the end in time sequence, the non-zero car number in the car number group is determined as the getting-off car number, and the acquisition time point of the archived image corresponding to the getting-off car number is used as the getting-off time point, and the getting-on car number, the getting-off car number, the getting-on time point and the getting-off time point are used as the getting-on and getting-off information.

7. The method for identifying companions according to any one of claim 6, characterized in that: The determining of the companions of the target person by other passengers in the target vehicle based on the similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle comprises: Determine the riding time interval of each passenger according to the boarding time point and the alighting time point corresponding to each passenger, and determine the riding movement interval of each passenger according to the boarding carriage number and the alighting carriage number corresponding to each passenger; Perform intersection and union calculations on the boarding time intervals corresponding to each other passenger in the target vehicle and the boarding time intervals corresponding to the target person, respectively, to obtain a first overlap result for each other passenger; Perform intersection and union calculations on the riding movement intervals corresponding to each other passenger in the target vehicle and the riding movement intervals corresponding to the target person, respectively, to obtain a second overlap result for each other passenger; The weighted average of the first overlap result and the second overlap result of each other passenger is used as the similarity between each other passenger and the target person, and all other passengers corresponding to the similarity greater than a preset threshold are determined as the fellow passengers.

8. A device for identifying fellow travelers based on image clustering, characterized in that: The companion identification device comprises: The video acquisition module is used to obtain the stop times of the target vehicle at N stations respectively, determine the collection time period corresponding to each station according to the stop time of each station and the preset collection time, and obtain the boarding and alighting videos of each carriage at each station within the corresponding collection time period, where N is an integer greater than one; An image clustering module is used to extract faces from all image frames contained in all boarding and alighting videos, obtain face images of corresponding image frames, perform clustering processing on all face images according to the similarity between face images, and obtain at least one cluster set, wherein the cluster set contains at least one face image, and one cluster set corresponds to one passenger; An image marking module is used to evaluate the image quality of all face images in any cluster set, obtain the quality evaluation factor of the corresponding face image, take the face image with a quality evaluation factor greater than a set threshold as an archived image, and mark each archived image according to the car number of the car corresponding to the boarding and alighting video to which the archived image belongs, so as to obtain the car number mark of each archived image in the cluster set; An information determination module, used to determine the boarding and alighting information of passengers corresponding to the cluster set according to the acquisition time point and carriage number mark of each archived image in the cluster set; The companion identification module is used to obtain the boarding and alighting information corresponding to a target person in the target vehicle, and to determine the companions of the target person among the other passengers in the target vehicle based on the degree of similarity between the boarding and alighting information corresponding to the target person and the boarding and alighting information corresponding to other passengers in the target vehicle.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the companion identification method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the companion identification method as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Peer identification method, device, system and computer storage medium

    CN109117714A

  • Facial recognition technology-based walkman analysis method and facial recognition technology-based walkman analysis system

    CN110084103A

  • Personnel set peer information mining method and device and electronic equipment

    CN111274287A

  • Method and device for mining persons in same track and electronic equipment

    CN113568941A

  • Image processing method and device

    CN114155576A

Cited By

  • Bus passenger re-identification control method and system

    CN120748014A