A monitoring method and system based on face recognition and license plate recognition

Abstract

The embodiment of the present invention provides a monitoring method and system based on face recognition and license plate recognition, which belongs to the technical field of traffic violation detection. The monitoring method includes: obtaining detection images under different viewing angles of the monitoring section; inputting different detection images into a non-motor vehicle target detection network to obtain non-motor vehicle coordinate information in the detection image; fusing non-motor vehicle cut images of different viewing angles according to the non-motor vehicle coordinate information; determining the number of people in the current vehicle according to the fused non-motor vehicle cut image; when the number of people is greater than a preset threshold, inputting the non-motor vehicle cut image into the face recognition network, and determining the face information according to the database; inputting the non-motor vehicle cut image into the license plate recognition network; determining the final illegal person information according to the face information and license plate information, and reporting it. The monitoring method can quickly and efficiently determine the information of non-motor vehicle illegal personnel and report it.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic violation detection, and in particular to a monitoring method and system based on face recognition and license plate recognition. Background Art

[0002] In cities, with the rapid growth of the number of motor vehicles, non-motor vehicles have become an indispensable part of urban or rural road traffic. Illegal non-motor vehicle carrying of passengers occurs from time to time, posing a serious hidden danger to traffic safety. Traditional manual patrol or fixed monitoring systems are often unable to effectively detect and record these violations, so an efficient and accurate non-motor vehicle illegal carrying of passengers monitoring method is needed. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a monitoring method and system based on face recognition and license plate recognition, which can quickly and efficiently determine the information of non-motor vehicle illegal personnel and report them.

[0004] In order to achieve the above objectives, on the one hand, an embodiment of the present invention provides a monitoring method based on face recognition and license plate recognition, the monitoring method comprising:

[0005] Obtain detection images from different viewing angles of the monitored road section;

[0006] Inputting different detection images into a non-motor vehicle target detection network to obtain non-motor vehicle coordinate information in the detection images;

[0007] fusing non-motor vehicle cut images from different perspectives according to the non-motor vehicle coordinate information;

[0008] Determine the number of people in the current vehicle according to the fused non-motor vehicle cut map;

[0009] When the number of people is greater than a preset threshold, the non-motor vehicle cut image is input into a face recognition network, and face information is determined according to a database;

[0010] Inputting the non-motor vehicle cut image into a license plate recognition network to obtain license plate information;

[0011] The final information of the offender is determined based on the facial information and license plate information and reported.

[0012] Optionally, obtaining detection images of the monitored road section from different viewing angles includes:

[0013] The monitoring information is displayed in real time through the front-end system, and the camera rtsp address A and rtsp address B of the monitoring section are set as two viewing angle cameras in the front-end page;

[0014] According to the two set viewing angle cameras, the corresponding real-time video streams are obtained respectively;

[0015] According to the corresponding real-time video stream, the screen image of the corresponding camera is decoded, that is, the detection screen under different viewing angles.

[0016] Optionally, fusing non-motor vehicle cut images from different perspectives according to the non-motor vehicle coordinate information includes:

[0017] The feature point detection network is trained using a simple geometric shape data set to obtain an initial feature point detection network;

[0018] Get real training images;

[0019] The real training image is transformed by multiple composite geometric transformations, where the number of transformations is marked as 1, ..., N, the first one is the original image, and the remaining N-1 images are images formed by randomly generated composite geometric transformations;

[0020] Extracting feature points from the real training image through the initial feature point detection network;

[0021] The initial feature point detection network maps N-1 images corresponding to the real training image back to the feature points of the original image and accumulates them to form new feature points of the original image, thereby completing the self-labeling of the feature points of the real training image;

[0022] Inputting the real training image and the feature point self-annotation into the feature point detection network for training to obtain an enhanced feature point network;

[0023] Inputting non-motor vehicle cut images of different perspectives corresponding to the non-motor vehicle coordinate information into the enhanced feature point network to obtain a corresponding feature point descriptor set;

[0024] Determine a subset of the feature point descriptions, and determine the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance between a feature point of another subset of the feature point descriptions and its feature point;

[0025] When the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance is less than a set threshold, the feature point is determined to be a matching point, and the fusion of the non-motor vehicle cut graph is achieved based on the matching point.

[0026] Optionally, the composite geometric transformation formula of multiple composite geometric transformations is formula (1):

[0027] , formula (1)

[0028] in, A point in the image from the first perspective, Represents the image from another perspective The corresponding points, represents the composite geometric transformation matrix, is the transformation coefficient.

[0029] Optionally, when the number of people is greater than a preset threshold, the non-motor vehicle cut image is input into a face recognition network, and the face information is determined according to a database, including:

[0030] Inputting the non-motor vehicle cut image into a face network to obtain face coordinate information;

[0031] Acquire a face cut image according to the face coordinate information;

[0032] Sending the face cut image to a face feature extraction network to obtain face features;

[0033] The facial features are matched with the facial information in the database to determine final personnel information.

[0034] Optionally, matching the facial features with facial information in the database to determine final personnel information includes:

[0035] Inputting the face cut image into a face feature extraction network to obtain a face feature vector to be matched;

[0036] Inputting all the face images to be matched in the database into the face feature extraction network to obtain their respective face feature vectors;

[0037] Divide all the facial feature vectors into m groups, each group having n facial feature vectors;

[0038] Splicing each group of facial feature vectors into a feature matrix, wherein the number of rows of the feature matrix is ​​equal to the number of columns of the facial feature vectors to be matched;

[0039] Multiplying the face feature vector to be matched with the feature matrix of m groups to obtain a similarity score of the database comparison of each group;

[0040] Take the maximum similarity score of each group and keep the record vector column index;

[0041] Take the maximum value and corresponding column index of all the maximum similarity scores of each group;

[0042] Determining whether the maximum value is greater than a similarity score threshold;

[0043] In the case that the maximum value is greater than the similarity score threshold, the personnel information of the column index corresponding to the maximum value is obtained as the final personnel information.

[0044] Optionally, the non-motor vehicle target detection network is a yolov5n model, and the actual collected image annotations are used as a training set to optimize the yolov5n model, and then the non-motor vehicle in the detection picture is identified and located by the optimized yolov5n model.

[0045] Optionally, the license plate recognition network is an OCR model.

[0046] Optionally, determining the final offender information based on the facial information and license plate information and reporting it includes:

[0047] Determine whether the license plate information and final personnel information are obtained at the same time;

[0048] When the license plate information and the final person information are obtained at the same time, the person corresponding to the license plate information and the final person information are both listed as the final illegal person;

[0049] In the case where the license plate information and the final person information are not obtained at the same time, the person corresponding to the obtained license plate information or the final person information is listed as the final violator.

[0050] On the other hand, the present invention also provides a monitoring system based on face recognition and license plate recognition, the monitoring system comprising:

[0051] Camera, used to obtain detection images from different viewing angles of the monitored road section;

[0052] The background execution unit is used to receive the detection picture and execute the monitoring method based on face recognition and license plate recognition as described above.

[0053] Through the above technical scheme, the monitoring method and system based on face recognition and license plate recognition provided by the present invention obtains the detection picture under different viewing angles of the monitoring section, and then the detection picture can be input into the non-motor vehicle target detection network, so that the coordinate information of the non-motor vehicle in the detection picture can be obtained. After obtaining the coordinate information of the non-motor vehicle, the non-motor vehicle cut-outs of different viewing angles can be fused according to the coordinate information, and after fusion, the number of people in the current vehicle can be determined according to the fused non-motor vehicle cut-outs. In the case where the number of people is greater than the preset threshold, it can be determined that the non-motor vehicle is illegal, and it is necessary to determine the information of the people related to the non-motor vehicle, such as the driver information, or the non-motor vehicle attribution information, so the non-motor vehicle cut-out can be input into the face recognition network, and the face information can be determined according to the comparison of the database. After obtaining the face information, the non-motor vehicle cut-out can be input into the license plate recognition network, so that the license plate information can be obtained. According to the face information and the license plate information, the final illegal person information can be determined and reported. The monitoring method can quickly and efficiently determine the information of the non-motor vehicle illegal personnel.

[0054] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0056] Figure 1 is a flow chart of a monitoring method based on face recognition and license plate recognition according to an embodiment of the present invention;

[0057] Figure 2 It is a flow chart of obtaining a detection screen of a monitoring method based on face recognition and license plate recognition according to an embodiment of the present invention;

[0058] Figure 3 is a flow chart of a cutting and fusion of a monitoring method based on face recognition and license plate recognition according to an embodiment of the present invention;

[0059] Figure 4 is a flow chart of determining face information of a monitoring method based on face recognition and license plate recognition according to an embodiment of the present invention;

[0060] Figure 5 is a flow chart of determining final personnel information of a monitoring method based on face recognition and license plate recognition according to an embodiment of the present invention;

[0061] Figure 6 The present invention is a flowchart of a monitoring method for determining lawbreakers based on face recognition and license plate recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0063] Figure 1 1 is a flow chart of a monitoring method based on face recognition and license plate recognition according to an embodiment of the present invention. In the present invention, the process of the monitoring method may include:

[0064] In step S1, detection images of the monitored road section at different viewing angles are obtained.

[0065] In step S2, different detection images are input into a non-motor vehicle target detection network to obtain non-motor vehicle coordinate information in the detection images.

[0066] In step S3, non-motor vehicle cut images of different perspectives are fused according to the non-motor vehicle coordinate information.

[0067] In step S4, the number of people in the current vehicle is determined based on the fused non-motor vehicle cut graph.

[0068] In step S5, when the number of people is greater than a preset threshold, the non-motor vehicle cut image is input into the face recognition network, and the face information is determined according to the database.

[0069] In step S6, the non-motor vehicle cut image is input into the license plate recognition network to obtain the license plate information.

[0070] In step S7, the final information of the offender is determined based on the facial information and license plate information and reported.

[0071] In the present invention, when monitoring non-motor vehicles at intersections, detection images under different viewing angles of the monitored road section can be obtained, and then the detection images under different viewing angles can be input into the non-motor vehicle target detection network, so that the coordinate information of the non-motor vehicle in the detection image can be obtained. After obtaining the coordinate information of the non-motor vehicle, a non-motor vehicle cut image about the non-motor vehicle can be obtained according to the coordinate information of the non-motor vehicle, and then the non-motor vehicle cut images of different viewing angles can be fused. After fusion, the number of people in the current vehicle in the image can be determined according to the fused non-motor vehicle cut image. If the number of passengers on the vehicle is greater than a preset threshold, it can be indicated that the vehicle may have illegal situations such as overloading, and the non-motor vehicle cut image can be input into the face recognition network, and the face information of the possible illegal situation can be determined according to the database. The non-motor vehicle cut image can also be input into the license plate recognition network, so that the license plate information can be obtained. After obtaining the personnel information and face information of the vehicle that may have illegal situations, the owner information of the illegal vehicle can be determined according to the license plate information, and the information of the personnel on the non-motor vehicle can be determined through the face information. If the owner information is consistent with one of the personnel information on the non-motor vehicle, it can be shown that the owner of the non-motor vehicle has illegal situations, and the illegal personnel can be determined according to the owner information. Even if the remaining personnel on the non-motor vehicle do not obtain face information by comparison in the database, their illegal behavior can be determined according to the owner information. If the owner information and face information corresponding to the license plate information are inconsistent, the owner information and the face information after face comparison can be included in the final illegal personnel together and reported. If only one of the license plate information or face information is obtained, the other item can be skipped and locked through one of them. Therefore, the final illegal personnel information can be determined according to the face information and the license plate information, and can be reported. This monitoring method can quickly and efficiently determine the personnel information of non-motor vehicle violations.

[0072] In one embodiment of the present invention, Figure 2 As shown, the process of obtaining the detection picture may include:

[0073] In step S8, the monitoring information is displayed through the front-end system, and the camera rtsp address A and rtsp address B of the monitored section are set as two viewing angle cameras in the front-end page.

[0074] In step S9, corresponding real-time video streams are obtained respectively according to the two set viewing angle cameras.

[0075] In step S10, the screen image of the corresponding camera is decoded according to the corresponding real-time video stream, that is, the detection screen under different viewing angles.

[0076] In the present invention, the image acquisition of non-motor vehicles can obtain comprehensive traffic scene information through cameras such as road electric police and ball cameras. The monitoring information can be displayed in real time through the front-end system, and the user can set the camera rtsp address A and rtsp address B of the monitoring section as two viewing angle cameras in the front-end page. The back-end system can obtain real-time video streams under different viewing angles according to the camera address input by the user, and according to the corresponding real-time video stream, the screen image of the corresponding camera can be decoded, which is also the detection screen under different viewing angles. After obtaining the detection screen under different viewing angles, various information about non-motor vehicles can be obtained according to the detection screen.

[0077] In one embodiment of the present invention, Figure 3 As shown, the process of cutting and merging images may include:

[0078] In step S11, the feature point detection network is trained using a simple geometric shape data set to obtain an initial feature point detection network.

[0079] In step S12, a real training image is obtained.

[0080] In step S13, the real training image is transformed through multiple composite geometric transformations, where the number of transformations is marked as 1, ..., N, the first one is the original image, and the remaining N-1 images are images formed by randomly generated composite geometric transformations.

[0081] In step S14, feature points are extracted from the real training image through an initial feature point detection network.

[0082] In step S15, the initial feature point detection network maps the N-1 images corresponding to the real training images back to the feature points of the original images and accumulates them to form new feature points of the original images, thereby completing the self-labeling of the feature points of the real training images.

[0083] In step S16, the real training image and the feature point self-annotation are input into the feature point detection network for training to obtain an enhanced feature point network.

[0084] In step S17, non-motor vehicle cut images of different perspectives corresponding to the non-motor vehicle coordinate information are input into the enhanced feature point network to obtain a corresponding feature point descriptor set.

[0085] In step S18, a feature point description subset is determined, and the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance of a feature point of another feature point description subset and its feature point is determined.

[0086] In step S19, when the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance is less than a set threshold, the feature point is determined as a matching point, and the non-motor vehicle cut image is fused according to the matching point.

[0087] In the present invention, when the cut images are fused, the fusion model needs to be trained first. The feature point detection network can be trained using a simple geometric shape data set, which can be composed of images whose feature points are relatively easy to determine, such as line segments, polygons, and cubes. The feature point extraction of the basic data set can be performed using properties such as scale-invariant feature transformation to obtain the data set and the true value of the feature point, and then the feature point detection network can be trained using the labeled simple geometric shape data set, so that a primary feature point detection network can be obtained. After obtaining the real training image, the real training image can be transformed using multiple composite geometric transformations. The number of transformations can be marked as 1,...,N, the first can be the original image, and the remaining 2,...N can be images formed by randomly generated composite geometric transformations of the original image. Then, the feature point extraction of the real training image can be performed using the initial feature point detection network. The initial feature point detection network can map the N-1 images corresponding to the real training image back to the feature point accumulation of the original image, so that new feature points of the original image can be formed, and the feature point self-labeling of the real training image can be completed. After inputting the real training image and the feature point self-annotation into the feature point detection network for training, the feature point extraction accuracy can be improved, so that an enhanced feature point network can be obtained. After obtaining the enhanced feature point network, the non-motor vehicle coordinate information and the corresponding non-motor vehicle cut images of different perspectives can be input into the enhanced feature point network, so that the corresponding feature point description subset can be obtained. The subset sum can include multiple feature points. After obtaining the subset sum under multiple perspectives, a feature point description subset can be determined, and the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance of the feature point of another feature point description subset and the feature point in the subset sum can be determined. When the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance of the feature point in the two feature point description subsets is less than a set threshold, the feature point can be determined as a matching point, and then the non-motor vehicle cut image can be fused according to the matching point, so that the accurate fusion of the non-motor vehicle cut images under two different perspectives can be achieved.

[0088] In one embodiment of the present invention, the composite geometric transformation formula of the multiple composite geometric transformations may be formula (1):

[0089] , formula (1)

[0090] in, A point in the image from the first perspective, Represents the image from another perspective The corresponding points, represents the composite geometric transformation matrix, is the transformation coefficient. In order to fuse the information from different perspectives, it is necessary to first find the winning relationship of different perspectives. The adaptive homography transformation can be used to solve the composite geometric transformation matrix of different perspectives, that is, the composite geometric transformation matrix can be calculated through two images with different perspectives.

[0091] In one embodiment of the present invention, Figure 4 As shown, the process of determining facial information may include:

[0092] In step S20, a non-motor vehicle cut image is obtained and input into a face network to obtain face coordinate information.

[0093] In step S21, a face cut image is obtained according to the face coordinate information.

[0094] In step S22, the face cut image is sent to a face feature extraction network to obtain face features.

[0095] In step S23, the facial features are matched with the facial information in the database to determine the final person information.

[0096] In the present invention, when determining facial information, the non-motor vehicle cut image can be first input into the facial network, so that the facial coordinate information can be obtained. The facial cut image can be obtained according to the facial coordinate information. After obtaining the facial cut image, the facial cut image can be sent to the facial feature extraction network for feature extraction to obtain facial features. After obtaining the facial features, the facial features can be matched with the facial information in the database, and the database can be a face library, including multiple face images, and can include personnel information corresponding to the face. The facial features can be matched with the database to obtain the corresponding personnel information, so that the final personnel information can be determined.

[0097] In one embodiment of the present invention, Figure 5 As shown, the process of determining the final personnel information may include:

[0098] In step S24, the face cut image is input into a face feature extraction network to obtain a face feature vector to be matched.

[0099] In step S25, all the face images to be matched in the database are input into the face feature extraction network to obtain their respective face feature vectors.

[0100] In step S26, all facial feature vectors are divided into m groups, each group having n facial feature vectors.

[0101] In step S27, each group of facial feature vectors is concatenated into a feature matrix, wherein the number of columns of the feature matrix is ​​equal to the number of rows of the facial feature vectors to be matched.

[0102] In step S28, the feature vector of the face to be matched is multiplied by the feature matrix of the m groups to obtain the similarity score of the database comparison of each group.

[0103] In step S29, the maximum similarity score of each group is taken, and the record vector column index is retained.

[0104] In step S30, the maximum value and the corresponding column index are taken for all the maximum similarity scores of each group.

[0105] In step S31, it is determined whether the maximum value is greater than the similarity score threshold.

[0106] In step S32, when the maximum value is greater than the similarity score threshold, the personnel information of the column index corresponding to the maximum value is obtained as the final personnel information.

[0107] In the present invention, when matching facial features with facial information in a database, a vector multiplication method can be used to obtain the top 1 candidate. Compared with the traditional method for calculating vector similarity, this method can have faster and better face matching speed and efficiency. When matching, the face cut image can be input into the face feature extraction network, so that the face feature vector to be matched can be obtained. All face images to be matched in the database are input into the face feature extraction network to obtain their respective face feature vectors, and then all face feature vectors can be divided into m groups, each group can have n face feature vectors. After obtaining the face feature vectors of m groups, the face feature vectors of each group can be spliced ​​into a feature matrix. The number of columns of the feature matrix can be equal to the number of rows of the face feature vector to be matched. After obtaining the feature matrix, the face feature vector to be matched can be multiplied with the feature matrix of m groups, so that the similarity score of the database comparison of each group can be obtained. For example, after multiplying with a group of feature matrices in m groups, n values ​​can be obtained, and the n values ​​can be the similarity score of the group. After obtaining the similarity score of each group, the maximum value of the similarity score of each group can be taken, and the record vector column index can be retained. After obtaining the maximum similarity score of each group, the maximum value and the corresponding column index of all the maximum similarity scores of each group can be taken. After obtaining the maximum value, it can be determined whether the maximum value is greater than the similarity score threshold. When the maximum value is greater than the similarity score threshold, it can be indicated that the face cut image corresponding to the maximum value can be matched to the corresponding face data in the database, and the person information according to the column index corresponding to the maximum value can be determined as the final person information. If the maximum value is not greater than the similarity score threshold, it can be indicated that the face cut image has not found the corresponding matching person in the database.

[0108] In one embodiment of the present invention, the non-motor vehicle target detection network can be a yolov5n model, and the actual collected image annotations can be used as a training set to optimize the yolov5n model, and then the optimized yolov5n model can be used to identify and locate the non-motor vehicle in the detection picture.

[0109] In one embodiment of the present invention, the license plate recognition network can be an OCR model, which can be divided into two stages: character detection and character recognition. First, the license plate cut image is input into the character detection model to obtain the character coordinate position to achieve character positioning, and then the located character image is input into the character recognition model for recognition to obtain character text information.

[0110] In one embodiment of the present invention, Figure 6 As shown, the process of identifying offenders may include:

[0111] In step S33, it is determined whether the license plate information and the final person information are obtained at the same time.

[0112] In step S34, when the license plate information and the final person information are obtained at the same time, the person corresponding to the license plate information and the final person information are both listed as the final illegal person.

[0113] In step S35, when the license plate information and the final person information are not obtained at the same time, the person corresponding to the obtained license plate information or the final person information is listed as the final offender.

[0114] In the present invention, the final offender can be determined based on the combination of face information and license plate information. Determine whether the license plate information and the final person information are obtained at the same time. When the license plate information and the final person information are obtained at the same time, the person corresponding to the license plate information and the final person information can be listed as the final offender. Considering that the camera cannot guarantee that the face and the license plate can be photographed under different viewing angles, it can be locked by one of them. In the case that the license plate information and the final person information are not obtained at the same time, the person corresponding to the obtained license plate information or the final person information can be listed as the final offender.

[0115] On the other hand, the present invention can also provide a monitoring system based on face recognition and license plate recognition, and the monitoring system can include: a camera and a background execution unit. The camera can be used to obtain detection images under different viewing angles of the monitored road section. The background execution unit can be used to receive the detection images and execute a monitoring method based on face recognition and license plate recognition as described above.

[0116] Through the above technical scheme, the monitoring method and system based on face recognition and license plate recognition provided by the present invention obtains the detection picture under different viewing angles of the monitoring section, and then the detection picture can be input into the non-motor vehicle target detection network, so that the coordinate information of the non-motor vehicle in the detection picture can be obtained. After obtaining the coordinate information of the non-motor vehicle, the non-motor vehicle cut-outs of different viewing angles can be fused according to the coordinate information, and after fusion, the number of people in the current vehicle can be determined according to the fused non-motor vehicle cut-outs. In the case where the number of people is greater than the preset threshold, it can be determined that the non-motor vehicle is illegal, and it is necessary to determine the information of the people related to the non-motor vehicle, such as the driver information, or the non-motor vehicle attribution information, so the non-motor vehicle cut-out can be input into the face recognition network, and the face information can be determined according to the comparison of the database. After obtaining the face information, the non-motor vehicle cut-out can be input into the license plate recognition network, so that the license plate information can be obtained. According to the face information and the license plate information, the final illegal person information can be determined and reported. This monitoring method can quickly and efficiently determine the information of the illegal non-motor vehicle personnel.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0122] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0123] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0125] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A monitoring method based on face recognition and license plate recognition, It is characterized in that The monitoring method comprises: Obtain detection images from different viewing angles of the monitored road section; Inputting different detection images into a non-motor vehicle target detection network to obtain non-motor vehicle coordinate information in the detection images; fusing non-motor vehicle cut images from different perspectives according to the non-motor vehicle coordinate information; Determine the number of people in the current vehicle according to the fused non-motor vehicle cut map; When the number of people is greater than a preset threshold, the non-motor vehicle cut image is input into a face recognition network, and face information is determined according to a database; Inputting the non-motor vehicle cut image into a license plate recognition network to obtain license plate information; Determine the final information of the offender based on the facial information and license plate information, and report it; The non-motor vehicle cut images of different perspectives are merged according to the non-motor vehicle coordinate information, including: The feature point detection network is trained using a simple geometric shape data set to obtain an initial feature point detection network; Get real training images; The real training image is transformed by multiple composite geometric transformations, where the number of transformations is marked as 1, ..., N, the first one is the original image, and the remaining N-1 images are images formed by randomly generated composite geometric transformations; Extracting feature points from the real training image through the initial feature point detection network; The initial feature point detection network maps N-1 images corresponding to the real training image back to the feature points of the original image and accumulates them to form new feature points of the original image, thereby completing the self-labeling of the feature points of the real training image; Inputting the real training image and the feature point self-annotation into the feature point detection network for training to obtain an enhanced feature point network; Inputting non-motor vehicle cut images of different perspectives corresponding to the non-motor vehicle coordinate information into the enhanced feature point network to obtain a corresponding feature point descriptor set; Determine a subset of the feature point descriptions, and determine the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance between a feature point of another subset of the feature point descriptions and its feature point; When the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance is less than a set threshold, the feature point is determined to be a matching point, and the fusion of the non-motor vehicle cut graph is achieved based on the matching point.

2. The monitoring method according to claim 1, It is characterized in that The detection images obtained from different perspectives of the monitored road section include: The monitoring information is displayed in real time through the front-end system, and the camera rtsp address A and rtsp address B of the monitoring section are set as two viewing angle cameras in the front-end page; According to the two set viewing angle cameras, the corresponding real-time video streams are obtained respectively; According to the corresponding real-time video stream, the screen image of the corresponding camera is decoded, that is, the detection screen under different viewing angles.

3. The monitoring method according to claim 1, It is characterized in that The composite geometric transformation formula of multiple composite geometric transformations is formula (1): , Formula (1) in, A point in the image from the first perspective, Represents the image from another perspective The corresponding points, represents the composite geometric transformation matrix, is the transformation coefficient.

4. The monitoring method according to claim 1, It is characterized in that When the number of people is greater than a preset threshold, the non-motor vehicle cut image is input into a face recognition network, and the face information is determined according to the database, including: Inputting the non-motor vehicle cut image into a face network to obtain face coordinate information; Acquire a face cut image according to the face coordinate information; Sending the face cut image to a face feature extraction network to obtain face features; The facial features are matched with the facial information in the database to determine final personnel information.

5. The monitoring method according to claim 4, It is characterized in that Matching the facial features with the facial information in the database to determine the final person information includes: Inputting the face cut image into a face feature extraction network to obtain a face feature vector to be matched; Inputting all the face images to be matched in the database into the face feature extraction network to obtain their respective face feature vectors; Divide all the facial feature vectors into m groups, each group having n facial feature vectors; Splicing each group of facial feature vectors into a feature matrix, wherein the number of rows of the feature matrix is ​​equal to the number of columns of the facial feature vectors to be matched; Multiplying the face feature vector to be matched with the feature matrix of m groups to obtain a similarity score of the database comparison of each group; Take the maximum similarity score of each group and keep the record vector column index; Take the maximum value and corresponding column index of all the maximum similarity scores of each group; Determining whether the maximum value is greater than a similarity score threshold; In the case that the maximum value is greater than the similarity score threshold, the personnel information of the column index corresponding to the maximum value is obtained as the final personnel information.

6. The monitoring method according to claim 1, It is characterized in that The non-motor vehicle target detection network is a yolov5n model, and the actual collected picture annotations are used as a training set to optimize the yolov5n model, and then the non-motor vehicle in the detection picture is identified and located by the optimized yolov5n model.

7. The monitoring method according to claim 5, It is characterized in that The license plate recognition network is an OCR model.

8. The monitoring method according to claim 7, It is characterized in that The final information of the offender is determined based on the facial information and license plate information, and the information is reported including: Determine whether the license plate information and final personnel information are obtained at the same time; When the license plate information and the final person information are obtained at the same time, the person corresponding to the license plate information and the final person information are both listed as the final illegal person; In the case where the license plate information and the final person information are not obtained at the same time, the person corresponding to the obtained license plate information or the final person information is listed as the final violator.

9. A monitoring system based on face recognition and license plate recognition, It is characterized in that The monitoring system comprises: Camera, used to obtain detection images from different viewing angles of the monitored road section; A background execution unit is used to receive a detection picture and execute a monitoring method based on face recognition and license plate recognition as described in any one of claims 1-8.

Images