Multi-information fusion pedestrian retrieval method and system based on video surveillance and robot

Abstract

The present disclosure relates to the field of robot vision technology, and proposes a multi-information fusion pedestrian retrieval method and system based on video surveillance and robots. The method includes the following steps: obtaining image data collected by a surveillance camera, performing pedestrian re-identification processing using a pedestrian re-identification module, and obtaining pedestrian re-identification results; obtaining image data collected by a mobile robot, and performing face recognition processing on image information collected by multiple mobile robots; fusing the image information collected by the surveillance camera and the mobile robot after processing, and supplementing the detection results of the pedestrian re-identification module with the face information collected by the robot in the area where the pedestrian is located according to the positional relationship between the mobile robot and the pedestrian, to obtain the target pedestrian retrieval result. The regional monitoring visual sensor and the robot visual sensor cooperate with each other to search the relevant area from the global and local perspectives, respectively, to make up for the problems caused by a single visual sensor and realize the search of the target person.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to robot vision, and more specifically, to a multi-information fusion pedestrian retrieval method and system based on video surveillance and robots. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] The search for specific people in a large scene in large public places is of great significance in dealing with emergency tasks such as the loss of the elderly and children. With the comprehensive coverage of surveillance cameras, it is feasible to build a system to retrieve a specific target person in the scene. For the retrieval of specific target people in the scene, the current mainstream technology is mainly pedestrian re-identification, which searches the video image information collected by multiple cameras to find the coverage of the camera where the specific target appears. However, the detection effect is significantly reduced when the target is blocked over a large area, changes clothes, changes in indoor ambient light, etc. In addition, it is impossible to effectively search for areas where no surveillance system is deployed. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a multi-information fusion pedestrian retrieval method and system based on video surveillance and robots, which utilizes the cooperation between regional monitoring visual sensors and robot visual sensors to search the relevant areas from global and local perspectives respectively, to make up for the problems caused by a single visual sensor, to build a specific target retrieval system, and to realize the search of target persons.

[0005] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0006] One or more embodiments provide a multi-information fusion pedestrian retrieval method based on video surveillance and robots, comprising the following steps:

[0007] Obtain image data collected by the surveillance camera, use the pedestrian re-identification module to perform pedestrian re-identification processing, and obtain the pedestrian re-identification result;

[0008] Acquire image data collected by the mobile robot, and perform face recognition processing on the image information collected by multiple mobile robots;

[0009] The image information collected by the surveillance camera and the mobile robot is processed and fused. According to the positional relationship between the mobile robot and the pedestrian, the facial information collected by the robot in the pedestrian area is used to supplement the detection results of the pedestrian re-identification module.

[0010] One or more embodiments provide a multi-information fusion pedestrian retrieval system based on video surveillance and robots, including an image data acquisition end, a device communication end, an image processing end, and a business application end;

[0011] The image data acquisition end includes a surveillance camera and a mobile robot visual sensor, which are used to jointly collect image information;

[0012] The device communication end is used to send the collected image information to the terminal processor of the image processing end for processing;

[0013] Image processing end: used to process the collected images using the above-mentioned pedestrian retrieval method;

[0014] Business application end: According to the corresponding business logic, different fusion strategies are adopted to realize pedestrian retrieval.

[0015] One or more embodiments provide a multi-information fusion pedestrian retrieval system based on video surveillance and robots, including:

[0016] Pedestrian re-identification module: configured to obtain image data collected by the surveillance camera, perform pedestrian re-identification processing using the pedestrian re-identification module, and obtain a pedestrian re-identification result;

[0017] The pedestrian re-identification results include the classified pedestrian image and the similarity information between the pedestrian image and the target pedestrian image to be found;

[0018] Face recognition module: configured to obtain image data collected by the mobile robot and perform face recognition processing on the image information collected by multiple mobile robots;

[0019] Fusion module: It is configured to process and fuse the image information collected by the surveillance camera and the mobile robot. According to the positional relationship between the mobile robot and the pedestrian, the facial information collected by the robot in the area where the pedestrian is located is used to supplement the detection results of the pedestrian re-identification module to obtain the target pedestrian retrieval results.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention solves the problems of poor detection effect and inapplicability of current pedestrian retrieval tasks by adding face recognition results to the retrieval results of pedestrian re-identification. In cases where a large area of ​​the target is obscured or the ambient light changes, face information is added to the results of pedestrian re-identification through face recognition, which can more accurately locate and retrieve the target pedestrian.

[0022] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0024] Figure 1 is an overall framework diagram of a pedestrian retrieval system according to Embodiment 1 of the present disclosure;

[0025] Figure 2 It is a flowchart of Socket communication between the mobile robot end and the image processing end in Embodiment 1 of the present disclosure;

[0026] Figure 3 is a schematic diagram of a flow chart of data processing performed by an image data processing terminal in Embodiment 1 of the present disclosure;

[0027] Figure 4 This is an overall model architecture diagram of the image data processing end of Embodiment 1 of the present disclosure;

[0028] Figure 5 It is a framework diagram of the business application end of Embodiment 1 of the present disclosure;

[0029] Figure 6 This is an example of an error result of pedestrian re-identification detection alone in Embodiment 1 of the present disclosure. DETAILED DESCRIPTION

[0030] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0032] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0033] With the popularization of mobile robots, they are playing an increasingly important role in catering, medical care, and elderly care. Mobile robots are usually equipped with cameras for local environment perception, which can make up for the information collection problems in blind spots and areas without monitoring equipment.

[0034] A single visual sensor has some problems to a certain extent. Therefore, combining the advantages of regional monitoring and mobile robot visual sensors, a pedestrian retrieval system that integrates monitoring cameras and mobile robot visual sensors is proposed to solve the practical problems of poor detection effect and inapplicability in current pedestrian retrieval tasks. The present invention uses the cooperation between regional monitoring visual sensors and robot visual sensors to search the relevant areas from the global and local perspectives, respectively, to make up for the problems caused by a single visual sensor, build a specific target retrieval system, and realize the search of target persons. The following is an explanation with specific embodiments.

[0035] Example 1

[0036] In the technical solutions disclosed in one or more embodiments, Figure 1-Figure 6 As shown, the multi-information fusion pedestrian retrieval system based on video surveillance and robots includes: an image data acquisition end, a device communication end, an image processing end, and a business application end.

[0037] The image data acquisition end includes surveillance cameras and mobile robot vision sensors, which can jointly collect image information;

[0038] The device communication end sends the collected image information to the terminal processor of the image processing end for processing;

[0039] The image processing end is the most important part of the entire framework. As it carries a graphics card with high computing power and can run a large deep neural network, it is configured to perform tasks such as feature extraction and feature matching of the collected images, serving as the core of the overall architecture.

[0040] After the collected images are processed accordingly at the image processing end, the business application end uses different fusion strategies based on the corresponding business logic to utilize the processed information to meet business needs, which can be used to achieve pedestrian retrieval.

[0041] Optionally, the image data acquisition end includes: a surveillance camera installed in the scene and a visual sensor carried by the robot itself.

[0042] Since surveillance cameras have the characteristics of wide field of view and wide coverage, they can be used to collect data from multiple surveillance cameras to obtain most of the image information in the scene. However, since the clarity of surveillance cameras is often not high and is greatly affected by sudden situations such as light intensity, crowd density, and obstacles, the collected image data often contains certain noise data. The image information collected by the surveillance camera will be used as the input of the image processing end, as shown in formula (1):

[0043] Input1={X1,X2,…,X n} (1)

[0044] Among them, X j The j-th frame of image data collected, Input1 represents the data input to the surveillance camera of the image processing end.

[0045] The mobile robot can move flexibly within the scene. Through the visual sensor mounted on the robot, it can supplement the information that the surveillance camera cannot collect, and adjust the position of the robot in time according to the required information, so that the mobile robot can continuously obtain more useful and timely image information in real time. The mobile robot is used as the input of the image processing end, as shown in formula (2):

[0046] Input2={Y1,Y2,…,Y n} (2)

[0047] Among them, Y j The j-th frame of image data collected by the visual sensor carried by the mobile robot, Input2 represents the data of the visual sensor carried by the mobile robot input to the image processing end.

[0048] Optionally, the data from the surveillance camera and the visual sensor carried by the mobile robot can be combined as the overall input of the image processing end, as shown in Equation 3 and Equation 4:

[0049] Input=Input1+Input2 (3)

[0050] Input={X1, Y1, X2, Y2,…,X n , Y n} (4)

[0051] In some embodiments, specifically, the device communication end includes a communication network, and the monitoring camera transmits the image information to the image processing end through the network. Specifically, wireless network communication may be adopted, and the transmission may be performed through a Wifi network.

[0052] The mobile robot and the image processing end use Socket to communicate, and the visual sensor of the mobile robot is called in real time to collect data. The data collected by the mobile robot can be packaged and sent to the image processing end for processing.

[0053] In this embodiment, each mobile robot is a client, and the image processing end is a server. The mobile robot and the image processing end communicate using Socket, such as Figure 2As shown in the figure, the image processing end creates a Socket object, binds a protocol to the Socket object, assigns a name to it, monitors the Socket object, and changes the Socket object from the default active type to the passive type; while the robot end creates a Socket object as a client and connects to the server's Socket object. Socket communication can form a physical connection between the server and the client, and transmit data, realize real-time information interaction between the server and the client, and can encrypt data transmission, with high data security.

[0054] Furthermore, the image processing end includes a pedestrian re-identification module and a face recognition module. The pedestrian re-identification module is used to process data collected by the surveillance camera, and the face recognition module is used to process image data collected by the mobile robot. Both the pedestrian re-identification module and the face recognition module use deep neural networks for recognition.

[0055] Mobile robots often cannot carry high-computing power equipment. At the same time, the mobile robots themselves need to complete other tasks such as navigation and obstacle avoidance. Mobile robots cannot directly process the images collected by the visual sensors they carry, so they package the collected image data and send it to the image processing end.

[0056] The image processing end uses the high-computing GPU carried by the terminal to process the image data using a deep neural network. The main process is as follows: Figure 3 shown.

[0057] The image data collected by the surveillance camera is mainly processed using the pedestrian re-identification module.

[0058] Pedestrian re-identification module: It is configured to perform pedestrian re-identification processing on the images obtained from the surveillance camera, including pedestrian detection and feature matching in sequence to obtain pedestrian images and matching results; first, pedestrian recognition is performed, and pedestrian detection is performed on the pedestrians in the scene. After detection, the pedestrians in the image will be framed, and then the features of the framed pedestrians will be extracted, and the features of the provided target pedestrians will also be extracted, and the similarity between the two will be calculated to obtain the final result.

[0059] The pedestrian images include multiple pedestrian images in the scene that have high similarity with the pedestrian to be retrieved, and also include images of other pedestrians except the person to be retrieved.

[0060] Optionally, the pedestrian re-identification module includes a pedestrian detection module and a feature matching module.

[0061] In this embodiment, the pedestrian detection module can use the Yolact network to perform pedestrian detection, including multiple cascaded convolutional layers and a fully connected layer.

[0062] Yolact (short for You Only Look At CoefficienTs) is a real-time instance segmentation detection network that combines detection effect and real-time performance.

[0063] The image size of Yolact network input is 550*550, the backbone network is ResNet101, including five convolution modules; the convolution formula of the five convolution modules is as follows:

[0064]

[0065] Among them, Input1 is the data of the surveillance camera at the image processing end, f1, f2, f3, f4, and f5 represent the results of convolution, and Conv1, Conv2, Conv3, Conv4, and Conv5 represent five convolution modules.

[0066] Finally, it is connected to a fully connected layer with 1000-dimensional output and Softmax classification is performed, as shown in formula (6):

[0067] Output1=SoftMax(FC(f5)) (6)

[0068] Among them, FC represents the fully connected layer, f5 represents the output result of the convolution module, and SoftMax represents the multi-classification function that performs multi-classification on the result of the convolution module, and finally obtains the result Output1.

[0069] During the training process of the pedestrian detection module, i.e. the Yolact network model, the loss function is the sum of the classification loss and the regression loss.

[0070] Specifically, the regression loss can use SmoothL1 to calculate the regression loss of the detection box, as shown in formula (7-8):

[0071]

[0072] L reg (l, g) = ∑ i∈{x，y，w，h} smooth L1 (lg i ) (8)

[0073] Among them, a is the difference between the predicted box and the real box, w and h represent the length and width of the box, l represents the coordinates and length and width of the predicted box, and g i Represents the coordinates of the marked real box, x and y represent the coordinates and length and width of the box.

[0074] Specifically, Softmax is used to calculate the classification loss, as shown in formula (9):

[0075]

[0076] Among them, z is the predicted category value, t is the label value of the category, Indicates that the predicted box i matches the real box j about category p. The higher the probability prediction of p, the smaller the loss. pos indicates that it is detected as a target object, and neg indicates that it is detected as a background. Indicates that there is no object in the prediction box, then the higher the probability of predicting it as background, the smaller the loss; The calculation is shown in formula (10):

[0077]

[0078] in, Indicates z i The probability of belonging to class p, n represents the number of all categories, ∑ n exp(z i ) represents the sum of all class probabilities.

[0079] The overall loss function is shown in formula (11):

[0080]

[0081] Among them, L cls (t, z) is the category loss function, L loc (t, l, g) is the detection box loss, and α is a coefficient that determines the ratio between the two.

[0082] Feature matching module: It is configured to extract features from the pedestrian image after pedestrian detection, compare and match the similarities and differences between the pedestrian image and the target pedestrian image to be found, and obtain the similarity information between the pedestrian image and the target pedestrian image to be found.

[0083] When a pedestrian is detected, the target in the pedestrian frame is extracted and sent to the feature matching model. Triplet Loss is used for feature matching. The training process is regarded as an image retrieval task. The feature distance of the same pedestrian image is smaller than the feature distance between different pedestrians. The specific description is shown in formula (12):

[0084] L tri (i, j, k) = max(ρ + d ij +d ik ,0) (12)

[0085] Among them, d ij , d ik They represent the Euclidean distance between the same pedestrian images and the Euclidean distance between different pedestrian images respectively.

[0086] In some embodiments, the face recognition module performs face recognition processing on the images collected by the mobile robot to obtain face recognition results in the images.

[0087] Optionally, the face_recognition face detection module can be used. face_recognition is a deep learning face recognition library based on dlib. face_recognition is used to perform real-time face recognition processing on image information collected by multiple mobile robots.

[0088] The face recognition module can use ResNet50 as the backbone network, and the input face image is 128 dimensions.

[0089] Output2=FaceRecognition(Input2) (13)

[0090] Among them, Input2 is the image information collected by the mobile robot, FaceRecognition is the face_recognition face detection module, and Output2 is the output result of the face recognition module.

[0091] like Figure 4 As shown in the figure, it is the overall architecture diagram of the model, including pedestrian detection module, face recognition module and fusion module; the pedestrian detection module and face recognition module input the target image information to be found, extract features of the picture containing the target to be found, perform face recognition and pedestrian re-identification, and prepare for the matching and comparison of the subsequent fusion module.

[0092] In this embodiment, the face recognition module also includes multiple convolutional layers and fully connected layers, which is similar to the network structure of the pedestrian re-identification module.

[0093] In some embodiments, the specific business of the business application end may be target person retrieval, behavior detection, etc. This embodiment takes target person retrieval as an example for explanation.

[0094] Pedestrian detection mainly uses the yoloct detection network to perform pedestrian detection in real time, and the feature extraction network mainly uses ResNet50, which is the backbone network shared with the face recognition module, and finally obtains the result of pedestrian re-identification.

[0095] Furthermore, it also includes a fusion part, which is configured to fuse the pedestrian image data and feature matching results output by the pedestrian re-identification module and the recognition results output by the face recognition module, and according to the positional relationship between the mobile robot and the pedestrian, the face information collected by the robot in the area where the pedestrian is located is used to supplement the detection results of the pedestrian re-identification module to obtain the pedestrian retrieval results.

[0096] Optionally, a fusion method can adopt condition-based judgment, that is, when the surveillance camera detection effect is poor and there is no way to detect the target person, the face recognition module of the mobile robot is started; when the surveillance camera detection effect is good and the target person can be detected, the mobile robot is no longer called for face recognition.

[0097] Optionally, based on the position relationship between the mobile robot and the pedestrian, the determination method is: the position of the mobile robot can be determined by its own positioning, and the moving position of the mobile robot has certain restrictions, and the position of the pedestrian can obtain the position information provided by mobile devices such as mobile phones carried by the pedestrian himself, or through the positioning of a monocular camera.

[0098] The facial information collected by the robot in the area where the pedestrians are located is used to supplement the detection results of the pedestrian re-identification module to obtain the target pedestrian retrieval results. The supplementary method is specifically: when the pedestrian re-identification detection effect is poor or there is no surveillance camera in the area, the facial information is used to supplement the situation where the pedestrian re-identification effect is poor.

[0099] The image information collected by the surveillance camera and the mobile robot is processed and fused. After the image processing is completed, according to the corresponding business logic, a series of pedestrian images with high similarity to the pedestrians to be retrieved in the scene are obtained. However, this series of images will contain many other pedestrian images. At this time, according to the position relationship between the mobile robot and the pedestrians, the face information collected by the robot in the area where the pedestrians are located can be used to supplement them, as shown in formula (14):

[0100]

[0101] Among them, Output2 is the result of face recognition output in the image processing end, Output3 is the result of pedestrian re-identification module output in the image processing end, and targetperson is the target pedestrian result obtained by the fusion of the information of the two.

[0102] The specific business logic may change according to the specific task, and can be further analyzed according to the business. For example, when the elderly and children are lost, the image of the missing person can be found in the existing surveillance video, and the full-body photos collected and the facial photos provided by relatives can be used to search in a large area; or when an outbreak occurs in a large public area or a close contact with an outbreak occurs, it is possible to quickly locate which areas in the public place they have passed through, and focus on the people in these areas. The overall process is as follows: Figure 5 shown.

[0103] In order to illustrate the retrieval effect of the system in this embodiment, an experiment was conducted.

[0104] The above-mentioned system is built and corresponding verification experiments are carried out, which mainly include calling the surveillance cameras and mobile robots in the venue, processing the image data collected by each, and finally giving an objective evaluation of the experimental results.

[0105] This experiment was set up in a school corridor. Surveillance cameras were installed at the front and back ends of the corridor, and a mobile robot was deployed in the venue. In the experiment, the mobile robot communicated with the host in real time. The mobile robot was used as the client and the host was used as the image processing end. The two communicated with each other. The delay between the terminal processor of the mobile robot end and the image processing end was only 0.7 seconds, while the clarity was 720P. The delay between the surveillance camera end and the terminal processor was only 0.5 seconds, while the clarity was 1080P. According to the results, both the image data acquisition end and the device communication end achieved good results, achieving low latency and real-time performance while ensuring clarity.

[0106] In the core image processing link, the target pedestrian is set as No. 10. After the target is set, the terminal device equipped with 1080Ti is used to process the collected and sent image data. When only pedestrian re-identification is used for detection, the detection effect is poor. When the lighting changes, the detection error occurs, such as Figure 6 As shown in Figure 2, where ReID stands for person re-identification.

[0107] It can be clearly seen from the figure that in the images collected by different cameras, the pedestrian in black clothes on the left side was mistakenly detected as No. 4 and No. 1 respectively, but No. 1 is the correct number, but No. 4 is the wrong number. The pedestrian in white clothes on the right side was correctly classified as No. 2. The main reason for this problem is that due to different lighting conditions, the pants of the pedestrian in black clothes appear black and blue respectively. This makes it impossible to effectively match the features of pedestrian re-identification, resulting in detection errors.

[0108] When such problems occur, the mobile robot collects image information for face detection. If the face recognition is correct, it means that No. 1 has been found. Using face recognition to verify the target person and supplement and correct the detection results solves the problem of reduced detection effect that often occurs in surveillance cameras and improves the detection effect.

[0109] Example 2

[0110] Based on the system in Example 1, this embodiment provides a multi-information fusion pedestrian retrieval method based on video surveillance and robots, which can be implemented in an image processing end, including the following steps:

[0111] Step 1: Obtain image data collected by the surveillance camera, use the pedestrian re-identification module to perform pedestrian re-identification processing, and obtain the pedestrian re-identification result;

[0112] The pedestrian re-identification results include the classified pedestrian image and the similarity information between the pedestrian image and the target pedestrian image to be found;

[0113] Step 2: Obtain image data collected by the mobile robot, and perform face recognition processing on the image information collected by multiple mobile robots;

[0114] Step 3: After processing, the image information collected by the surveillance camera and the mobile robot is fused. According to the positional relationship between the mobile robot and the pedestrian, the facial information collected by the robot in the area where the pedestrian is located is used to supplement the detection results of the pedestrian re-identification module to obtain the pedestrian retrieval results.

[0115] This embodiment solves the problems of poor detection effect and inapplicability in current pedestrian retrieval tasks by adding face recognition results to the retrieval results of pedestrian re-identification. In cases where the target is obscured over a large area or the ambient light changes, face information is added to the results of pedestrian re-identification through face recognition, which can more accurately locate and retrieve the target pedestrian.

[0116] The pedestrian re-identification process in step 1 includes pedestrian detection and feature matching in sequence to obtain a pedestrian image and matching results, including the following steps:

[0117] Step 11, pedestrian detection: perform pedestrian detection on the data obtained from the surveillance camera, select the area where pedestrians exist in the image, and obtain multiple pedestrian images. This corresponds to the pedestrian detection module implementation method in Example 1.

[0118] Step 12, feature matching: extract features from the pedestrian image after pedestrian detection, compare and match the similarities and differences between the pedestrian image and the target pedestrian image to be found, and obtain similarity information between the pedestrian image and the target pedestrian image to be found. This step corresponds to the implementation method of the feature matching module in Example 1.

[0119] In step 2, the face recognition module performs face recognition processing on the image collected by the mobile robot to obtain the face recognition result in the image.

[0120] Optionally, the facerecognition face detection module can be used. facerecognition is a deep learning face recognition library based on dlib. face_recognition is used to perform real-time face recognition processing on image information collected by multiple mobile robots.

[0121] The face recognition module can use ResNet50 as the backbone network, and the input face image is 128 dimensions.

[0122] Output2=FaceRecognition(Input2)

[0123] Among them, Input2 is the image information collected by the mobile robot, FaceRecognition is the face_recognition face detection module, and Output2 is the output result of the face recognition module.

[0124] In step 3, optionally, the fusion method can adopt condition-based judgment, that is, when the detection effect of the surveillance camera is poor and the target person cannot be detected, the face recognition module of the mobile robot is started; when the detection effect of the surveillance camera is good and the target person can be detected, the mobile robot is no longer called for face recognition.

[0125] Optionally, based on the position relationship between the mobile robot and the pedestrian, the determination method is: the position of the mobile robot can be determined by its own positioning, and the moving position of the mobile robot has certain restrictions, and the position of the pedestrian can obtain the position information provided by mobile devices such as mobile phones carried by the pedestrian himself, or through the positioning of a monocular camera.

[0126] The facial information collected by the robot in the area where the pedestrian is located is used to supplement the detection results of the pedestrian re-identification module. Specifically, the supplementation method is as follows: after the pedestrian image information collected by the surveillance camera is used for pedestrian re-identification, if the detection effect is good and the target pedestrian can be identified, the mobile robot is no longer called for face recognition. However, if the pedestrian re-identification detection effect is poor, or there is no surveillance camera at the location where the target person is to be found, the mobile robot is mobilized to make up for it. After the supplementation, the processing results of the image collected by the surveillance camera and the image data processing results collected by the mobile robot are combined to obtain the target person information.

[0127] Example 3

[0128] Based on Example 2, this embodiment provides a multi-information fusion pedestrian retrieval system based on video surveillance and robots, including:

[0129] Pedestrian re-identification module: configured to obtain image data collected by the surveillance camera, perform pedestrian re-identification processing using the pedestrian re-identification module, and obtain a pedestrian re-identification result;

[0130] The pedestrian re-identification results include the classified pedestrian image and the similarity information between the pedestrian image and the target pedestrian image to be found;

[0131] Face recognition module: configured to obtain image data collected by the mobile robot and perform face recognition processing on the image information collected by multiple mobile robots;

[0132] Fusion module: It is configured to process and fuse the image information collected by the surveillance camera and the mobile robot. According to the positional relationship between the mobile robot and the pedestrian, the facial information collected by the robot in the area where the pedestrian is located is used to supplement the detection results of the pedestrian re-identification module, so that the target pedestrian can be located and retrieved more accurately.

[0133] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0134] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A multi-information fusion pedestrian retrieval method based on video surveillance and robots, characterized in that: The steps include: Obtain image data collected by the surveillance camera, use the pedestrian re-identification module to perform pedestrian re-identification processing, and obtain the pedestrian re-identification result; Acquire image data collected by the mobile robot, and perform face recognition processing on the image information collected by multiple mobile robots; The image information collected by the surveillance camera and the mobile robot is processed and fused. According to the positional relationship between the mobile robot and the pedestrian, the facial information collected by the robot in the area where the pedestrian is located is used to supplement the detection results of the pedestrian re-identification module to obtain the target pedestrian retrieval results. The method of fusing the image information collected by the surveillance camera and the mobile robot after processing is specifically as follows: when the detection effect of the surveillance camera is poor and the target person cannot be detected, the mobile robot is mobilized and the recognition result of the face recognition module is supplemented, and the image processing result collected by the surveillance camera is combined with the image data processing result collected by the mobile robot to obtain the target person information.

2. The multi-information fusion pedestrian retrieval method based on video surveillance and robot as claimed in claim 1 is characterized by: The pedestrian re-identification method includes the following steps: Perform pedestrian detection on the data obtained from the surveillance camera, select the area where pedestrians exist in the image, and obtain multiple pedestrian images; After pedestrian detection, feature extraction is performed on the pedestrian image, and the similarities and differences between the pedestrian image and the target pedestrian image to be found are compared and matched to obtain the similarity information between the pedestrian image and the target pedestrian image to be found.

3. The multi-information fusion pedestrian retrieval method based on video surveillance and robot as claimed in claim 1, characterized in that: The face recognition module uses the face_recognition face detection module.

4. A multi-information fusion pedestrian retrieval system based on video surveillance and robots, characterized by: It includes image data acquisition end, device communication end, image processing end and business application end; The image data acquisition end includes a surveillance camera and a mobile robot visual sensor, which are used to jointly collect image information; The device communication end is used to send the collected image information to the terminal processor of the image processing end for processing; Image processing end: used to process the collected images using the pedestrian retrieval method described in any one of claims 1 to 3; Business application end: According to the corresponding business logic, different fusion strategies are adopted to realize pedestrian retrieval.

5. The multi-information fusion pedestrian retrieval system based on video surveillance and robots as claimed in claim 4 is characterized by: The device communication end includes a communication network. The surveillance camera transmits image information to the image processing end through the network. The mobile robot communicates with the image processing end using Socket.

6. The multi-information fusion pedestrian retrieval system based on video surveillance and robots as claimed in claim 4, characterized in that: The image processing end includes a pedestrian re-identification module and a face recognition module. The pedestrian re-identification module is used to process data collected by the surveillance camera, and the face recognition module is used to process image data collected by the mobile robot. Both the pedestrian re-identification module and the face recognition module use deep neural networks for recognition.

7. The multi-information fusion pedestrian retrieval system based on video surveillance and robots as claimed in claim 4, characterized in that: Pedestrian re-identification module: configured to perform pedestrian re-identification processing on the images obtained from the surveillance camera, including pedestrian detection and feature matching in sequence, to obtain pedestrian images and matching results; The pedestrian re-identification module includes a pedestrian detection module and a feature matching module.

8. The multi-information fusion pedestrian retrieval system based on video surveillance and robots as claimed in claim 7, characterized in that: The pedestrian detection module uses the Yolact network for pedestrian detection, including multiple cascaded convolutional layers and fully connected layers; During the training of the pedestrian detection module, the loss function is the sum of the classification loss and the regression loss; Smooth L1 is used to calculate the regression loss of the detection box; Softmax is used to calculate the classification loss; Feature matching module: It is configured to extract features from the pedestrian image after pedestrian detection, compare and match the similarities and differences between the pedestrian image and the target pedestrian image to be found, and obtain the similarity information between the pedestrian image and the target pedestrian image to be found.

9. A multi-information fusion pedestrian retrieval system based on video surveillance and robots, characterized in that: include: Pedestrian re-identification module: configured to obtain image data collected by the surveillance camera, perform pedestrian re-identification processing using the pedestrian re-identification module, and obtain a pedestrian re-identification result; The pedestrian re-identification results include the classified pedestrian image and the similarity information between the pedestrian image and the target pedestrian image to be found; Face recognition module: configured to obtain image data collected by the mobile robot and perform face recognition processing on the image information collected by multiple mobile robots; Fusion module: It is configured to process and fuse the image information collected by the surveillance camera and the mobile robot. According to the positional relationship between the mobile robot and the pedestrian, the facial information collected by the robot in the area where the pedestrian is located is used to supplement the detection result of the pedestrian re-identification module to obtain the target pedestrian retrieval result; The method of fusing the image information collected by the surveillance camera and the mobile robot after processing is specifically as follows: when the detection effect of the surveillance camera is poor and the target person cannot be detected, the mobile robot is mobilized and the recognition result of the face recognition module is supplemented, and the image processing result collected by the surveillance camera is combined with the image data processing result collected by the mobile robot to obtain the target person information.

Images