Image processing method and system based on end-cloud cooperation, mobile robot and cloud

Abstract

The embodiment of the application provides an image processing method and system based on end-cloud cooperation, a mobile robot and a cloud server. In the embodiment of the application, the mobile robot is responsible for human body detection and human body tracking, and the cloud server is responsible for human body detection, human body tracking, face detection, face tracking and face recognition. The cloud server updates the local processing result with the cloud processing result. Under the support of the powerful function of the cloud server, the mobile robot can perceive more abundant human body information. Even if the mobile robot does not shoot the face from a relatively positive angle, the face recognition can be completed, the face recognition effect of the mobile robot is improved, and the intelligence and human-computer interaction friendliness of the mobile robot are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile robots, in particular to an end-cloud collaborative image processing method and system, a mobile robot and a cloud server. BACKGROUND

[0002] For a mobile robot, it needs to have the ability to interact with people in a friendly manner, including identifying who the person standing in front of it is, the person's expression, age, gender, line of sight, clothing, etc., so that it can interact with the person in a personalized manner. Among them, identifying who the person is is very important, because if the user feels that the mobile robot knows him, the user's sense of experience will be greatly improved.

[0003] For a mobile robot, the most common method to identify who a person is is to perform face recognition. However, to obtain a reliable face recognition result, the mobile robot must be able to capture the face at a relatively frontal angle, otherwise it cannot complete face recognition. In actual application, in some time periods, the mobile robot may not be able to capture the face at a relatively frontal angle, and thus cannot complete face recognition. SUMMARY

[0004] Aspects of the present application provide an end-cloud collaborative image processing method and system, a mobile robot and a cloud server, to improve the face recognition effect of the mobile robot, and to improve the intelligence and human-computer interaction friendliness of the mobile robot.

[0005] An end-cloud collaborative image processing method is provided in an embodiment of the present application and applied to a mobile robot. The method comprises: obtaining a video stream obtained by image collection on an environment where the mobile robot is located; performing human body detection and human body tracking on each frame of image in the video stream to obtain a local processing result of each frame of image, wherein the local processing result of each frame of image in at least part of the frames of image comprises first human body information of at least one human body in the image, and the first human body information comprises a human body tracking box ID and a position of the human body tracking box; in response to first detecting a human body from the image of the video stream, sending an identity recognition request comprising the video stream to a cloud server, so that the cloud server performs a plurality of image processing on each frame of image in the video stream to obtain a cloud processing result of each frame of image, wherein the plurality of image processing comprises human body detection, human body tracking, face detection, face tracking and face recognition, and the cloud processing result of each frame of image in at least part of the frames of image comprises second human body information of at least one human body in the image, and the second human body information comprises a human body tracking box ID, a face tracking box ID, a face ID, a position of the human body tracking box and a position of the face tracking box; receiving the cloud processing result of each frame of image returned by the cloud server, and updating the local processing result of each frame of image by using the cloud processing result of each frame of image.

[0006] Further, the local processing result of each frame image is updated by using the cloud processing result of each frame image, including: for each frame image in at least part of the frame images, determining an intersection-over-union between two person body tracking boxes according to positions of the two person body tracking boxes, and generating an intersection-over-union matrix according to each intersection-over-union, wherein one of the two person body tracking boxes is from the person body tracking box in the local processing result of the image, and the other of the two person body tracking boxes is from the person body tracking box in the cloud processing result of the image; matching based on the intersection-over-union matrix by using a Hungarian algorithm to obtain a matching result, the matching result indicating whether the person body tracking box in the local processing result of the image and the person body tracking box in the cloud processing result of the image are matched successfully; and updating the local processing result of each frame image by using the cloud processing result of each frame image according to the matching result.

[0007] Further, the local processing result of each frame image is updated by using the cloud processing result of each frame image according to the matching result, including: for the person body tracking box matched successfully, replacing first person body information of a person body corresponding to the person body tracking box included in the local processing result of the image with second person body information of the person body corresponding to the person body tracking box included in the cloud processing result of the image; and for a newly added person body tracking box in the cloud processing result of the image, taking the second person body information of the person body corresponding to the newly added person body tracking box as first person body information of a new person body, and adding the first person body information of the new person body in the local processing result of the image.

[0008] Further, the detection accuracy of the person body detection function of the mobile robot is lower than the detection accuracy of the person body detection function of the cloud, or the tracking accuracy of the person body tracking function of the mobile robot is lower than the tracking accuracy of the person body tracking function of the cloud.

[0009] The embodiment of the present application also provides an end-cloud collaborative image processing method, applied to a cloud, including: receiving an identity recognition request sent by a mobile robot and including a video stream; in response to the identity recognition request, performing multiple image processing on each frame image in the video stream to obtain a cloud processing result of each frame image, wherein the multiple image processing includes: person body detection, person body tracking, face detection, face tracking, and face recognition, and the cloud processing result of each frame image includes second person body information of at least one person body in the image, and the second person body information includes a person body tracking box ID, a face tracking box ID, a face ID, a position of the person body tracking box, and a position of the face tracking box; and returning the cloud processing result of each frame image to the mobile robot.

[0010] The embodiment of the present application also provides an end-cloud collaborative system, including a mobile robot and a cloud; the mobile robot is used to execute steps in the end-cloud collaborative image processing method; and the cloud is used to execute steps in the end-cloud collaborative image processing method.

[0011] The embodiment of the present application also provides a mobile robot, comprising a memory and a processor; the memory is used for storing a computer program; the processor is coupled to the memory and is used for executing the computer program to execute the steps in the end-cloud cooperative image processing method.

[0012] The embodiment of the present application also provides a cloud, comprising a memory and a processor; the memory is used for storing a computer program; the processor is coupled to the memory and is used for executing the computer program to execute the steps in the end-cloud cooperative image processing method.

[0013] The embodiment of the present application also provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can execute the steps in the end-cloud cooperative image processing method.

[0014] In the embodiment of the present application, the mobile robot is responsible for human body detection and human body tracking, the cloud is responsible for human body detection, human body tracking, face detection, face tracking and face recognition, the cloud processing result updates the local processing result, and under the support of the powerful function of the cloud, the mobile robot can perceive more rich human body information, even if the mobile robot does not shoot the face from a relatively positive angle, the face recognition can also be completed, the face recognition effect of the mobile robot is improved, and the intelligence and the man-machine interaction friendliness of the mobile robot are improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0016] Figure 1 An application scenario graph provided for the embodiment of the present application;

[0017] Figure 2 A signaling interaction graph of the end-cloud cooperative image processing method provided for the embodiment of the present application;

[0018] Figure 3 A flowchart of the end-cloud cooperative image processing method provided for the embodiment of the present application;

[0019] Figure 4 A flowchart of another end-cloud cooperative image processing method provided for the embodiment of the present application;

[0020] Figure 5 A structural schematic diagram of a mobile robot provided for the embodiment of the present application;

[0021] Figure 6 A structural schematic diagram of a cloud provided for the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions of the present application with reference to the embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0023] In the embodiments of the present application, “at least one” means one or more, and “multiple” means two or more. “And / or” describes the access relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, wherein A and B can be singular or plural. In the textual description of the present application, the character “ / ” generally represents that the associated objects before and after it are in an “or” relationship. In addition, in the embodiments of the present application, “first”, “second”, “third”, and the like are only to distinguish the contents of different objects, and have no other special meanings.

[0024] For a mobile robot, it is necessary to have the ability to interact with people in a friendly manner, including identifying who the person standing in front is, the person's expression, age, gender, line of sight, clothing, and the like, so as to interact with the person in a personalized manner. Among them, identifying who the person is is very important, because if the user feels that the mobile robot recognizes him, the user's sense of experience will be greatly improved.

[0025] For a mobile robot, the most commonly used method to identify who a person is is to perform face recognition. However, in order to obtain a reliable face recognition result, the mobile robot must be able to capture the face at a relatively frontal angle, otherwise it cannot complete face recognition. In actual application, in some time periods, the mobile robot can not be able to capture the face at a relatively frontal angle, and cannot complete face recognition.

[0026] Therefore, the embodiments of the present application provide an end-cloud collaborative image processing method and system, a mobile robot and a cloud end. In the embodiments of the present application, the mobile robot is responsible for human body detection and human body tracking, and the cloud end is responsible for human body detection, human body tracking, face detection, face tracking and face recognition. The cloud end updates the local processing result with the processing result, and under the support of the powerful function of the cloud end, the mobile robot can perceive more rich human body information. Even if the mobile robot does not capture the face at a relatively frontal angle, it can also complete face recognition, improve the face recognition effect of the mobile robot, and is conducive to improving the intelligence and human-computer interaction friendliness of the mobile robot.

[0027] Figure 1An application scenario is provided for an embodiment of the present application. Referring to Figure 1 In the application scenario, a mobile robot and a cloud are included. The mobile robot has a human body detection function and a human body tracking function, and the cloud has a human body detection function, a human body tracking function, a face detection function, a face tracking function, and a face recognition function, etc. The human body detection function refers to a target detection function with a human body as a detection target; the human body tracking function refers to a target tracking function with a human body as a tracking target; the face detection function refers to a target detection function with a face as a detection target; and the face tracking function refers to a target tracking function with a face as a tracking target. The detection accuracy of the human body detection function of the cloud is higher than the detection accuracy of the human body detection function of the mobile robot; and the tracking accuracy of the human body tracking function of the cloud is higher than the tracking accuracy of the human body tracking function of the mobile robot.

[0028] In the application scenario, the mobile robot collects images of the environment and obtains a video stream. The mobile robot performs human body detection and human body tracking on each frame of image in the video stream to obtain a local processing result of each frame of image. After the mobile robot detects a human body from the image of the video stream for the first time, the mobile robot requests the cloud to perform multiple image processing on each frame of image in the collected video stream to obtain a cloud processing result of each frame of image. Due to the influence of communication delay, for the images collected at the same time in the video stream, the time of image processing on the mobile robot side is earlier than the time of image processing on the cloud side. When the cloud returns the cloud processing result of each frame of image to the mobile robot, for the images collected at the same time, the local processing result of the image is updated according to the cloud processing result of the image. In this way, even if the mobile robot does not shoot the face at a relatively positive angle, according to the human body tracking box ID of the human body recognized in a frame of image, the face ID of the human body recognized in the frame of image can be obtained by querying the local processing result of a plurality of frames of image collected before or after the frame of image, and the face recognition is completed. In addition, under the support of the powerful functions of the cloud, the mobile robot can perceive more rich human body information, which is conducive to improving the intelligence and human-computer interaction friendliness of the mobile robot.

[0029] It should be noted that, Figure 1 The application scenario shown is only an exemplary application scenario, and the embodiments of the present application are not limited to the application scenario. The embodiments of the present application do not limit the devices included in the Figure 1 application scenario, nor do they limit the positional relationship between the devices in the Figure 1 application scenario.

[0030] In the embodiments of the present application, the mobile robot can interact with the cloud through a wired network or a wireless network. The cloud refers to a system providing cloud computing services, which can be understood as a computer system deployed in a remote terminal. The cloud can be a single cloud server or a distributed cloud server. For example, the wired network can include coaxial cables, twisted pairs and optical fibers, etc., and the wireless network can be a 2G (2 Generation) network, a 3G (3 Generation) network, a 4G (4 Generation) network or a 5G (5 Generation) network, a Wireless Fidelity (WIFI) network, etc. The present application does not limit the specific type or specific form of interaction, as long as it can realize the function of interaction between the mobile robot and the cloud.

[0031] In order to better understand the technical solutions provided by the embodiments of the present application, the following will be introduced in conjunction with the signaling interaction diagram shown in the figure. Figure 2

[0032] Figure 2 The signaling interaction diagram of the image processing method provided by the embodiments of the present application.

[0033] Referring to Figure 2 , the method can include the following steps:

[0034] 201. The mobile robot acquires a video stream obtained by image collection of the environment where the mobile robot is located.

[0035] Specifically, the mobile robot acquires a video stream by image collection of the environment where the mobile robot is located using an image collection device. According to the structural characteristics of the sensor, the image collection device can adopt a face array camera or a line array camera. For example, according to the supported picture resolution, the image collection device can adopt an analog camera or a digital camera. For example, according to the supported signal type, the image collection device can adopt an analog video camera or a digital video camera. For example, according to the number of camera lenses included in the camera, the image collection device can adopt a monocular camera or a binocular camera.

[0036] 202. The mobile robot performs human body detection and human body tracking on each frame of image in the video stream to obtain a local processing result of each frame of image.

[0037] ​In the embodiment, the mobile robot performs human body detection and human body tracking on each frame image in the video stream to obtain a local processing result of each frame image. The human body detection function can be used to detect at least one human body in the image, the human body tracking frame ID of each human body, and the position of the human body tracking frame. The human body tracking frame ID is used to identify the human body tracking frame. The human body tracking function can be used to track the position of the human body tracking frame corresponding to the detected human body in the image. In actual application, human body tracking is performed at high frequency, and human body detection is performed at low frequency. For example, human body detection can be performed on images in the video stream every several frames, and human body tracking can be performed on each frame image in the video stream. Here, the human body information detected by the mobile robot is referred to as first human body information, and the local processing result of each frame image in at least part of the frame images includes the first human body information of at least one human body in the image. The first human body information includes the human body tracking frame ID and the position of the human body tracking frame. Of course, if no human body appears in the image in the video stream, the local processing result of the frame image indicates that no human body is detected, and there is no first human body information of the human body.

[0038] 203. In response to first detecting a human body from the image of the video stream, sending an identity recognition request including the video stream to the cloud.

[0039] 204. The cloud responds to the identity recognition request to perform a plurality of image processing on each frame image in the video stream to obtain a cloud processing result of each frame image.

[0040] In the embodiment, after the mobile robot first detects a human body from the image of the video stream, the cloud is requested to perform a plurality of image processing on each frame image in the collected video stream to obtain a cloud processing result of each frame image, wherein the plurality of image processing includes human body detection, human body tracking, face detection, face tracking, and face recognition.

[0041] Specifically, the cloud can use the human body detection function to detect at least one human body in the image, the human body tracking frame ID of each human body, and the position of the human body tracking frame. The human body tracking function can be used to track the position of the human body tracking frame corresponding to the detected human body in the image. The image region in the human body tracking frame of the detected human body is subjected to face detection to detect the face tracking frame ID of the human body. The face tracking frame ID is used to identify the face tracking frame. The face recognition function is used to perform face recognition on the face image in the face tracking frame to determine the face ID identifying the identity of the human body. In actual application, human body tracking and face tracking can be performed at high frequency, and human body detection, face detection, and face recognition can be performed at low frequency.

[0042] Here, the human body information detected by the cloud-side human body detection is referred to as second human body information, the cloud-side processing result of each frame of image in at least part of the frame images includes the second human body information of at least one human body in the image, and the second human body information includes a human body tracking box ID, a face tracking box ID, a face ID, a position of the human body tracking box, and a position of the face tracking box. Of course, if no human body or face appears in the image in the video stream, the cloud-side processing result of the frame image indicates that no human body or face is detected, and there is no second human body information of the human body.

[0043] Further optionally, the detection accuracy of the human body detection function of the mobile robot is lower than the detection accuracy of the cloud-side human body detection function, and / or the tracking accuracy of the human body tracking function of the mobile robot is lower than the tracking accuracy of the cloud-side human body tracking function. It can be understood that making full use of the powerful computing power of the cloud can enable the mobile robot to perceive more rich human body information.

[0044] Further optionally, in order to improve the detection accuracy of the human body detection function, the mobile robot can use a lightweight yolo (You Only Look Once) target detection model for human body detection. The yolo target detection model is a model for target detection using a convolutional neural network.

[0045] Further optionally, in order to improve the tracking accuracy of the human body tracking function, the mobile robot can use a KCF (Kernel Correlation Filter) algorithm for human body tracking.

[0046] Further optionally, in order to improve the detection accuracy of the human body detection function, the cloud can use a larger yolo target detection model or a Faster R-CNN model for human body detection. The Faster R-CNN model is improved on the R-CNN (Regions with CNN features) model, and compared with the R-CNN model, an ROI pooling (Region of Interest Pooling) layer is introduced to improve the algorithm running speed.

[0047] Further optionally, when training the yolo target detection model or the Faster R-CNN model on the cloud side, the yolo target detection model or the Faster R-CNN model obtained by pre-training can be first trained using a large amount of pedestrian detection data set, and then fine-tuned based on the pedestrian detection data set in the scene served by the mobile robot to obtain the final yolo target detection model or Faster R-CNN model for human body detection.

[0048] Further optionally, the cloud-side human body tracking algorithm can adopt a target tracking algorithm based on deep learning, for example, SiamRPN++ algorithm. The SiamRPN++ algorithm uses an end-to-end training deep Siamese network for visual tracking, alleviates the problem of translation invariance by introducing a spatial perception sampling strategy (random offset), increases the network depth, and improves the algorithm effect.

[0049] Further optionally, the cloud-side human face detection algorithm can adopt a classic human face detection algorithm to obtain a better human face detection effect. The human face detection algorithm is, for example, MTCNN or RetinaFace. The MTCNN algorithm is a human face detection and face alignment method based on deep learning, which can simultaneously complete human face detection and face alignment. RetinaFace adds a human face key point regression branch and a self-supervised learning branch, and is a human face detector with strong robustness.

[0050] 205、the cloud returns the cloud processing result of each frame image to the mobile robot.

[0051] 206、the mobile robot updates the local processing result of each frame image by using the cloud processing result of each frame image.

[0052] In the embodiment, the mobile robot can periodically, in real time, at a fixed time, or at an indefinite time, return the cloud processing result of each frame image processed by the mobile robot to the mobile robot. After receiving the cloud processing result of each frame image returned by the cloud, the mobile robot takes the image acquisition time as the screening condition, finds the cloud processing result and the local processing result of a frame image, and updates the local processing result of the frame image by using the cloud processing result of the frame image. Specifically, for the human body tracking box that appears in the cloud processing result and the local processing result at the same time, the first human body information of the human body corresponding to the human body tracking box included in the local processing result of the image is replaced with the second human body information of the human body corresponding to the human body tracking box included in the cloud processing result. For the human body tracking box newly added in the cloud processing result of the image, that is, the human body tracking box that does not appear in the local processing result of the image, the second human body information of the human body corresponding to the newly added human body tracking box is taken as the first human body information of the new human body, and the first human body information of the new human body is newly added in the local processing result of the image.

[0053] Further, in order to better update the local processing result of each frame image, when the cloud processing result of each frame image is used to update the local processing result of each frame image, for each frame image in at least part of the frame images, the intersection over union between two human body tracking boxes can be determined according to the positions of the two human body tracking boxes, and an intersection over union matrix can be generated according to each intersection over union, wherein one of the two human body tracking boxes is from the human body tracking box in the local processing result of the image, and the other of the two human body tracking boxes is from the human body tracking box in the cloud processing result of the image; the Hungarian algorithm is used to perform matching based on the intersection over union matrix to obtain a matching result, and the matching result indicates whether the matching between the human body tracking box in the local processing result of the image and the human body tracking box in the cloud processing result of the image is successful; and the cloud processing result of each frame image is used to update the local processing result of each frame image according to the matching result.

[0054] Specifically, the intersection over union (IoU) can quantify the degree of fit of two human body tracking boxes. For each frame image in the frame images, the intersection over union between two human body tracking boxes can be determined according to the positions of the two human body tracking boxes, and one of the two human body tracking boxes is from the human body tracking box in the local processing result of the image, and the other of the two human body tracking boxes is from the human body tracking box in the cloud processing result of the image. For example, the local processing result of a certain frame image indicates that there are 5 human body tracking boxes, and the cloud processing result of the certain frame image indicates that there are 8 human body tracking boxes, and the intersection over union between each human body tracking box in the 5 human body tracking boxes of the local processing result and each human body tracking box in the 8 human body tracking boxes of the cloud processing result is determined.

[0055] After obtaining the multiple intersection over unions corresponding to each frame image, an intersection over union matrix can be generated according to the multiple intersection over unions. The intersection over union matrix includes KxM elements, K and M are positive integers, K is the number of human body tracking boxes in the local processing result, M is the number of human body tracking boxes in the cloud processing result, and one element in the intersection over union matrix corresponds to one intersection over union.

[0056] In this embodiment, the Hungarian algorithm is used to perform matching based on the intersection over union matrix to obtain a matching result, and the matching result indicates whether the matching between the human body tracking box in the local processing result of the image and the human body tracking box in the cloud processing result of the image is successful. The Hungarian algorithm is a classic algorithm for solving the minimum weight matching problem of a bipartite graph.

[0057] In this embodiment, after obtaining the matching result, for the human body tracking frame that matches successfully, that is, the human body tracking frame that appears in both the cloud processing result and the local processing result, the first human body information of the human body corresponding to the human body tracking frame in the local processing result in the image is replaced with the second human body information of the human body corresponding to the human body tracking frame in the cloud processing result. For the human body tracking frame that fails to match, that is, for the human body tracking frame that is newly added in the cloud processing result of the image, the second human body information of the human corresponding to the newly added human body tracking frame is taken as the first human body information of the new human body, and the first human body information of the new human body is newly added in the local processing result of the image.

[0058] In the embodiments of the present application, due to the influence of communication delay, for the images with the same collection time in the video stream, the time for image processing of the image on the mobile robot side is earlier than the time for image processing of the image on the cloud side. When the cloud returns the cloud processing result of each frame of image to the mobile robot, for the images with the same collection time, the local processing result of the image is updated according to the cloud processing result of the image. In this way, even if the mobile robot does not shoot the face at a relatively positive angle, according to the human body tracking frame ID of the human body recognized in a certain frame of image, the face ID of the human body recognized in the certain frame of image can be obtained by querying the local processing result of a plurality of frames of image before or after the certain frame of image, and the face recognition is completed. With the powerful support of the cloud, the mobile robot can perceive more rich human body information, which is beneficial to improve the intelligence and human-computer interaction friendliness of the mobile robot.

[0059] The technical scheme provided in the embodiments of the present application is that the mobile robot is responsible for human body detection and human body tracking, the cloud is responsible for human body detection, human body tracking, face detection, face tracking and face recognition, the cloud processing result updates the local processing result, and through the end-cloud collaborative manner, under the powerful support of the cloud, the mobile robot can perceive more rich human body information, even if the mobile robot does not shoot the face at a relatively positive angle, the face recognition can be completed, the face recognition effect of the mobile robot is improved, and the intelligence and human-computer interaction friendliness of the mobile robot are improved.

[0060] Figure 3 A flowchart of an end-cloud collaborative image processing method provided in the embodiments of the present application. The method is applied to a mobile robot, and the method can include the following steps: Figure 3

[0061] 301, a video stream obtained by image collection on an environment where a mobile robot is located is obtained.

[0062] ​302. Perform human detection and human tracking on each frame of the video stream to obtain the local processing results of each frame. The local processing results of each frame include first human information of at least one human body in the image, and the first human information includes the human body tracking box ID and the position of the human body tracking box.

[0063] 303. In response to the first detection of a human body in an image from a video stream, an identity recognition request for the video stream is sent to the cloud so that the cloud can perform various image processing on each frame of the video stream to obtain the cloud processing results for each frame of the image. The various image processing includes: human body detection, human body tracking, face detection, face tracking and face recognition. The cloud processing results for each frame of at least some of the frames include second human body information of at least one human body in the image. The second human body information includes human body tracking box ID, face tracking box ID, face ID, the position of the human body tracking box and the position of the face tracking box.

[0064] 304. Receive the cloud processing results of each frame of image returned from the cloud, and update the local processing results of each frame of image using the cloud processing results of each frame of image.

[0065] Optionally, the local processing results of each frame image are updated using the cloud processing results of each frame image, including: for each frame image in at least some frames, determining the cross-union ratio (CUP) between two human tracking boxes based on their respective positions, and generating an CUP matrix based on each CUP, wherein one human tracking box in the two human tracking boxes comes from the human tracking box in the local processing results of the image, and the other human tracking box in the two human tracking boxes comes from the human tracking box in the cloud processing results of the image; performing matching based on the CUP matrix using the Hungarian algorithm to obtain a matching result, the matching result indicating whether the human tracking box in the local processing results of the image and the human tracking box in the cloud processing results are successfully matched; and updating the local processing results of each frame image based on the matching result using the cloud processing results of each frame image.

[0066] Optionally, based on the matching results, the local processing results of each frame image are updated using the cloud processing results of each frame image, including: for successfully matched human body tracking boxes, replacing the first human body information of the human body corresponding to the human body tracking box included in the local processing results of the image with the second human body information of the human body corresponding to the human body tracking box included in the cloud processing results; for newly added human body tracking boxes in the cloud processing results of the image, using the second human body information of the newly added human body tracking corresponding to the human body as the first human body information of the new human body, and adding the first human body information of the new human body to the local processing results of the image.

[0067] Further optionally, the detection accuracy of the human body detection function of the mobile robot is lower than the detection accuracy of the human body detection function of the cloud, or the tracking accuracy of the human body tracking function of the mobile robot is lower than the tracking accuracy of the human body tracking function of the cloud.

[0068] The implementation manners and technical effects of the steps in the method provided in the embodiments of the present application can be referred to the related descriptions of the foregoing embodiments, and will not be described here.

[0069] Figure 4 A flowchart of another end-cloud collaborative image processing method provided in the embodiments of the present application. The method is applied to the cloud, and the method can include the following steps: Figure 4

[0070] 401, receiving an identity recognition request including a video stream sent by a mobile robot;

[0071] 402, in response to the identity recognition request, performing multiple image processing on each frame of image in the video stream to obtain a cloud processing result of each frame of image, wherein the multiple image processing includes human body detection, human body tracking, face detection, face tracking and face recognition, and the cloud processing result of each frame of image includes second human body information of at least one human body in the image, and the second human body information includes a human body tracking box ID, a face tracking box ID, a face ID, a position of the human body tracking box and a position of the face tracking box;

[0072] 403, returning the cloud processing result of each frame of image to the mobile robot.

[0073] Further optionally, the cloud adopts a target tracking algorithm based on deep learning, adopts an MTCNN or RetinaFace face detection algorithm, and adopts a yolo target detection model or a Faster R-CNN model for human body detection, wherein the yolo target detection model or the Faster R-CNN model is obtained by fine-tuning a pre-trained yolo target detection model or Faster R-CNN model using a pedestrian detection dataset in a scene served by the mobile robot.

[0074] The implementation manners and technical effects of the steps in the method provided in the embodiments of the present application can be referred to the related descriptions of the foregoing embodiments, and will not be described here.

[0075] It should be noted that the execution subject of each step of the method provided in the foregoing embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 401 to 403 can be device A; for another example, the execution subject of steps 401 and 402 can be device A, and the execution subject of step 403 can be device B; and so on.

[0076] ​In addition, in some of the processes described in this specification, including the processes described in the description and the drawings, multiple operations are described in a particular, sequential order. However, it should be understood that unless otherwise specifically stated in this specification, the order of the operations can be changed, and that the operations can be performed in parallel rather than sequentially. Also, it should be understood that unless otherwise specifically stated in this specification, a specific number of operations need not be performed. For example, in some implementations, an operation can be performed one or more times, or can be omitted. Moreover, it should be understood that unless otherwise specifically stated in this specification, operations can be added to or removed from the processes described in this specification. Also, the processes described in this specification can include more or fewer operations than those described in this specification. In addition, the processes described in this specification can be implemented via one or more computing devices, such as computing device 1000 described in this specification.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0078] Figure 5 A structural schematic diagram of a mobile robot is provided for an embodiment of the present application. As shown in the figure, the mobile robot comprises a memory 51 and a processor 52. Figure 5

[0079] The memory 51 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phonebook data, messages, pictures, videos, etc.

[0080] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0081] ​The processor 52 is coupled with the memory 51 and is configured to execute a computer program stored in the memory 51 to: acquire a video stream obtained by image acquisition of an environment where the mobile robot is located; perform human body detection and human body tracking on each frame image in the video stream to obtain a local processing result of each frame image, wherein the local processing result of each frame image in at least part of the frame images includes first human body information of at least one human body in the image, and the first human body information includes a human body tracking box ID and a position of the human body tracking box; in response to first detecting a human body from the image in the video stream, sending an identity recognition request including the video stream to the cloud to perform a plurality of image processing on each frame image in the video stream to obtain a cloud processing result of each frame image, wherein the plurality of image processing includes human body detection, human body tracking, face detection, face tracking and face recognition, and the cloud processing result of each frame image in at least part of the frame images includes second human body information of at least one human body in the image, and the second human body information includes a human body tracking box ID, a face tracking box ID, a face ID, a position of the human body tracking box and a position of the face tracking box; receiving the cloud processing result of each frame image returned by the cloud, and updating the local processing result of each frame image by using the cloud processing result of each frame image.

[0082] Further optionally, when the processor 52 updates the local processing result of each frame image by using the cloud processing result of each frame image, the processor 52 is specifically configured to: for each frame image in at least part of the frame images, determine an intersection-over-union between two human body tracking boxes according to positions of the two human body tracking boxes, and generate an intersection-over-union matrix according to each intersection-over-union, wherein one of the two human body tracking boxes is from the human body tracking box in the local processing result of the image, and the other of the two human body tracking boxes is from the human body tracking box in the cloud processing result of the image; performing matching based on the intersection-over-union matrix by using the Hungarian algorithm to obtain a matching result, the matching result indicating whether the human body tracking box in the local processing result of the image and the human body tracking box in the cloud processing result of the image are successfully matched; and updating the local processing result of each frame image according to the matching result by using the cloud processing result of each frame image.

[0083] Further optionally, when the processor 52 updates the local processing result of each frame image according to the matching result by using the cloud processing result of each frame image, the processor 52 is specifically configured to: for the human body tracking box that is successfully matched, replacing the first human body information of the human body corresponding to the human body tracking box included in the local processing result of the image with the second human body information of the human body corresponding to the human body tracking box included in the cloud processing result; and for the human body tracking box newly added in the cloud processing result of the image, taking the second human body information of the human body corresponding to the newly added human body tracking as the first human body information of the new human body, and adding the first human body information of the new human body in the local processing result of the image.

[0084] Further optionally, as Figure 5As shown, the mobile robot further includes a communication component 53, a display 54, a power supply component 55, an audio component 56, and other components. Figure 5 Only some components are shown schematically, and it is not meant that the mobile robot only includes Figure 5 the components shown.

[0085] Detailed implementation procedures of the processor performing each action can refer to the related description in the foregoing method embodiments or device embodiments, which will not be described here.

[0086] Figure 6 A structural schematic diagram of a cloud provided by an embodiment of the present application is shown in FIG. 6. Figure 6 As shown, the cloud includes a memory 61 and a processor 62.

[0087] The memory 61 is configured to store computer programs and can be configured to store other various data to support operations on the computing platform. Examples of the data include instructions of any application program or method for operating on the computing platform, contact data, phonebook data, messages, pictures, videos, and the like.

[0088] The memory 61 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random-access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0089] The processor 62 is coupled to the memory 61 and is configured to execute the computer programs in the memory 61 to: receive an identity recognition request including a video stream sent by a mobile robot; in response to the identity recognition request, perform a plurality of image processing on each frame image in the video stream to obtain a cloud processing result of each frame image, wherein the plurality of image processing includes human body detection, human body tracking, face detection, face tracking, and face recognition, the cloud processing result of each frame image includes second human body information of at least one human body in the image, and the second human body information includes a human body tracking box ID, a face tracking box ID, a face ID, a position of the human body tracking box, and a position of the face tracking box; and return the cloud processing result of each frame image to the mobile robot.

[0090] Further optionally, the cloud end adopts a target tracking algorithm based on deep learning, adopts a face detection algorithm of MTCNN or RetinaFace, and adopts a human body detection model of a yolo target detection model or a Faster R-CNN model, wherein the yolo target detection model or the Faster R-CNN model is obtained by fine-tuning a pre-trained yolo target detection model or Faster R-CNN model using a pedestrian detection dataset in a scene served by the mobile robot.

[0091] Further optionally, as shown in Figure 6 the cloud end further includes a communication component 63, a display 64, a power supply component 65, an audio component 66, and other components. Figure 6 The cloud end in the embodiment only shows some components schematically, and does not mean that the cloud end only includes Figure 6 the components shown in the figure. In addition, Figure 6 the components in the dashed box in the embodiment are optional components, rather than mandatory components, and can be determined according to the product form of the cloud end. The cloud end in the embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, or an IOT (Internet of Things) device, or can be implemented as a server device such as a conventional server, a cloud server, or a server array. If the cloud end in the embodiment is implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, etc., the cloud end can include Figure 6 the components in the dashed box in the embodiment; if the cloud end in the embodiment is implemented as a server device such as a conventional server, a cloud server, or a server array, the cloud end can not include Figure 6 the components in the dashed box in the embodiment.

[0092] The detailed implementation process of the processor performing each action can be referred to the related description in the foregoing method embodiments or device embodiments, which will not be described here again.

[0093] Correspondingly, the embodiment of the application further provides a computer readable storage medium storing a computer program, and the computer program is executed to implement each step that can be executed by the computer device in the foregoing method embodiments.

[0094] Correspondingly, the embodiment of the application further provides a computer program product, including a computer program / instruction, when the computer program / instruction is executed by the processor, the processor can implement each step in the method of image processing based on end-cloud cooperation.

[0095] The communication component is configured to facilitate wired or wireless communication between the device on which the communication component is installed and other devices. The device on which the communication component is installed can access a wireless network based on a communication standard or standards, such as WiFi, 2G (2 Generation), 3G (3 Generation), 4G (4 Generation) / LTE (Long Term Evolution), 5G (5 Generation), or the like, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0096] The display includes a screen, which can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touch or a slide action, but also detect a duration and a pressure related to a touch or a slide operation.

[0097] The power supply component provides power to various components of the device on which the power supply component is installed. The power supply component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device on which the power supply component is installed.

[0098] The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device on which the audio component is installed is in a particular mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0099] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0100] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0101] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0103] In one typical configuration, the computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0104] Memory can include non-persistent memory, Random Access Memory (RAM), and / or non-volatile memory, such as Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. In accordance with the teachings herein, a "non-transitory" is not a medium that has a changeable state, and does not persistently store data. In other words, a non-transitory medium does not include a medium that can change state and persistently store data. Memory is an example of a computer-readable medium.

[0105] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0106] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus.

[0107] The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. An edge-cloud collaborative image processing method, characterized in that, When applied to mobile robots, the detection accuracy of human body detection functions on mobile robots is lower than that of human body detection functions in the cloud. The tracking accuracy of the human body tracking function of the mobile robot is lower than that of the human body tracking function in the cloud. The method includes: Acquire video streams obtained from image acquisition of the environment in which the mobile robot is located; Human detection and human tracking are performed on each frame of the video stream to obtain the local processing result of each frame. The local processing result of each frame includes first human information of at least one human body in the image, which includes human body tracking box ID and the position of the human body tracking box. The mobile robot performs human detection at low frequency and human tracking at high frequency. In response to the first detection of a human body in the images of the video stream, an identity recognition request including the video stream is sent to the cloud. The cloud then performs various image processing operations on each frame of the video stream to obtain cloud processing results for each frame. These various image processing operations include: human detection, human tracking, face detection, face tracking, and face recognition. At least some frames of the image processing result include second human information for at least one human body in the image. This second human information includes a human tracking box ID, a face tracking box ID, a face ID, the position of the human tracking box, and the position of the face tracking box. The cloud performs high-frequency human tracking and face recognition. The system performs low-frequency human detection, face detection, and face recognition. For any given frame image, the cloud performs human detection to obtain at least one human body in the image, the human body tracking box ID for each human body, and the position of the human body tracking box. The cloud tracks the position of the human body tracking box corresponding to the detected human body in the image. The cloud performs face detection on the image region within the detected human body tracking box to detect the human body's face tracking box ID, which is used to identify the face tracking box. The cloud uses face recognition to perform face recognition on the face image within the face tracking box to determine the face ID that identifies the human body. The cloud processing results of each frame image returned by the cloud are received, and the local processing results of each frame image are updated using the cloud processing results of each frame image. Specifically, based on the human body tracking box ID of any frame image in the local processing results, the face ID of the human body with the same human body tracking box ID in several frames before or after the acquisition time of the frame image is searched in the local processing results, and used as the face ID of the human body identified in the frame image to complete face recognition.

2. The method according to claim 1, characterized in that, The local processing results for each frame are updated using the cloud processing results for each frame, including: For each frame of at least some frames, the cross-union ratio between the two human body tracking boxes is determined according to the respective positions of the two human body tracking boxes, and an cross-union ratio matrix is ​​generated according to each cross-union ratio. In this case, one human body tracking box in the two human body tracking boxes comes from the human body tracking box in the local processing result of the image, and the other human body tracking box in the two human body tracking boxes comes from the human body tracking box in the cloud processing result of the image. The Hungarian algorithm is used to perform matching based on the intersection-union matrix to obtain the matching result. The matching result indicates whether the human body tracking box in the local processing result of the image and the human body tracking box in the cloud processing result are successfully matched. Based on the matching results, the local processing results of each frame image are updated using the cloud processing results of each frame image.

3. The method according to claim 2, characterized in that, Based on the matching results, the local processing results of each frame image are updated using the cloud processing results of each frame image, including: For a successfully matched human body tracking box, the first human body information of the human body corresponding to the human body tracking box, which is included in the local processing result of the image, is replaced with the second human body information of the human body corresponding to the human body tracking box, which is included in the cloud processing result. For the newly added human body tracking box in the cloud processing result of the image, the second human body information corresponding to the newly added human body tracking is used as the first human body information of the new human body, and the first human body information of the new human body is added in the local processing result of the image.

4. An edge-cloud collaborative image processing method, characterized in that, When applied to the cloud, the detection accuracy of the mobile robot's human detection function is lower than that of the cloud-based human detection function; the tracking accuracy of the mobile robot's human tracking function is lower than that of the cloud-based human tracking function. The mobile robot performs human detection at a low frequency and human tracking at a high frequency. The method includes: Receive identity verification requests, including video streams, sent by mobile robots; In response to the identity recognition request, various image processing methods are applied to each frame of the video stream to obtain the cloud processing results for each frame. These methods include human detection, human tracking, face detection, face tracking, and face recognition. The cloud processing result for each frame includes second human information for at least one human body in the image. This second human information includes a human tracking box ID, a face tracking box ID, a face ID, the position of the human tracking box, and the position of the face tracking box. The cloud performs high-frequency human tracking and face tracking, and low-frequency human detection, face detection, and face recognition. For any given frame, the cloud performs human detection to obtain at least one human body in the image, the human tracking box ID for each human body, and the position of the human tracking box. The cloud tracks the position of the human tracking box corresponding to the detected human body in the image. The cloud performs face detection on the image region within the detected human tracking box to detect the human body's face tracking box ID, which is used to identify the face tracking box. The cloud uses face recognition to perform face recognition on the face image within the face tracking box to determine the face ID that identifies the human body. The cloud processing results of each frame image are returned to the mobile robot so that the mobile robot can update the local processing results of each frame image using the cloud processing results of each frame image. Specifically, based on the human body tracking box ID of any frame image in the local processing results, the face ID of the human body with the same human body tracking box ID in several frames before or after the acquisition time of the frame image is searched in the local processing results and used as the face ID of the human body identified in the frame image to complete face recognition.

5. The method according to claim 4, characterized in that, The cloud-based system employs a deep learning-based target tracking algorithm, an MTCNN or RetinaFace face detection algorithm, and a YOLO target detection model or a Faster R-CNN model for human detection. The YOLO target detection model or Faster R-CNN model is obtained by fine-tuning a pre-trained YOLO target detection model or Faster R-CNN model using a pedestrian detection dataset from the scene served by the mobile robot.

6. An edge-cloud collaborative system, characterized in that, Including mobile robots and the cloud; The mobile robot is used to perform the steps in the method according to any one of claims 1-3; The cloud is used to execute the steps in the method according to any one of claims 4-5.

7. A mobile robot, characterized in that, include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps of the method according to any one of claims 1-3.

8. A cloud platform, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is coupled to the memory for executing the computer program to perform the steps of the method according to any one of claims 4-5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-3.

Images