Multi-camera based video presentation method, apparatus and storage medium

Abstract

The application provides a multi-camera-based video display method and device and a storage medium. A collection instruction is sent to a plurality of slaves, and video data returned by the slaves is received. A difference between a local timestamp and a video timestamp of each video data is calculated. Video data with a difference between the local timestamp and the video timestamp less than or equal to a preset value is retained. A video to be displayed is generated according to the retained video data, and the video to be displayed is displayed. The display effect of the video can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video, in particular to a multi-camera-based video display method and device and storage medium. BACKGROUND

[0002] With the development of science and technology, people can use various terminal devices to conduct online meetings, such as voice meetings, video meetings, etc. The online meeting method does not require people to arrive at the agreed location at the agreed time to conduct the meeting, making the meeting more efficient and convenient.

[0003] For a slightly larger conference room, multiple slave devices need to be deployed to collect conference data, and then the master device synthesizes the conference data collected by the slave devices into a video image and displays the video image on the screen of the display party. However, due to the time difference between the master device and the slave device, the synthesized video image may have a problem of not connecting the picture, which reduces the display effect of the video. SUMMARY

[0004] The present application provides a multi-camera-based video display method, device and storage medium, which can improve the display effect of the video.

[0005] In a first aspect, the present application provides a multi-camera-based video display method, comprising:

[0006] issuing a collection instruction to the plurality of slaves and receiving video data returned by the slaves;

[0007] calculating the difference between the local timestamp and the video timestamp of each video data;

[0008] retaining video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value;

[0009] generating a video to be displayed according to the retained video data, and displaying the video to be displayed.

[0010] Optionally, in some embodiments of the present application, the video to be displayed is determined according to the retained video data, comprising:

[0011] extracting scene images and scene sounds from the retained video data;

[0012] performing face detection on the scene images;

[0013] cutting out a portrait image corresponding to each face in the scene image based on the detection result;

[0014] generating a video to be displayed according to the scene sounds and the portrait images.

[0015] Optionally, in some embodiments of the present application, the cutting out of the portrait image corresponding to each face in the scene image based on the detection result comprises:

[0016] constructing a face frame covering the face in the scene image based on the detection result;

[0017] segmenting out the portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame.

[0018] Optionally, in some embodiments of the present application, the segmenting out of the portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame comprises:

[0019] determining the position of the face in the face frame in the scene image according to the coordinate information of the face frame;

[0020] determining the face size of the face in the face frame based on the size information of the face frame;

[0021] segmenting out the portrait image corresponding to each face in the scene image according to the position and the face size.

[0022] Optionally, in some embodiments of the present application, the method further comprises:

[0023] detecting whether the same face exists in the segmented portrait image;

[0024] when detecting that the same face exists in the segmented portrait image, obtaining the three-dimensional coordinates of the preset face feature;

[0025] determining the two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame;

[0026] recognizing the orientation of the face in the portrait image with the same face based on the two-dimensional coordinates and the three-dimensional coordinates;

[0027] determining the target portrait image in the portrait image with the same face according to the orientation of the face and the size information.

[0028] Optionally, in some embodiments of the present application, the recognizing the orientation of the face in the portrait image with the same face based on the two-dimensional coordinates and the three-dimensional coordinates comprises:

[0029] constructing the conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates;

[0030] determining the face area corresponding to the face in the face frame according to the conversion relationship;

[0031] Normalizing the face region, and extracting image features of the normalized image;

[0032] Convolving the image features, obtaining a value corresponding to the image features, and determining the orientation of the face in the portrait image with the same face based on the value.

[0033] Optionally, in some embodiments of the present application, the detection of whether the same face exists in the segmented portrait image comprises:

[0034] Normalizing the pixels of the segmented portrait image, and extracting a feature vector of the face in the normalized portrait image;

[0035] The feature vector is a low-dimensional feature vector;

[0036] Determining an array corresponding to the converted feature vector;

[0037] According to the array corresponding to the candidate portrait image, it is determined whether the same face exists in the segmented portrait image.

[0038] Optionally, in some embodiments of the present application, it further comprises:

[0039] Performing pedestrian detection on the scene image;

[0040] Constructing a pedestrian frame covering the pedestrian in the scene image according to the pedestrian detection result;

[0041] According to the size information of the face frame and the coordinate information of the face frame, the portrait image corresponding to each face is segmented in the scene image, comprising: according to the size information of the pedestrian frame, the coordinate information of the pedestrian frame, the size information of the face frame and the coordinate information of the face frame, the portrait image corresponding to each face is segmented in the scene image

[0042] In a second aspect, the present application further provides an audio synchronization device of a distributed microphone, comprising:

[0043] The receiving module is configured to issue a collection instruction to the plurality of slaves, and receive video data returned by the slaves;

[0044] The computing module is configured to calculate the difference between the local timestamp and the video timestamp of each video data;

[0045] The retaining module is configured to retain the video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value;

[0046] The generating module is configured to generate a video to be displayed according to the retained video data;

[0047] a display module configured to display the to-be-displayed video.

[0048] In a third aspect, the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed to implement the multi-camera-based video display method.

[0049] The multi-camera-based video display method, device and storage medium provided by the present application are characterized in that after the acquisition instruction is sent to the plurality of slave machines and the video data returned by the slave machines is received, the difference between the local timestamp and the video timestamp of each video data is calculated, then the video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value is reserved, finally, the to-be-displayed video is generated according to the reserved video data, and the to-be-displayed video is displayed. In the video display scheme provided by the present application, the video data whose difference between the local timestamp and the video timestamp is greater than the preset value is filtered, so that the time delay between the reserved video data is within a smaller range, and the problem that the synthesized video image may not be connected is avoided, thereby improving the display effect of the video. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments are briefly introduced as follows, and the drawings in the following description are only corresponding drawings of some embodiments of the present application. For those skilled in the art, other drawings of embodiments can also be obtained without creative labor on the basis of these drawings.

[0051] Figure 1 The flowchart of the multi-camera-based video display method provided by the embodiment of the present application is shown in the figure.

[0052] Figure 2 The schematic diagram of displaying the to-be-displayed video in the multi-camera-based video display method provided by the embodiment of the present application is shown in the figure.

[0053] Figure 3 The structure schematic diagram of the optimization model in the multi-camera-based video display method provided by the embodiment of the present application is shown in the figure.

[0054] Figure 4 The schematic diagram of extracting the portrait image in the multi-camera-based video display method provided by the embodiment of the present application is shown in the figure.

[0055] Figure 5 The scene schematic diagram of the multi-camera-based video display method provided by the embodiment of the present application is shown in the figure.

[0056] Figure 6A structural schematic diagram of a multi-camera-based video display device provided by an embodiment of the present application is shown in FIG. 1.

[0057] Figure 7 Another structural schematic diagram of a multi-camera-based video display device provided by an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0060] Referring to FIG. 1, Figure 1 A multi-camera-based video display method provided by an embodiment of the present application includes:

[0061] 101. issuing a collection instruction to multiple slave machines and receiving video data returned by the slave machines.

[0062] For example, the master machine and the slave device (such as a camera) are connected to the same local area network, the master machine searches for all Internet protocol (IP) addresses in the local area network, the slave machine responds to the search, the master machine determines whether the slave machine is found according to whether there is a response, and establishes communication with the slave machine. Then, in response to a conference starting operation (such as a user clicking a certain control to trigger the start of a video conference software), a collection instruction is generated according to the conference starting operation, and then the master machine issues the collection instruction to multiple slave machines and receives video data returned by the slave machines according to the collection instruction, wherein the collection instruction carries collection parameters (such as collection duration, collection volume, etc.) for the conference.

[0063] 102. calculating the difference between the local timestamp and the video timestamp of each video data.

[0064] It can be understood that, due to the time difference between the master machine and the slave device, in order to facilitate subsequent video display, in the present application, the difference between the local timestamp (i.e., the timestamp of the master machine) and the video timestamp of each video data is calculated, and when video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value is detected, it is retained; when video data whose difference between the local timestamp and the video timestamp is greater than the preset value is detected, it is filtered out.

[0065] 103. The video data with a difference between the local timestamp and the video timestamp being less than or equal to a preset value is reserved.

[0066] 104. A video to be displayed is generated according to the reserved video data, and the video to be displayed is displayed.

[0067] For example, specifically, scene images and scene sounds can be extracted from the reserved video data, and the scene images and the scene sounds are given to generate the video to be displayed. It should be noted that because the distances between the slave machine and the personnel in the conference room are different, the images of some personnel in the collected video data can be particularly large, and the images of some personnel can be particularly small, so that the size difference of the images of different personnel in the conference personnel is too large when the video is displayed subsequently, that is, the display effect of the video is poor. Therefore, the face detection result and the reserved video data are used to generate the video to be displayed, and the step of “generating the video to be displayed according to the reserved video data” can specifically include:

[0068] (11) scene images and scene sounds are extracted from the reserved video data;

[0069] (12) face detection is performed on the scene images;

[0070] (13) an image of each face is cropped from the scene images based on the detection result;

[0071] (14) the video to be displayed is generated according to the scene sounds and the image of each face.

[0072] For example, after the scene images and the scene sounds are extracted from the reserved video data, face detection is performed on the scene images. Specifically, a face frame corresponding to each face can be generated based on the detection result, and then an image of each face is cropped from the scene images according to the coordinates and the size of the face frame, that is, in some embodiments, the step of “cropping an image of each face from the scene images based on the detection result” can specifically include:

[0073] (21) a face frame covering a face is constructed in the scene images based on the detection result;

[0074] (22) an image of each face is segmented from the scene images according to the size information of the face frame and the coordinate information of the face frame.

[0075] It should be noted that the original picture obtained by a single slave machine (such as a conference camera) is a panoramic picture. For example, face detection can be performed on the panoramic picture to obtain the two-dimensional coordinates of all faces on the panoramic picture and face size information. Optionally, in some embodiments, the coordinate information and the face size information of the face can be determined through the face detection result, that is, the face detection result is a face frame covering the face, and the coordinate information of the face can be determined according to the position of the face frame. Meanwhile, the size of the face can be determined according to the size of the face frame. After that, according to the position of the face and the size of the face, a portrait image corresponding to each face is segmented in the scene image, that is, the step of “segmenting a portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame” can specifically include:

[0076] (31) determining the position of the face in the scene image in the face frame according to the coordinate information of the face frame;

[0077] (32) determining the face size of the face in the face frame based on the size information of the face frame;

[0078] (33) segmenting a portrait image corresponding to each face in the scene image according to the position and the face size.

[0079] It should be noted that in this embodiment, the purpose of segmenting a portrait image corresponding to each face in the scene image based on the face size is that the distance between each person and the slave machine (conference camera) is different, and segmentation based on the face size can ensure that the size of the face image in the final displayed picture is close. Specifically, the face image can be segmented in the following ways. Way 1: a reference face size is preset, and then all face sizes are adjusted to the reference face size, thereby obtaining a portrait image corresponding to each face. Way 2: the actual face size is adjusted to a close size, for example, the face size of face a is 8x10 inches, the face size of face b is 4x6 inches, and the face size of face c is 16x20 inches. Then, the face size of face b can be enlarged to 10x10 inches based on the face size of face a, and the face size of face c can be reduced to 10x10 inches. It can be understood that due to different shooting angles, the face sizes of face b and face c cannot be adjusted to 10x10 inches in the actual adjustment process. Therefore, the actual adjusted face size is as close to 10x10 inches as possible to ensure the aesthetic degree of the picture in subsequent display.

[0080] The above embodiments describe the case of a single camera, when multiple cameras are used, the segmented portrait image may have the same face, when the same face is detected, the three-dimensional coordinates of the general facial features and the two-dimensional coordinates of the face can be used to remove the repeated faces, that is, optionally, in some embodiments, specifically, the method can further include:

[0081] (41) detecting whether the segmented portrait image has the same face;

[0082] (42) when it is detected that the segmented portrait image has the same face, obtaining the three-dimensional coordinates of the preset facial features;

[0083] (43) determining the two-dimensional coordinates of the face corresponding to the face frame according to the coordinate information of the face frame;

[0084] (44) identifying the orientation of the face in the portrait image with the same face based on the two-dimensional coordinates and the three-dimensional coordinates;

[0085] (45) determining the target portrait image in the portrait image with the same face according to the orientation and size information of the face.

[0086] The application adopts a novel repeated face detection method, specifically as follows: through a preset algorithm, the face image is converted into a 128-dimensional array, then, by comparing the arrays corresponding to different face images, the more similar the arrays are, the more likely they are the same person, that is, optionally, in some embodiments, the step of "detecting whether the segmented portrait image has the same face" can specifically include:

[0087] (51) normalizing the pixels of the segmented portrait image, and extracting the feature vector of the face in the normalized portrait image;

[0088] (52) converting the feature vector into a low-dimensional feature vector;

[0089] (53) determining the array corresponding to the converted feature vector;

[0090] (54) determining whether the segmented portrait image has the same face according to the array corresponding to the selected portrait image.

[0091] It should be noted that the face information array converted from each person's picture is not very effective, so for each person, in some embodiments of the application, 10 pictures will be collected, each picture will be converted into a 128-dimensional face information array, and the average of the arrays will be taken as the face feature array of the person, then, the repeated face determination is based on the face feature array.

[0092] After the face detection is performed, a face recognition algorithm is used to obtain the two-dimensional coordinates of the face in the face frame, which can be the two-dimensional coordinates of the facial features (center of the eyes, tip of the nose, and both sides of the mouth). Then, a three-dimensional model of the face is established using the three-dimensional coordinates of the facial features. Next, the orientation of the face is obtained by combining the three-dimensional coordinates and the two-dimensional coordinates of the facial features using the PNP algorithm. Finally, the most frontal and largest image of the face is determined as the target face image from the portrait images with the same face based on the orientation and size information of the face. In the conference room scenario, the face is generally on the same horizontal line as the camera, so the orientation of the face in the horizontal direction can be used to determine whether the face is a profile or a frontal face.

[0093] Further, in some embodiments, the two-dimensional coordinates of the face and the three-dimensional coordinates of the preset facial features can also be used to define the face region corresponding to the face, and then determine the corresponding value of the face region and determine the corresponding orientation based on the value. In some embodiments, the step of "identifying the orientation of the face in the portrait images with the same face based on the two-dimensional coordinates and the three-dimensional coordinates" can specifically include:

[0094] (61) constructing a conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates;

[0095] (62) determining the face region corresponding to the face in the face frame according to the conversion relationship;

[0096] (63) normalizing the face region and extracting the image features of the normalized image;

[0097] (64) performing convolution processing on the image features to obtain the value corresponding to the image features, and determining the orientation of the face in the portrait images with the same face based on the value.

[0098] For example, the resolution of the face region is adjusted to a preset value (such as 224x224), and the image pixels are normalized to the range of 0-1. Then, the image is converted to a standard normal distribution, i.e., the mean value of the image is pulled to 0.485, 0.456, and 0.406, and the standard deviation is [0.229, 0.224, 0.225]. Then, the features of the face region are extracted, which can be extracted using a standard resnet50 structure. After multiple rounds of convolution by resnet50, a 7x7x2048 feature matrix is obtained. Then, the features are mapped to a face orientation space. After the face features are extracted, they are mapped to a sample space. In simple terms, the 7x7x2048 matrix is multiplied by a matrix to convert it to a 3x3 vector representing a rotation matrix. The value representing the horizontal direction in the rotation matrix is converted to a specific value between -90 degrees and 90 degrees, which is used to determine whether the face is a profile or a frontal face.

[0099] In addition, in some embodiments of the present application, the scene image can also be subjected to pedestrian detection, and a pedestrian frame of the pedestrian can be overlaid in the scene image according to the pedestrian detection result. The step of "segmenting a portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame" can specifically include: segmenting a portrait image corresponding to each face in the scene image according to the size information of the pedestrian frame, the coordinate information of the pedestrian frame, the size information of the face frame and the coordinate information of the face frame.

[0100] After the de-duplication processing and the selection, the video to be displayed can be generated according to the number of recognized persons and the scene sound. Optionally, the speaker can be determined according to the scene sound, and the face image of the speaker can be placed in a preset position of the video, and the size of the face image of the speaker is greater than the size of the face images of the remaining persons. Then, the positions of the remaining persons in the video are randomly arranged, thereby generating the video to be displayed. It should be noted that when the speaker finishes speaking at the time t according to the scene sound, the size of the face image of the speaker is reduced to ensure that the size of the face image of the speaker after speaking is similar to or the same as the size of the face images of the remaining persons, as shown in FIG. 6. Figure 2

[0101] In order to further understand the face de-duplication scheme of the present application, the following will be further described taking the conference scene of multiple conference machines as an example.

[0102] The picture of the conference machine is sent to the processor through WIFI or network cable. After the processor receives the portrait pictures returned by multiple conference machines, the portrait pictures of the conference machines can be combined together to generate a final picture. However, two problems can be encountered at this time. One is that the same portrait picture can be returned by different conference machines, and the portrait picture needs to be subjected to de-duplication processing. The other problem is that when the repeated portrait picture appears, the best portrait picture needs to be selected for display. Here, two core algorithms of the processor are involved: a de-duplication algorithm and a selection algorithm after the repeated portrait picture appears. The present application provides an optimization model as shown in FIG. 7, which includes a face recognition neural network D1, a face orientation recognition neural network D2 and a pedestrian recognition neural network D3, and the details are as follows. Figure 3

[0103] The input of the face recognition neural network D1 (transforming the face into an array that can be expressed by mathematics) is a 112x112 resolution face frame. After inputting the face frame, the neural network can return a 128-dimensional array (the range of each number in the array is 0-1). By comparing the arrays generated by different faces, the more similar the arrays are, the more likely it is that the faces are repeated.

[0104] ​​Specifically, the pixels of the face image are normalized: the resolution of the face image is 112x112, so each pixel in the RGB image is reduced by 127.5 and then divided by 128.

[0105] Extracting facial feature information: Using CNN to transform the normalized face into a feature vector. We use the standard MobileNetV2 structure for feature extraction CNN, which generates a 7x7x1280 matrix of features. We can simply understand it as dividing the face image into 49 small blocks of 7x7, and each small block has a corresponding 1280-dimensional feature vector.

[0106] Converting facial features into low-dimensional feature vectors: For each face image, a 7x7x1280 matrix can be generated using the CNN described above. This matrix is ​​equivalent to digitizing the facial information. By comparing the digitized information generated from different faces, we can determine whether faces are similar. However, this matrix has 7x7x1280 = 62720 dimensions of data, so we need to convert this high-dimensional matrix into a low-dimensional array:

[0107] 1. For each 7x7 layer of data, take a weighted average. The closer the data is to the center of the 7x7, the higher its weight is when taking the average.

[0108] 2. After taking the weighted average, we get a 1x1280-dimensional array. We then perform a 1x1x128 convolution on this array to get the final 128-dimensional array.

[0109] After obtaining the 128-dimensional array for face recognition (each number in the array is in the range of 0-1), we calculate the cosine distance between the two arrays. The specific formula is shown below. The cosine distance ranges from 0 to 1. We consider the two arrays to be similar when this number is less than 0.36, that is, to be a duplicate face. Specifically, as shown in Formula 1, where x and y are the arrays corresponding to the face images.

[0110]

[0111] For example, specifically, after the scene image is input into the face recognition network D1, the face recognition network D1 performs face detection on the scene image and constructs face bounding boxes covering the faces in the scene image based on the face detection results. Each face bounding box corresponds to one face. Subsequently, based on the size and coordinate information of the face bounding box, the size of the face is adjusted, such as... Figure 4 As shown, the specific adjustment method is similar to that in the previous embodiment, and will not be repeated here. In addition, the face recognition network D1 can also output a 128-dimensional array corresponding to each face, which can be used to determine whether faces are similar.

[0112] Face orientation recognition neural network D2 (converts a face to an angle of the face in the horizontal orientation): When face duplication occurs, a better face should be selected, which should be a more frontal face, rather than a profile face. Based on the scene in which the conference machine is used, the face is basically in the same horizontal line as the conference machine, so we only need to determine the orientation of the face in the horizontal direction to determine whether the face is a profile face or a frontal face. The input of the algorithm is consistent with the face recognition neural network, which is also a 112x112 resolution face frame. After inputting the face frame, the neural network can return a number, the range of the number is -90 degrees to +90 degrees, and 0 degrees indicates that the person is the most frontal face.

[0113] Similarly, the input of the face orientation recognition neural network D2 is also the scene image, and its output is a numerical value of the orientation of the face image in the horizontal direction. Based on the numerical value, the face image is filtered, for example, for the same person, the corresponding face image a, face image b and face image c, and the numerical value of the orientation of the face image a in the horizontal direction is 6, the numerical value of the orientation of the face image b in the horizontal direction is 16, and the numerical value of the orientation of the face image c in the horizontal direction is 3, then the face image a and the face image b can be deleted, and the face image c is retained.

[0114] Pedestrian recognition neural network D3 (converts a person image into an array that can be expressed mathematically): Many people wear masks when using the conference machine to hold a meeting, and the accuracy of face recognition is poor after wearing a mask, so a person image recognition algorithm is trained. The input of the algorithm is a 128x256 person image (not just a face), and the output is a 256-dimensional array. By comparing the arrays generated by different person images, the more similar the arrays, the more likely it is the same person.

[0115] It should be noted that pedestrian recognition is similar to face recognition, which is equivalent to changing the input from a face to a full-body picture, but some details are not quite the same. When doing face recognition, the cut-out face will basically cover the complete picture frame, but the pedestrian image will not. The pedestrian image sometimes only occupies half of the resolution of the picture. Optionally, the pedestrian image is converted into different resolutions, such as 128x256, 64x128 and 32x64, to extract the features of the image to reduce the impact caused by the different sizes of the pedestrian image. Then, the network structure of OmniScaleNet is used to extract the features, and by comparing the features converted by different pedestrian images, it is determined whether the pedestrian image is duplicated.

[0116] After the scene image is input to the pedestrian recognition neural network D3, the pedestrian recognition neural network D3 performs face detection on the scene image, and constructs a pedestrian frame covering pedestrians in the scene image based on the pedestrian detection result. It can be understood that in some embodiments, the pedestrian recognition neural network D3 only detects objects that are running, so as to avoid the conflict between the result output by the pedestrian recognition neural network D3 and the result output by the face recognition neural network D1, and affect the subsequent video display.

[0117] The host obtains all video streams and video information, and sends the synthesized video image to the conference opposite party through video data and video information analysis, for example, the local conference party collects the scene image p1 containing the participant A and the participant B through the conference machine a, the conference machine b collects the scene image p2 containing the participant A and the participant B, the host uses the built-in optimization model to process the scene image p1 and the scene image p2 respectively, and after de-duplication processing and optimal selection, the image of the participant A and the image of the participant B are obtained. Then, the image of the participant A and the image of the participant B are spliced to obtain the synthesized video data combined with the microphone data and the sound data, and the synthesized video data is sent to the remote conference party. Finally, the synthesized video data is displayed on the screen of the remote conference party, as shown in Figure 5 .

[0118] The video display method based on multiple cameras provided in the embodiment calculates the difference between the local timestamp and the video timestamp of each video data after issuing the collection instruction to the multiple slaves and receiving the video data returned by the slave, then retains the video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value, and finally generates the video to be displayed according to the retained video data and displays the video to be displayed. In the video display scheme provided in the application, the video data whose difference between the local timestamp and the video timestamp is greater than the preset value is filtered, so that the time delay between the retained video data is within a small range, avoiding the problem that the synthesized video image may not be connected, thereby improving the display effect of the video.

[0119] Correspondingly, please refer to Figure 6 , the embodiment of the application provides a video display device of multiple cameras (hereinafter referred to as a display device), characterized by comprising:

[0120] The receiving module 201 is configured to issue a collection instruction to a plurality of slaves and receive video data returned by the slaves.

[0121] The calculation module 202 is configured to calculate the difference between the local timestamp and the video timestamp of each video data.

[0122] The reservation module 203 is configured to reserve video data with a difference between the local timestamp and the video timestamp less than or equal to a preset value.

[0123] The generation module 204 is configured to generate the video to be displayed according to the reserved video data.

[0124] Optionally, in some embodiments, the generation module 204 can specifically include:

[0125] The extraction unit is configured to extract a scene image and a scene sound from the reserved video data.

[0126] The detection unit is configured to perform face detection on the scene image.

[0127] The cropping unit is configured to crop a portrait image corresponding to each face in the scene image based on the detection result.

[0128] The generation unit is configured to generate the video to be displayed according to the scene sound and the portrait image.

[0129] Optionally, in some embodiments, the cropping unit can specifically include:

[0130] The construction sub-unit is configured to construct a face frame covering the face in the scene image based on the detection result.

[0131] The segmentation sub-unit is configured to segment a portrait image corresponding to each face in the scene image according to size information of the face frame and coordinate information of the face frame.

[0132] Optionally, in some embodiments, the segmentation sub-unit can be specifically configured to: determine a position of the face in the scene image in the face frame according to the coordinate information of the face frame; determine a face size of the face in the face frame based on the size information of the face frame; and segment a portrait image corresponding to each face in the scene image according to the position and the face size.

[0133] Optionally, in some embodiments, referring to Figure 7 The display device of the present application can further include the detection module 206, which can specifically include:

[0134] The detection unit is configured to detect whether the segmented portrait image contains the same face.

[0135] The acquisition unit is configured to acquire three-dimensional coordinates of preset facial features when it is detected that the segmented portrait image contains the same face.

[0136] The first determination unit is configured to determine two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame.

[0137] The recognition unit is configured to recognize the orientation of the face in the portrait image with the same face based on the two-dimensional coordinates and the three-dimensional coordinates.

[0138] The second determination unit is configured to determine the target portrait image in the portrait images with the same face according to the orientation of the face and the size information.

[0139] Optionally, in some embodiments, the detection unit can be specifically configured to: perform normalization processing on the pixels of the segmented portrait image, and extract a feature vector of the face in the normalized portrait image; the feature vector is a low-dimensional feature vector; determine an array corresponding to the converted feature vector; and determine whether the same face exists in the segmented portrait image according to the array corresponding to the candidate portrait image.

[0140] Optionally, in some embodiments, the recognition unit can be specifically configured to: construct a conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates; determine a face area corresponding to the face in the face frame according to the conversion relationship; perform normalization processing on the face area, and extract an image feature of the normalized image; perform convolution processing on the image feature to obtain a numerical value corresponding to the image feature, and determine the orientation of the face in the portrait image with the same face based on the numerical value.

[0141] The display module 205 is configured to display the to-be-displayed video.

[0142] As can be seen from the above, in the video display device based on multiple cameras, after the receiving module 201 sends the collection instruction to the multiple slaves and receives the video data returned by the slaves, the calculation module 202 calculates the difference between the local timestamp and the video timestamp of each video data, then the reservation module 203 reserves the video data with the difference between the local timestamp and the video timestamp less than or equal to the preset value, finally, the generation module 204 generates the to-be-displayed video according to the reserved video data, and the display module 205 displays the to-be-displayed video. In the video display scheme provided by the present application, the video data with the difference between the local timestamp and the video timestamp greater than the preset value is filtered, so that the time delay between the reserved video data is within a small range, avoiding the problem that the synthesized video image may not be connected, thereby improving the display effect of the video

[0143] The embodiment of the present application further provides a computer device, including a processor and a memory, the memory has a computer program stored therein, the computer program is loaded and executed by the controller, and the method steps of any one of the above method embodiments are realized.

[0144] The embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed to realize the method steps in any of the method embodiments of the present application.

[0145] In the above-described embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0146] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0147] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software function unit. If the integrated unit is realized in the form of software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium.

[0148] Based on such an understanding, the technical solutions of the present application, in essence or the parts that make contributions, or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, and includes a number of instructions to make a computer device (which can be a mobile terminal, a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0149] In summary, although the application has been disclosed with preferred embodiments as above, the scope of protection of the application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions of the application within the technical scope disclosed by the application, and such should be covered within the scope of protection of the application.

[0150] The technical features of the above-described embodiments can be combined in any manner. For brevity, not all possible combinations of the technical features in the above-described embodiments are described, but it should be understood that any combination of the technical features is within the scope of the present disclosure as long as the combination does not result in a contradiction.

Claims

1. A multi-camera based video presentation method, characterized by, The method comprises the following steps: issuing a collection instruction to the plurality of slaves and receiving video data returned by the slaves; calculating the difference between the local timestamp and the video timestamp of each video data; retaining video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value; generating a video to be displayed according to the retained video data, and displaying the video to be displayed; wherein the determination of the video to be displayed according to the retained video data comprises: extracting a scene image and a scene sound from the retained video data; performing face detection on the scene image, constructing a face frame covering the face in the scene image based on the detection result, and performing pedestrian detection on the scene image, constructing a pedestrian frame covering the pedestrian in the scene image according to the pedestrian detection result; segmenting a portrait image corresponding to each face in the scene image according to the size information of the pedestrian frame, the coordinate information of the pedestrian frame, the size information of the face frame and the coordinate information of the face frame; wherein different face frames are converted into arrays through a face recognition neural network, a face orientation value in the face frame is obtained through a face orientation recognition neural network, and different pedestrian frames are converted into arrays through a pedestrian recognition neural network to perform a de-duplication operation and an optimization operation on the portrait image; wherein the face recognition neural network outputs a multidimensional array corresponding to each face to determine whether the face is similar; the face orientation recognition neural network outputs a numerical value of the orientation of the face image in the horizontal direction to screen the face image; the pedestrian recognition neural network only performs face detection on the object in motion, and constructs a pedestrian frame covering the pedestrian in the scene image based on the pedestrian detection result; generating a video to be displayed according to the scene sound and the portrait image.

2. The method of claim 1, wherein, The segmentation of the portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame comprises: determining the position of the face in the face frame in the scene image according to the coordinate information of the face frame; determining the face size of the face in the face frame based on the size information of the face frame; segmenting the portrait image corresponding to each face in the scene image according to the position and the face size.

3. The method of claim 2, wherein, The method further comprises the following steps: detecting whether there is a same face in the segmented portrait image; when it is detected that there is a same face in the segmented portrait image, obtaining the three-dimensional coordinates of a preset face feature; determining the two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame; identifying the orientation of the face in the portrait image with the same face based on the two-dimensional coordinates and the three-dimensional coordinates; determining a target portrait image in the portrait image with the same face according to the orientation of the face and the size information.

4. The method of claim 3, wherein, The identification of the orientation of the face in the portrait image with the same face based on the two-dimensional coordinates and the three-dimensional coordinates comprises: constructing a conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates; determining a face area corresponding to the face in the face frame according to the conversion relationship; performing normalization processing on the face area and extracting image features of the normalized image; Convolving the image features to obtain a numerical value corresponding to the image features, and determining the orientation of the face in the portrait image with the same face based on the numerical value.

5. The method of claim 3, wherein, The detection includes: Normalizing the pixels of the segmented portrait image, and extracting a feature vector of the face in the normalized portrait image; The feature vector is a low-dimensional feature vector; Determining an array corresponding to the converted feature vector; According to the array corresponding to the candidate portrait image, it is determined whether there is a same face in the segmented portrait image.

6. An image processing apparatus characterized by comprising: The method includes: A receiving module configured to issue a collection instruction to the plurality of slaves and receive video data returned by the slaves; A calculation module configured to calculate the difference between the local timestamp and the video timestamp of each video data; A retention module configured to retain video data with a difference between the local timestamp and the video timestamp less than or equal to a preset value; A generation module configured to generate a video to be displayed according to the retained video data; A display module configured to display the video to be displayed; The method includes: Extracting a scene image and a scene sound from the retained video data; Performing face detection on the scene image, constructing a face frame covering the face in the scene image based on the detection result, performing pedestrian detection on the scene image, and constructing a pedestrian frame covering the pedestrian in the scene image according to the pedestrian detection result; According to the size information of the pedestrian frame, the coordinate information of the pedestrian frame, the size information of the face frame, and the coordinate information of the face frame, the portrait image corresponding to each face is segmented in the scene image; wherein the different face frames are converted into arrays by a face recognition neural network, the face orientation value in the face frame is obtained by a face orientation recognition neural network, and the different pedestrian frames are converted into arrays by a pedestrian recognition neural network to perform a de-duplication operation and a selection operation on the portrait image; wherein the face recognition neural network outputs a multi-dimensional array corresponding to each face to determine whether the face is similar; the face orientation recognition neural network outputs a numerical value of the orientation of the face image in the horizontal direction to filter the face image; the pedestrian recognition neural network only performs face detection on the object in motion, and constructs a pedestrian frame covering the pedestrian in the scene image based on the pedestrian detection result; According to the scene sound and the portrait image, a video to be displayed is generated.

7. A storage medium, characterized by A computer program is stored thereon, wherein the computer program is executed by a processor to implement the steps of the video display method of any one of claims 1-5.

Images