Method and apparatus for video display based on multiple cameras, and storage medium
By calculating the time stamp difference in the multi-camera video system and filtering abnormal data, the problem of unconnected pictures in the video display is solved, improving the display effect and ensuring the smoothness of the video.
Patent Information
- Application Number
- PCT/CN2023/106251
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2023-07-07
- Publication Date
- 2024-11-21
AI Technical Summary
In multi-camera video display, due to the time difference between the host device and the slave device, the synthesized video image may have problems with screen connection, which reduces the video display effect.
By calculating the difference between the local timestamp and the video timestamp, and retaining video data with a difference less than or equal to the preset value, filtering out data with a difference greater than the preset value, generating videos to be displayed, thereby maintaining the video data The delay is within a small range to avoid screen connection problems.
It improves the display effect of the video, ensures the synthesized video images are coherent and smooth, and enhances the interactive experience of the meeting.
Smart Images

Figure CN2023106251_21112024_PF_FP_ABST
Abstract
Description
Video display method, device and storage medium based on multiple cameras Technical Field
[0001] The present invention relates to the field of video technology, and in particular to a video display method, device and storage medium based on multiple cameras. Background Art
[0002] With the development of technology, people can use various terminal devices to conduct online meetings, such as voice conferencing, video conferencing, etc. Online conferencing does not require people to arrive at the agreed place at the agreed time to conduct the meeting, making the meeting more efficient and convenient.
[0003] For slightly larger conference rooms, multiple slave devices need to be deployed to collect conference data. Then, the host device synthesizes the conference data collected by the slave devices into a video image and displays the video image on the presenter's screen. However, due to the time difference between the host device and the slave devices, the synthesized video image may have problems with picture connection, which reduces the video display effect.
[0004] Summary of the Invention
[0005] The present invention provides a video display method, device and storage medium based on multiple cameras, which can improve the display effect of the video.
[0006] In a first aspect, the present invention provides a video display method based on multiple cameras, comprising:
[0007] Sending acquisition instructions to the multiple slave devices and receiving video data returned by the slave devices;
[0008] Calculate the difference between the local timestamp and the video timestamp of each video data;
[0009] retaining video data where the difference between the local timestamp and the video timestamp is less than or equal to a preset value;
[0010] A video to be displayed is generated based on the retained video data, and the video to be displayed is displayed.
[0011] In a second aspect, the present invention further provides a video display device based on multiple cameras, comprising:
[0012] a receiving module, configured to send acquisition instructions to the plurality of slave devices and receive video data returned by the slave devices;
[0013] A calculation module, configured to calculate a difference between a local timestamp and a video timestamp of each video data;
[0014] A retaining module, configured to retain video data where the difference between the local timestamp and the video timestamp is less than or equal to a preset value;
[0015] A generation module, used to generate a video to be displayed based on the retained video data;
[0016] The display module is used to display the video to be displayed.
[0017] In a third aspect, the present invention further proposes a computer storage medium, wherein a computer program is stored in the computer storage medium. When the computer program is executed, any one of the above-mentioned multi-camera based video display methods is implemented.
[0018] The present invention provides a multi-camera video display method, device, and storage medium. After sending acquisition instructions to the multiple slaves and receiving video data returned by the slaves, the method calculates the difference between the local timestamp and the video timestamp of each video data, then retains the video data for which the difference between the local timestamp and the video timestamp is less than or equal to a preset value. Finally, based on the retained video data, a video to be displayed is generated and the video to be displayed is displayed. In the video display solution provided by the present invention, video data for which the difference between the local timestamp and the video timestamp is greater than a preset value is filtered, so that the time delay between the retained video data is within a smaller range, thereby avoiding the problem of inconsistent screen connections in the synthesized video image, thereby improving the video display effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. The drawings described below are only drawings corresponding to some embodiments of the present invention. For ordinary technicians in this field, without paying any creative work, they can also obtain drawings of other embodiments based on these drawings.
[0020] FIG1 is a schematic diagram of a flow chart of a method for providing a video display based on multiple cameras according to an embodiment of the present invention;
[0021] FIG2 is a schematic diagram of displaying a video to be displayed in a multi-camera based video display method provided by an embodiment of the present invention;
[0022] FIG3 is a schematic structural diagram of an optimization model in a multi-camera video display method according to an embodiment of the present invention;
[0023] FIG4 is a schematic diagram of extracting a portrait image in a multi-camera video display method provided by an embodiment of the present invention;
[0024] FIG5 is a schematic diagram of a scenario of a multi-camera video display method provided by an embodiment of the present invention;
[0025] FIG6 is a schematic structural diagram of a multi-camera video display device according to an embodiment of the present invention;
[0026] FIG7 is another structural diagram of a multi-camera-based video display device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.
[0028] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0029] Referring to FIG1 , a video display method based on multiple cameras of the present invention includes:
[0030] 101. Send acquisition instructions to multiple slave devices and receive video data returned by the slave devices.
[0031] For example, a master machine and slave devices (such as cameras) are connected to the same local area network. The master machine searches for and accesses all Internet Protocol (IP) addresses within the local area network. The slave devices respond to the access requests. The master machine determines whether a slave device has been found based on whether there is a response and establishes communication with the slave device. Then, in response to a conference start operation (such as a user clicking a control to trigger the video conferencing software to start), a capture instruction is generated based on the conference start operation. The master machine then issues the capture instruction to multiple slave devices and receives video data returned by the slave devices based on the capture instruction. The capture instruction carries capture parameters specific to the conference (such as capture duration, capture volume, etc.).
[0032] 102. Calculate the difference between the local timestamp and the video timestamp of each video data.
[0033] It can be understood that due to the time difference between the master machine and the slave device, in order to facilitate subsequent video display, in the present invention, the difference between the local timestamp (i.e., the timestamp of the master machine) and the video timestamp of each video data is calculated. When it is detected that the difference between the local timestamp and the video timestamp is less than or equal to the preset value of the video data, it is retained; when it is detected that the difference between the local timestamp and the video timestamp is greater than the preset value of the video data, it is filtered out.
[0034] 103. Retain video data in which the difference between the local timestamp and the video timestamp is less than or equal to a preset value.
[0035] 104. Generate a video to be displayed based on the retained video data, and display the video to be displayed.
[0036] For example, specifically, scene images and scene sounds can be extracted from the retained video data, and given the scene images and scene sounds, a video to be displayed can be generated. It should be noted that since the distance between the slave device and the people in the conference room is different in the conference room, the collected video data may contain portraits of some people that are particularly large and portraits of some people that are particularly small, resulting in excessive size differences in the portraits of different people in the conference room during subsequent video display, that is, the video display effect is poor. Therefore, the present invention uses the face detection results and the retained video data to generate a video to be displayed. The step of "generating a video to be displayed based on the retained video data" can specifically include:
[0037] (11) extracting scene images and scene sounds from the retained video data;
[0038] (12) Perform face detection on scene images;
[0039] (13) cropping a portrait image corresponding to each face in the scene image based on the detection results;
[0040] (14) Generate a video to be displayed based on the scene sound and portrait image.
[0041] For example, after extracting the scene image and scene sound from the retained video data, face detection is performed on the scene image. Specifically, a face frame corresponding to each face may be generated based on the detection result. Then, according to the coordinates and size of the face frame, a portrait image corresponding to each face is cropped from the scene image. That is, optionally, in some embodiments, the step of “cropping a portrait image corresponding to each face from the scene image based on the detection result” may specifically include:
[0042] (21) constructing a face frame covering the face in the scene image based on the detection result;
[0043] (22) Based on the size information and coordinate information of the face frame, a portrait image corresponding to each face is segmented in the scene image.
[0044] It should be noted that the original image obtained by a single slave device (such as a conference camera) is a panoramic image. For example, by performing face detection on the panoramic image, the two-dimensional coordinates of all faces in the panoramic image and the face size information can be obtained. Optionally, in some embodiments, the face coordinate information and face size information can be determined by the result of face detection, that is, the face detection result is a face frame covering the face. The face coordinate information can be determined according to the position of the face frame. At the same time, the face size can be determined according to the size of the face frame. After that, the portrait image corresponding to each face is segmented in the scene image according to the position and size of the face. That is, the step of "segmenting the portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame" can specifically include:
[0045] (31) determining the position of the face in the face frame in the scene image according to the coordinate information of the face frame;
[0046] (32) determining the face size of the face in the face frame based on the size information of the face frame;
[0047] (33) According to the position and face size, the portrait image corresponding to each face in the scene image is segmented.
[0048] It should be noted that in this embodiment, the purpose of segmenting the scene image based on face size to produce a portrait image corresponding to each face is that each person is at a different distance from the slave device (conference camera). Segmenting based on face size ensures that the face images are of similar size in the final displayed image. Specifically, face images can be segmented in the following ways: Method 1: Predetermine a reference face size, then adjust the size of all faces to the reference face size to obtain a portrait image corresponding to each face. Method 2: Adjust according to the actual face size and adjust all face sizes to similar sizes. For example, the face size of face a is 8x10 inches, the face size of face b is 4x6 inches, and the face size of face c is 16x20 inches. Then, based on the face size of face a, the face size of face b can be enlarged to 10x10 inches, and at the same time, the face size of face c can be reduced to 10x10 inches. It is understandable that due to different shooting angles, it is not necessarily possible to adjust the face sizes of face b and face c to 10x10 inches in the actual adjustment process. Therefore, the actual adjusted face size will be as close to 10x10 inches as possible to ensure the beauty of the picture during subsequent display.
[0049] The above embodiments describe the case where a single camera is in operation. When multiple cameras are in operation, the same face may exist in the segmented portrait image. When the same face is detected, the common three-dimensional coordinates of the facial features and the two-dimensional coordinates of the face can be used to remove duplicate faces. That is, optionally, in some embodiments, the following may be specifically included:
[0050] (41) Detect whether there is the same face in the segmented portrait image;
[0051] (42) when it is detected that the same face exists in the segmented portrait image, obtaining the three-dimensional coordinates of the preset facial features;
[0052] (43) determining the two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame;
[0053] (44) Identifying the orientation of faces in portrait images with the same face based on two-dimensional coordinates and three-dimensional coordinates;
[0054] (45) According to the orientation and size information of the face, a target portrait image is determined among portrait images having the same face.
[0055] The present invention adopts a novel duplicate face detection method, which is specifically as follows: using a preset algorithm, facial images are converted into 128-dimensional arrays. Then, by comparing the arrays corresponding to different facial images, the more similar the arrays are, the more likely they are the same person. That is, optionally, in some embodiments, the step of "detecting whether the segmented portrait images contain the same face" may specifically include:
[0056] (51) normalizing the pixels of the segmented portrait image and extracting the feature vector of the face in the normalized portrait image;
[0057] (52) The eigenvector is converted to a low-dimensional eigenvector;
[0058] (53) Determine the array corresponding to the transformed eigenvector;
[0059] (54) According to the array corresponding to the selected portrait image, determine whether the same face exists in the segmented portrait image.
[0060] It should be noted that the facial information array converted from only one picture collected for each person is not very effective. Therefore, for each person, in some embodiments of the present invention, 10 pictures are collected, each of which is converted into a 128-dimensional array with facial information, and their average is used as the facial feature array of this person. Then, repeated faces are determined based on the facial feature array.
[0061] After face detection, the face recognition algorithm is used to obtain the two-dimensional coordinates of the face in the face frame, which can be specifically the two-dimensional coordinates of the facial features (the center of the eyes, the tip of the nose, and the sides of the mouth). Then, the three-dimensional coordinates of the common facial features are used to perform three-dimensional modeling of the face. Next, the three-dimensional and two-dimensional coordinates of the facial features are combined to obtain the orientation of the face through the PNP algorithm. Finally, based on the orientation and size information of the face, the image with the most upright and largest face in the portrait images with the same face is determined as the target face image. In the conference room scenario, the face is basically on the same horizontal line as the slave machine, so by judging the horizontal orientation of the face, it can be determined whether the face is a side face or a front face.
[0062] Furthermore, in some embodiments, the two-dimensional coordinates of the face and the three-dimensional coordinates of the preset facial features may be used to define a facial region corresponding to the face. Then, a numerical value corresponding to the facial region is determined, and the corresponding orientation is determined based on the numerical value. That is, optionally, in some embodiments, the step of "identifying the orientation of the face in a portrait image having the same face based on the two-dimensional coordinates and the three-dimensional coordinates" may specifically include:
[0063] (61) constructing a conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates;
[0064] (62) determining the face region corresponding to the face in the face frame according to the conversion relationship;
[0065] (63) Normalizing the face area and extracting image features of the normalized image;
[0066] (64) Performing convolution processing on the image features to obtain numerical values corresponding to the image features, and determining the orientation of the face in the portrait image having the same face based on the numerical values.
[0067] For example, specifically, the size of the face area resolution is adjusted to a preset value (such as 224x224), and the image pixels are normalized and converted into the range of 0-1. Then, the image is converted to a standard normal distribution, that is, the mean of the image is pulled to 0.485, 0.456, 0.406, and the standard deviation is [0.229, 0.224, 0.225]. Then, the features of the face area are extracted. The standard resnet50 structure can be used to extract the features. After multiple rounds of convolution of resnet50, a 7x7x2048 feature matrix is obtained. Then, the features are mapped to the face orientation space: after extracting the facial features, the facial features are mapped to the sample space. Simply put, the 7x7x2048 matrix is multiplied by a matrix to convert it into a final 3x3 vector representing the rotation matrix, and the value representing the horizontal direction in the rotation matrix is converted into a specific value from -90 degrees to 90 degrees for the final judgment of the front and side faces.
[0068] In addition, in view of the scenes where there may be people walking in the meeting, in some embodiments of the present invention, pedestrian detection can also be performed on the scene image, and pedestrian frames of pedestrians can be overlaid in the scene image based on the pedestrian detection results. The step of "segmenting the portrait image corresponding to each face in the scene image based on the size information of the face frame and the coordinate information of the face frame" can specifically include: segmenting the portrait image corresponding to each face in the scene image based on the size information of the pedestrian frame, the coordinate information of the pedestrian frame, the size information of the face frame and the coordinate information of the face frame.
[0069] After deduplication and optimal selection, the optimal video to be displayed can be generated based on the number of identified people and the scene sound. Optionally, the speaker can be determined based on the scene sound, and the speaker's facial image can be placed at a preset position in the video. The size of the speaker's facial image is larger than the sizes of the facial images of the remaining people. Subsequently, the positions of the remaining people in the video are randomly arranged to generate the video to be displayed. It should be noted that when it is determined based on the scene sound that the speaker has finished speaking at time t, the size of the speaker's facial image is reduced to ensure that the size of the speaker's facial image after speaking is similar to or the same as the size of the facial images of the remaining people, as shown in Figure 2.
[0070] In order to further understand the face deduplication solution of the present invention, the following is further explained using a conference scenario with multiple conference machines as an example.
[0071] The screen of the conference machine is sent to the processor via WIFI or network cable. After the processor receives the portrait images returned by multiple conference machines, it can combine the portraits of each conference machine and generate the final image. However, two problems will be encountered at this time. One is that different conference machines may return the same portrait image, and the portraits need to be deduplicated. The other problem is that when duplicate portraits appear, the best portrait must be selected to display. This involves the two core algorithms of the processor: a deduplication algorithm and a preferential algorithm after duplicate portraits occur. The present invention provides an optimization model, as shown in Figure 3, which includes a face recognition neural network D1, a face orientation recognition neural network D2, and a pedestrian recognition neural network D3, as follows:
[0072] The face recognition neural network D1 (which converts faces into mathematically expressible arrays) takes as input a 112x112 resolution face rectangle. After inputting the face rectangle, the neural network can return a 128-dimensional array (each number in the array ranges from 0 to 1). By comparing the arrays generated by different faces, the more similar the arrays are, the more likely it is a duplicate face.
[0073] Specifically, the pixels of the face image are normalized: the resolution of the face image is 112x112, and 127.5 is subtracted from each pixel in the RGB image, and then divided by 128.
[0074] Extracting facial feature information: Use CNN to convert the normalized face into a feature vector. We use the standard MobileNetV2 CNN structure for feature extraction, and the generated features are a 7x7x1280 matrix. We can simply understand it this way: the face image is divided into 49 small blocks of 7x7, each of which has a corresponding 1280-dimensional feature vector.
[0075] Converting facial features into low-dimensional feature vectors: For each face image, the CNN above can generate a 7x7x1280 matrix. This matrix is equivalent to digitizing the facial information. By comparing the digitized information generated by different faces, we can determine whether the faces are similar. However, this matrix has 7x7x1280 = 62720 dimensions, so this high-dimensional matrix needs to be converted into a low-dimensional array:
[0076] 1. For each layer of 7x7 data, take a weighted average. The closer the data is to the center of the 7x7, the higher the weight it has when taking the average.
[0077] 2. After taking the weighted average, we get a 1x1280-dimensional array. We perform a 1x1x128 convolution on this array to get the final 128-dimensional array.
[0078] After obtaining the 128-dimensional array of face recognition (each number in the array ranges from 0 to 1), we calculate the cosine distance between the two arrays. The specific formula is as follows. The value range of the cosine distance is 0-1. When this number is less than 0.36, the two arrays are considered similar.
[0079] For example, after a scene image is input into face recognition network D1, face recognition network D1 performs face detection on the scene image and, based on the face detection results, constructs face frames covering the faces in the scene image. Each face frame corresponds to a face. Subsequently, based on the size and coordinate information of the face frame, the face size is adjusted, as shown in FIG4 . The specific adjustment method is similar to that of the previous embodiment and will not be repeated here. Furthermore, face recognition network D1 can also output a 128-dimensional array corresponding to each face, which can be used to determine whether the faces are similar.
[0080] Facial orientation recognition neural network D2 (converts faces into horizontal orientation angles): When repeated faces occur, the better face needs to be selected. The better face should be a straight face, not a side face. Based on the scenario in which the conference machine is used, the face is basically on the same horizontal line as the conference machine, so we only need to judge the horizontal orientation of the face to distinguish whether the face is a side face or a straight face. The input of the algorithm is the same as that of the face recognition neural network, which is also a 112x112 resolution face box. After entering the face box, the neural network can return a number. The number ranges from -90 degrees to +90 degrees, and 0 degrees means that the person's face is the straightest.
[0081] Similarly, the input of the face orientation recognition neural network D2 is also the scene image, and its output is the numerical value of the horizontal orientation of the face image. The face images are screened based on this numerical value. For example, for the same face, it corresponds to face image a, face image b and face image c, and the numerical value of the horizontal orientation of face image a is 6, the numerical value of the horizontal orientation of face image b is 16, and the numerical value of the horizontal orientation of face image c is 3. Then, face image a and face image b can be deleted, and face image c can be retained.
[0082] Pedestrian recognition neural network D3 (converts human portraits into mathematically expressible arrays): Many people wear masks when using conference machines for meetings. With masks, facial recognition accuracy is poor, so we trained a portrait recognition algorithm. The algorithm inputs a 128x256 portrait image (not just a face) and outputs a 256-dimensional array. By comparing the arrays generated by different portraits, the more similar the arrays are, the more likely they are the same person.
[0083] It should be noted that pedestrian recognition is similar to face recognition, which is equivalent to changing the input from a face to a full-body image, but some details are different. When performing face recognition, the cut-out face will basically cover the entire image frame, but the pedestrian image will not. The pedestrian image sometimes only occupies half the image resolution. Optionally, the pedestrian image can be converted to different resolutions, such as 128x256, 64x128, and 32x64, and the image features are extracted from each scale to reduce the impact of the different sizes of the pedestrian images. Afterwards, the OmniScaleNet network structure is used to extract features, and by comparing the features converted from different pedestrian images, it is confirmed whether they are duplicate pedestrian images.
[0084] After the scene image is input into the pedestrian recognition neural network D3, the pedestrian recognition neural network D3 performs face detection on the scene image and constructs a pedestrian frame covering the pedestrian in the scene image based on the pedestrian detection result. It can be understood that, in some embodiments, the pedestrian recognition neural network D3 only detects moving objects to avoid conflicts between the output results of the pedestrian recognition neural network D3 and the output results of the face recognition neural network D1, which affects the subsequent video display.
[0085] The host obtains all video streams and video information, and synthesizes the video image by analyzing the video data and video information and sends it to the other party in the conference. For example, the local conference party collects the scene image p1 containing participants A and B through conference machine a, and the conference machine b collects the scene image p2 containing participants A and B. The host uses its built-in optimization model to process the scene image p1 and the scene image p2 respectively. After deduplication and preferential selection, the image of participant A and the image of participant B are obtained. Subsequently, the image of participant A and the image of participant B are spliced together by combining the microphone data and sound data to obtain synthesized video data, and the synthesized video data is sent to the remote conference party. Finally, the synthesized video data is displayed on the screen of the remote conference party, as shown in Figure 5.
[0086] The multi-camera video display method provided in this embodiment, after issuing acquisition instructions to multiple slaves and receiving video data returned by the slaves, calculates the difference between the local timestamp and the video timestamp of each video data, then retains the video data where the difference between the local timestamp and the video timestamp is less than or equal to a preset value, and finally, generates the video to be displayed based on the retained video data, and displays the video to be displayed. In the video display scheme provided by the present invention, video data where the difference between the local timestamp and the video timestamp is greater than a preset value is filtered, so that the time delay between the retained video data is within a smaller range, avoiding the problem of the synthesized video image having inconsistent pictures, thereby improving the video display effect.
[0087] Accordingly, referring to FIG6 , an embodiment of the present invention provides a multi-camera video display device (hereinafter referred to as a display device), characterized by including:
[0088] The receiving module 201 is used to send acquisition instructions to multiple slave devices and receive video data returned by the slave devices.
[0089] The calculation module 202 is configured to calculate the difference between the local timestamp and the video timestamp of each video data.
[0090] The retaining module 203 is configured to retain the video data for which the difference between the local timestamp and the video timestamp is less than or equal to a preset value.
[0091] The generating module 204 is configured to generate a video to be displayed based on the retained video data.
[0092] Optionally, in some embodiments, the generating module 204 may specifically include:
[0093] an extraction unit, configured to extract scene images and scene sounds from the retained video data;
[0094] A detection unit, configured to perform face detection on a scene image;
[0095] A cropping unit, configured to crop a portrait image corresponding to each face in the scene image based on the detection result;
[0096] The generation unit is used to generate a video to be displayed based on the scene sound and the portrait image.
[0097] Optionally, in some embodiments, the cropping unit may specifically include:
[0098] A construction subunit, configured to construct a face frame covering the face in the scene image based on the detection result;
[0099] The segmentation subunit is used to segment the portrait image corresponding to each face in the scene image according to the size information and coordinate information of the face frame.
[0100] Optionally, in some embodiments, the segmentation subunit can be specifically used to: determine the position of the face in the face frame in the scene image based on the coordinate information of the face frame; determine the face size of the face in the face frame based on the size information of the face frame; and segment the portrait image corresponding to each face in the scene image based on the position and face size.
[0101] Optionally, in some embodiments, referring to FIG. 7 , the display device of the present invention may further include a detection module 206 , which may include:
[0102] a detection unit, configured to detect whether the same human face exists in the segmented portrait image;
[0103] an acquiring unit, configured to acquire the three-dimensional coordinates of the preset facial features when detecting the presence of the same face in the segmented portrait image;
[0104] a first determining unit, configured to determine the two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame;
[0105] A recognition unit for recognizing the orientation of a face in a portrait image having the same face based on two-dimensional coordinates and three-dimensional coordinates;
[0106] The second determining unit determines a target portrait image from portrait images having the same face according to the orientation and size information of the face.
[0107] Optionally, in some embodiments, the detection unit can be specifically used to: normalize the pixels of the segmented portrait image and extract the feature vector of the face in the normalized portrait image; convert the feature vector into a low-dimensional feature vector; determine the array corresponding to the converted feature vector; and determine whether the same face exists in the segmented portrait image based on the array corresponding to the selected portrait image.
[0108] Optionally, in some embodiments, the recognition unit can be specifically used to: construct a conversion relationship between two-dimensional coordinates and the three-dimensional coordinates; determine the face area corresponding to the face in the face frame based on the conversion relationship; normalize the face area and extract image features of the normalized image; perform convolution processing on the image features to obtain numerical values corresponding to the image features, and determine the orientation of the face in the portrait image with the same face based on the numerical values.
[0109] The display module 205 is used to display the video to be displayed.
[0110] As can be seen from the above, in the video display device based on multiple cameras of this embodiment, after the receiving module 201 sends acquisition instructions to multiple slaves and receives the video data returned by the slaves, the calculation module 202 calculates the difference between the local timestamp and the video timestamp of each video data. Then, the retention module 203 retains the video data whose difference between the local timestamp and the video timestamp is less than or equal to the preset value. Finally, the generation module 204 generates the video to be displayed based on the retained video data, and the display module 205 displays the video to be displayed. In the video display scheme provided by the present invention, the video data whose difference between the local timestamp and the video timestamp is greater than the preset value is filtered, so that the time delay between the retained video data is within a smaller range, avoiding the problem that the synthesized video image may have a disconnected picture, thereby improving the video display effect.
[0111] An embodiment of the present invention further provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the controller, the method steps described in any of the above method embodiments are implemented.
[0112] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a computer program. When the computer program is executed, the method steps described in any one of the above method embodiments are implemented.
[0113] In the above-described embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components being combined or integrated into another system, or some features being ignored or not implemented. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interface, which may be electrical, mechanical, or other forms.
[0114] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0115] In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of software functional units. If the integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.
[0116] Based on this understanding, the technical solution of this application, or the contributing portion thereof, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which may be a mobile terminal, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.
[0117] In summary, although the present invention has been disclosed as above in terms of preferred embodiments, the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the concept of the technical solution of the present invention, should be covered by the scope of protection of the present invention.
[0118] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
Claims
1. A video display method based on multiple cameras, comprising: Sending acquisition instructions to the multiple slaves, and receiving video data returned by the slaves; Calculate the difference between the local timestamp and the video timestamp of each video data; Retain video data where the difference between the local timestamp and the video timestamp is less than or equal to a preset value; A video to be displayed is generated based on the retained video data, and the video to be displayed is displayed.
2. The method according to claim 1, wherein determining the video to be displayed based on the retained video data comprises: extracting scene images and scene sounds from the retained video data; Performing face detection on the scene image; Based on the detection result, a portrait image corresponding to each face is cut out from the scene image; A video to be displayed is generated according to the scene sound and the portrait image.
3. The method according to claim 2, wherein the step of cutting out a portrait image corresponding to each face in the scene image based on the detection result comprises: constructing a face frame covering the face in the scene image based on the detection result; According to the size information of the face frame and the coordinate information of the face frame, a portrait image corresponding to each face is segmented in the scene image.
4. The method according to claim 3, wherein segmenting the portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame comprises: Determining the position of the face in the face frame in the scene image according to the coordinate information of the face frame; Determining the face size of the face in the face frame based on the size information of the face frame; According to the position and the face size, a portrait image corresponding to each face is segmented in the scene image.
5. The method according to claim 4, wherein the method further comprises: Detect whether there is the same face in the segmented portrait image; When the same face is detected in the segmented portrait image, the three-dimensional coordinates of the preset facial features are obtained; Determine the two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame; Based on the two-dimensional coordinates and the three-dimensional coordinates, identifying the orientation of the face in the portrait image having the same face; According to the orientation of the face and the size information, in the portrait image having the same face, Target portrait image.
6. The method according to claim 5, wherein the step of identifying the orientation of a face in a portrait image having the same face based on the two-dimensional coordinates and the three-dimensional coordinates comprises: Constructing a conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates; Determine a face region corresponding to the face in the face frame according to the conversion relationship; Normalizing the face region and extracting image features of the normalized image; The image features are convoluted to obtain numerical values corresponding to the image features, and the orientations of the faces in the portrait images having the same face are determined based on the numerical values.
7. The method according to claim 5, wherein the detecting whether there is an identical face in the segmented portrait images comprises: Normalizing the pixels of the segmented portrait image and extracting the feature vector of the face in the normalized portrait image; The feature vector is a low-dimensional feature vector; Determine the array corresponding to the transformed eigenvector; According to the array corresponding to the candidate portrait images, it is determined whether the same human face exists in the segmented portrait images.
8. The method according to claim 3, wherein the method further comprises: Performing pedestrian detection on the scene image; Constructing a pedestrian frame covering the pedestrian in the scene image according to the pedestrian detection result; The method of segmenting a portrait image corresponding to each face in the scene image based on the size information of the face frame and the coordinate information of the face frame includes: segmenting a portrait image corresponding to each face in the scene image based on the size information of the pedestrian frame, the coordinate information of the pedestrian frame, the size information of the face frame and the coordinate information of the face frame.
9. A video display device based on multiple cameras, comprising: A receiving module, used for sending acquisition instructions to the multiple slaves and receiving video data returned by the slaves; A calculation module, used for calculating the difference between the local timestamp and the video timestamp of each video data; A retaining module, used for retaining video data whose difference between the local timestamp and the video timestamp is less than or equal to a preset value; A generation module, used to generate a video to be displayed based on the retained video data; A display module is used to display the video to be displayed.
10. The video display device according to claim 9, wherein the generating module comprises: An extraction unit, used for extracting scene images and scene sounds from the retained video data; A detection unit, used for performing face detection on a scene image; A cropping unit, used for cropping a portrait image corresponding to each face in the scene image based on the detection result; The generating unit is used to generate a video to be displayed according to the scene sound and the portrait image.
11. The video display device according to claim 10, wherein the cropping unit comprises: A construction subunit, configured to construct a face frame covering the face in the scene image based on the detection result; The segmentation subunit is used to segment the portrait image corresponding to each face in the scene image according to the size information of the face frame and the coordinate information of the face frame.
12. The video display device according to claim 11, wherein the segmentation subunit can be specifically used to: determine the position of the face in the face frame in the scene image based on the coordinate information of the face frame; determine the face size of the face in the face frame based on the size information of the face frame; and segment a portrait image corresponding to each face in the scene image based on the position and the face size.
13. The video display device according to claim 12, wherein the video display device further comprises a detection module, wherein the detection module specifically comprises: A detection unit, used for detecting whether the same human face exists in the segmented portrait image; An acquisition unit, configured to acquire the three-dimensional coordinates of the preset facial features when the same face is detected in the segmented portrait image; A first determining unit, configured to determine the two-dimensional coordinates corresponding to the face in the face frame according to the coordinate information of the face frame; A recognition unit, for recognizing the orientation of a face in a portrait image having the same face based on the two-dimensional coordinates and the three-dimensional coordinates; The second determining unit determines a target portrait image from portrait images having the same face according to the orientation and size information of the face.
14. According to the video display device of claim 13, the recognition unit can be specifically used to: construct a conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates; determine the face area corresponding to the face in the face frame according to the conversion relationship; normalize the face area and extract image features of the normalized image; perform convolution processing on the image features to obtain numerical values corresponding to the image features, and determine the orientation of the face in the portrait image with the same face based on the numerical values.
15. According to the video display device of claim 13, the detection unit can be specifically used to: normalize the pixels of the segmented portrait image and extract the feature vector of the face in the normalized portrait image; convert the feature vector into a low-dimensional feature vector; determine the array corresponding to the converted feature vector; and determine whether the same face exists in the segmented portrait image based on the array corresponding to the portrait image to be selected.
Citation Information
Patent Citations
Video data synthesis method and device
CN108881927A
Real-time face abstract service method and system for intelligent video conference terminal
CN111770299A
Video conference implementation method and device, and storage medium
CN113676690A
Virtual video conference method and related device
CN115424156A
Multiway audio-video conferencing
US20190104163A1
