Video image display method and device, electronic equipment and storage medium

Abstract

The invention provides a video image display method and device, electronic equipment and a storage medium, and relates to the technical field of video image processing, and the method comprises the steps: receiving a talkback video stream sent by first video talkback equipment, and obtaining a decoded image of the talkback video stream; determining a talkback person face in the decoded image, and obtaining a target video image containing the talkback person face from the decoded image, wherein the picture proportion of the target video image is matched with that of the second video talkback equipment; and displaying the target video image on the second video intercom equipment according to the display resolution of the second video intercom equipment. According to the technical scheme provided by the invention, the display effect of the video talkback picture can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a video image display method, device, electronic equipment and storage medium. Background Art

[0002] Video intercom allows people in different places to communicate as if they were face to face. For example, in a building, visitors can use the door phone of the video intercom system to call the indoor video intercom station in the room to establish a video call with the indoor user; during a video conference, the screen containing the speaker may be displayed on the screens of multiple terminals at the same time.

[0003] However, for both devices of the video intercom, the display size and resolution of the video screen are not standardized. For example, in the home video intercom system, the live video screen of the door machine or facial access control device is a vertical screen, while the screen of the video intercom indoor machine is designed to be horizontal in consideration of the aesthetics of indoor installation and the historical lineage of the product, and the live video screen is a horizontal screen. As a result, in the video intercom business, the vertical video screen sent by the door machine or facial access control device needs to be reduced before being displayed on the indoor machine, or stretched to the full screen of the indoor machine, thereby reducing the display effect of the picture. Summary of the invention

[0004] The present invention provides a video image display method, device, electronic device and storage medium, which are used to solve the problem in the prior art that the display effect of the video picture is poor due to the different picture ratios of the devices at both ends of the video intercom, so as to improve the display effect of the video intercom picture.

[0005] The present invention provides a video image display method, comprising:

[0006] Receiving an intercom video stream sent by a first video intercom device, and obtaining a decoded image of the intercom video stream;

[0007] Determine the face of the intercom user in the decoded image, and obtain a target video image containing the face of the intercom user and having a screen ratio matching that of the second video intercom device from the decoded image;

[0008] The target video image is displayed on the second video intercom device at a display resolution of the second video intercom device.

[0009] According to a video image display method provided by the present invention, the step of obtaining a target video image including the face of the intercom person and having a screen ratio matching that of the second video intercom device from the decoded image comprises:

[0010] In the case where the decoded image includes the face of the person who is traveling with the intercom user, with the goal of including the most faces of the person who is traveling with the intercom user, a video capture image including the face of the intercom user and having a screen ratio matching that of the second video intercom device is captured from the decoded image;

[0011] In the case that the face of the companion is not included in the peripheral preset pixel range of the video captured image in the decoded image, the video captured image is determined as the target video image.

[0012] A video image display method provided by the present invention further includes:

[0013] In the case where the face of the companion is included in the peripheral preset pixel range of the video capture image in the decoded image, the video capture image is expanded outward to the peripheral preset pixel range to obtain the target video image.

[0014] According to a video image display method provided by the present invention, the determining of the face of the speaker in the decoded image comprises:

[0015] In the case where the decoded image includes a human face, determining the human face as the face of the interlocutor in the decoded image;

[0016] In the case that the decoded image includes at least two human faces, face intercom information corresponding to the decoded image is obtained, and the intercom face in the decoded image is determined based on the face intercom information.

[0017] According to a video image display method provided by the present invention, the face intercom information includes at least one of face position information, face posture information, face distance information, face lip information and lip sound matching confidence; the step of determining the intercom person's face in the decoded image based on the face intercom information includes:

[0018] For each face in the decoded image, weighted sum each piece of information in the face intercom information corresponding to the face to obtain an intercom confidence score of the face;

[0019] The highest intercom confidence face determined based on the intercom confidence score is determined as the intercom person face.

[0020] A video image display method provided by the present invention further includes:

[0021] If there is a target face whose intercom confidence score is less than a first preset confidence threshold among other faces of the intercom person in the decoded image except the face of the intercom person, the target face is determined as the face of a fellow traveler corresponding to the face of the intercom person;

[0022] Alternatively, when there is a target face whose intercom confidence score is less than the first preset confidence threshold among other faces other than the intercom face in the decoded image, the behavior association confidence between the target face in the decoded image and the intercom face is determined, and the face among the target faces whose behavior association confidence is greater than the second preset confidence threshold is determined as the face of the companion corresponding to the intercom face.

[0023] According to a video image display method provided by the present invention, the step of obtaining face intercom information corresponding to the decoded image includes:

[0024] Acquire the face intercom information corresponding to the decoded image from the decoded data of the intercom video stream, wherein the face intercom information is obtained by performing face analysis on the collected video image by the first video intercom device;

[0025] Alternatively, face analysis is performed on the decoded image to obtain the face intercom information corresponding to the decoded image.

[0026] The present invention also provides a video image display device, comprising:

[0027] A receiving module, used for receiving an intercom video stream sent by a first video intercom device;

[0028] A decoding module, used to obtain a decoded image of the intercom video stream;

[0029] A determination module, used for determining the face of the speaker in the decoded image;

[0030] An acquisition module, used for acquiring a target video image containing the face of the intercom person and having a picture ratio matching that of the second video intercom device from the decoded image;

[0031] A display module is used to display the target video image on the second video intercom device at a display resolution of the second video intercom device.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described video image display methods when executing the computer program.

[0033] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video image display method described in any one of the above is implemented.

[0034] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video image display method as described above is implemented.

[0035] The video image display method, device, electronic device and storage medium provided by the present invention first obtain the decoded image of the intercom video stream sent by the first video intercom device, and determine the face of the intercom person in the decoded image; then obtain the target video image containing the face of the intercom person and whose screen ratio matches that of the second video intercom device from the decoded image, and then display the target video image on the second video intercom device at the display resolution of the second video intercom device. In this way, by obtaining the target video image matching the screen ratio of the second video intercom device and displaying it on the second video intercom device at the display resolution of the second video intercom device, the problem of poor display effect of the video intercom screen caused by the need to reduce the display or change the ratio when the screen ratios of the first video intercom device and the second video intercom device are different is avoided, and the display effect of the video intercom screen is improved. Moreover, when obtaining the target video image, the position of the intercom person's face and the covered screen area are taken into account, so as to ensure that the intercom person's face is included in the obtained target video image, and dynamic screen display based on the intercom person's face can be realized, further improving the display effect of the video intercom screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is one of the application scenario schematic diagrams of the video image display method provided by an embodiment of the present invention;

[0038] Figure 2 This is a second schematic diagram of an application scenario of the video image display method provided by an embodiment of the present invention;

[0039] Figure 3 is a flow chart of a video image display method provided by an embodiment of the present invention;

[0040] Figure 4 It is one of the principle schematic diagrams of capturing a target video image from a decoded image according to an embodiment of the present invention;

[0041] Figure 5 This is the second schematic diagram of the principle of capturing a target video image from a decoded image according to an embodiment of the present invention;

[0042] Figure 6is a structural diagram of a video image display device provided by an embodiment of the present invention

[0043] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be noted that the serial numbers assigned to the objects described in the present invention, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0046] In video intercom application scenarios, such as video conferencing, mobile terminal video calls, video intercom between facial access control devices and video intercom indoor units, etc., the devices at both ends of the video intercom may have different screen ratios. For example, the facial access control device has a 9:16 vertical screen (such as a display resolution of 1080*1920), while the corresponding video intercom indoor unit has a 16:9 horizontal screen (such as a display resolution of 1024*600). In response to this situation, in the related technology, in the intercom service, when the vertical screen sent by the facial access control device is displayed on the video intercom indoor unit, it can be reduced and displayed in a fixed area of ​​the video intercom indoor unit, or it can be proportionally stretched to full-screen display. However, reducing the display will leave a large amount of blank area on the screen of the video intercom indoor unit, and proportionally stretching it to full-screen display will cause the picture content to be deformed, which will reduce the display effect of the picture and affect the user experience.

[0047] Based on this, an embodiment of the present invention provides a video image display method, which decodes an intercom video stream received from a first video intercom device, determines the face of the intercom person in the decoded image, and obtains a video image whose screen ratio matches that of a second video intercom device based on the inclusion of the face of the intercom person from the decoded image, and then displays the video image on the second video intercom device at the display resolution of the second video intercom device. In this way, the intercom video stream sent by the first video intercom device can be displayed on the second video intercom device at a screen ratio adapted to the second video intercom device, which can avoid the problem of poor video intercom screen display effect caused by reduced display or variable ratio stretching, and realizes adaptive intercom screen display based on the face of the intercom person, thereby obtaining a better screen display.

[0048] The video image display method provided in the embodiment of the present invention can be applied to electronic devices, which include video intercom devices, or computing devices such as servers that are connected to the video intercom devices. The video intercom devices may include mobile phones, computers, vehicle-mounted devices, tablet computers, wearable devices, smart home devices, face access control devices, etc.; the servers may include independent servers, cluster servers or cloud servers, etc. The video image display method can also be applied to a video image display device set in an electronic device, which can be implemented by software, hardware or a combination of both.

[0049] For example, Figure 1 One of the application scenario diagrams of the video image display method provided in an embodiment of the present invention is exemplarily shown. The first video intercom device 110, the second video intercom device 120 and the server 130 constitute a video intercom system. The video image display method can be applied to the server 130 therein. The first video intercom device 110 and the second video intercom device 120 can implement video calls through the server 130.

[0050] The server 130 can receive the intercom video stream sent by the first video intercom device 110, and obtain the decoded image of the intercom video stream, then determine the face of the intercom person in the decoded image, and then obtain the target video image containing the face of the intercom person and whose screen ratio matches that of the second video intercom device 120 from the decoded image, and convert the display resolution of the target video image into the display resolution of the second video intercom device 120 and send it to the second video intercom device 120 for display.

[0051] For example, Figure 2 The second schematic diagram of the application scenario of the video image display method provided by the embodiment of the present invention is exemplarily shown. The first video intercom device 110 and the second video intercom device 120 can directly communicate with each other via video intercom to form a video intercom system. The video image display method can be applied to the first video intercom device 110 and the second video intercom device 120 respectively. Taking the application to the second video intercom device 120 as an example, the first video intercom device 110 sends the collected intercom video stream to the second video intercom device 120 in real time. The second video intercom device 120 obtains the decoded image of the intercom video stream and determines the face of the intercom person in the decoded image. Then, the second video intercom device 120 obtains the target video image containing the face of the intercom person and having a screen ratio matching its own from the decoded image, and displays the target video image after converting the display resolution of the target video image to its own display resolution.

[0052] It can be understood that, in the case that the first video intercom device 110 also has a video display function, the intercom video stream captured by the second video intercom device 120 can also be sent to the first video intercom device 110 in real time, and the first video intercom device 110 can perform the same processing as the second video intercom device 120 on the intercom video stream to display the decoded image of the video stream as a video image adapted to its own screen ratio and display resolution.

[0053] In a video intercom system, the video intercom devices at both ends can be capable of receiving and displaying the video images of the other end, such as video conferencing scenarios, mobile terminal video call scenarios, etc.; or only one end of the video intercom device can display the video image, and the other end can collect the video image, such as a video intercom system composed of a face access control device and a video intercom indoor unit, in which only the video intercom indoor unit can display the video image collected by the face access control device. Based on this, it can be understood that the video image display method provided in the embodiment of the present invention can be applied to the video intercom device capable of displaying the video image in the video intercom system.

[0054] Combine the following Figure 3-Figure 5 The video image display method of the present invention is described.

[0055] Figure 3 The schematic diagram of the flow chart of the video image display method provided by the embodiment of the present invention is exemplarily shown. Taking the video image display method applied to the second video intercom device capable of displaying video images in the video intercom system as an example, referring to Figure 3 As shown, the video image display method may include the following steps 310 to 340.

[0056] Step 310: Receive an intercom video stream sent by a first video intercom device, and obtain a decoded image of the intercom video stream.

[0057] The first video intercom device collects video images of the monitored area in real time, or collects video images and voice data, generates an intercom video stream, and sends the intercom video stream to the second video intercom device with which the video intercom is conducted. The second video intercom device can decode the received intercom video stream to obtain a decoded image.

[0058] Step 320: Determine the face of the speaker in the decoded image.

[0059] Exemplarily, after obtaining the decoded image, the number of faces in the decoded image can be determined first. If the decoded image includes only one face, the face is determined as the intercom face; if the decoded image includes at least two faces, the intercom face can be determined based on the face intercom information of each face. The face intercom information can include at least one of face position information, face posture information, face distance information, face lip information, and lip-sound matching confidence.

[0060] Among them, the face position information represents the image position of the face in the image, for example, it can be the relative distance relative to the center position of the image; the face posture information includes the pitch angle and left and right rotation angle of the face relative to the front of the camera of the first video intercom device; the face distance information is the distance between the face and the device; the face lip information can represent the frequency of face lip movement; the lip sound matching confidence can reflect the degree of matching between the face lip movement and the corresponding voice.

[0061] Exemplarily, a priority may be set for each item of information in the face intercom information, and the intercom person's face in the decoded image may be determined based on the priority. For example, the face intercom information may include face position information and face distance information, and the face distance information has a higher priority than the face position information. For each face in the decoded image, the face closest to the device may be determined as the intercom person's face based on the face distance information; if the face distance information of each face is the same, the face position information may be further compared, and the face with the smallest relative distance to the center of the image may be determined as the intercom person's face.

[0062] Exemplarily, the face of the intercom user may also be determined by the frequency of lip movements of the face within a period of time and / or the orientation information of the face:

[0063] In this embodiment, after determining the facial information in the decoded image, if the lip movement frequency corresponding to the facial information reaches a preset threshold, it can be determined that the face of the intercom whose movement frequency reaches the preset threshold is the intercom face; or, if the direction information corresponding to the facial information is pointing to the first intercom device, it is determined that the face corresponding to the first intercom device is the intercom face.

[0064] Furthermore, if the decoded image contains multiple faces whose lip movement frequencies reach a preset threshold, the face facing the first video intercom device can be determined as the intercom face based on its corresponding direction information; or, if the decoded image contains multiple faces whose direction information is pointing to the first intercom device, the face whose lip movement frequency reaches a preset threshold can be determined as the intercom face based on its corresponding lip movement frequencies.

[0065] Furthermore, if the decoded image contains multiple faces whose lip movement frequency reaches a preset threshold within a period of time, and the orientation information of the multiple faces is all pointing to the first video intercom device, then these multiple faces can be considered to be the faces of the intercom person. When displayed in the second video intercom device, the relative centers of the multiple intercom faces (which can be determined by analyzing the coordinates of the decoded image) are displayed at the relative center of the display device of the second video intercom device.

[0066] Among them, the orientation information can be obtained by analyzing the pitch angle and left and right rotation angle of the face relative to the front of the camera of the first video intercom device, or by eye tracking of the first video intercom device, without limitation here.

[0067] Exemplarily, for each face in the decoded image, the intercom confidence score of the face can be determined based on the face intercom information. For example, each item of information in the face intercom information can be represented by a numerical value, and each item of information in the face intercom information can be summed to obtain the intercom confidence score, or each item of information can be weighted and summed to obtain the intercom confidence score, and then the face with the highest intercom confidence determined based on the intercom confidence score is determined as the intercom face.

[0068] Exemplarily, the face intercom information may include at least one of face position information, face posture information, face distance information, and face lip information, as well as lip sound matching confidence. For each face in the decoded image, the second video intercom device may perform weighted summation on each piece of information in the face intercom information except the lip sound matching confidence to obtain an intercom confidence score, wherein the higher the intercom confidence score, the lower the confidence is considered, and then sort the intercom confidence scores of each face in the decoded image from small to large, and determine the faces corresponding to the first preset number of intercom confidence scores as faces to be determined. The intercom video stream may include video images and voice data. The second video intercom device can obtain the voice data after decoding the intercom video stream, and then perform voice recognition on the voice data to obtain voice text content. For each face to be judged, the voice text content can be matched with the face to be judged to obtain the lip and sound matching confidence of the face to be judged, and then the face with the highest lip and sound matching confidence among the faces to be judged is determined as the face of the intercom person. The higher the lip and sound matching degree, the higher the lip and sound matching confidence.

[0069] Step 330: Obtain a target video image from the decoded image that includes the face of the intercom user and has a screen ratio that matches that of the second video intercom device.

[0070] For example, a target video image containing the face of the intercom user and having a screen ratio matching that of the second video intercom device may be captured from the decoded image. Figure 4One of the principle schematic diagrams of intercepting a target video image in a decoded image is exemplified. Taking the case where the live video image captured by the first video intercom device is a vertical image with a screen ratio of 9:16 and the live video image captured by the second video intercom device is a horizontal image with a screen ratio of 16:9 as an example, refer to Figure 4 As shown, the second video intercom device decodes the intercom video stream sent by the first video intercom device to obtain a decoded image 41, from which the intercom person's face 42 can be determined. At this time, the second video intercom device can capture a horizontal image area 43 containing the intercom person's face 42 and with a screen ratio of 16:9 from the decoded image 41, and determine the horizontal image area 43 as the target video image.

[0071] Exemplarily, the decoded image may be filled, such as by expanding and filling the background portion outward, so as to adjust the aspect ratio of the decoded image to match the aspect ratio of the second video intercom device.

[0072] For example, assuming that the aspect ratio of the decoded image is 16:9, the aspect ratio of the second video intercom device is 4:3, and the display resolution of the first video intercom device is smaller than that of the second video intercom device, the background image of the decoded image can be expanded and filled outward in the height direction, and the aspect ratio of the decoded image can be adjusted to 4:3 by filling, and its display resolution can be adjusted to the same as that of the second video intercom device. For example, the background image can be intelligently analyzed to generate a filling image that matches the scene of the background image for filling. Alternatively, a filling image can be generated based on the facial information in the decoded image. For example, if the face of the intercom person in the decoded image is not a complete face, the missing part of the intercom person's face can be obtained by using face mirroring, that is, a face supplementary image can be obtained, and then a filling image is generated based on the face supplementary image to fill the decoded image.

[0073] Exemplarily, if the aspect ratio of the decoded image is the same as that of the second video intercom device, but the display resolution is different, the display resolution of the decoded image can be adjusted to the same as the display resolution of the second video intercom device by geometric compression or geometric stretching.

[0074] Step 340: Display the target video image on the second video intercom device at the display resolution of the second video intercom device.

[0075] After obtaining the target video image, the display resolution of the target video image can be adjusted to the display resolution of the second video intercom device, and then displayed on the second video intercom device.

[0076] For example, Figure 4For example, assuming that the display resolution of the live video image of the second video intercom device is 1024*600, if the display resolution of the decoded image 41 is greater than 1024*600, the 1024*600 horizontal image area 43 can be directly cut out from the decoded image 41, and the horizontal image area 43 can be directly displayed on the second video intercom device. If the display resolution of the maximum horizontal image area 43 cut out from the decoded image 41 is less than 1024*600, the cut horizontal image area 43 can be proportionally enlarged to a display resolution of 1024*600 and then displayed on the second video intercom device.

[0077] The video image display method provided by the embodiment of the present invention first obtains the decoded image of the intercom video stream sent by the first video intercom device, and determines the face of the intercom person in the decoded image; then obtains the target video image containing the face of the intercom person and whose screen ratio matches that of the second video intercom device from the decoded image, and then displays the target video image on the second video intercom device at the display resolution of the second video intercom device. In this way, by obtaining the target video image matching the screen ratio of the second video intercom device and displaying it on the second video intercom device at the display resolution of the second video intercom device, the problem of poor display effect of the video intercom screen caused by the need to reduce the display or change the ratio when the screen ratios of the first video intercom device and the second video intercom device are different is avoided, and the display effect of the video intercom screen is improved. Moreover, when obtaining the target video image, the position of the intercom person's face and the covered screen area are taken into account, so as to ensure that the intercom person's face is included in the obtained target video image, and dynamic screen display based on the intercom person's face can be realized, further improving the display effect of the video intercom screen.

[0078] based on Figure 3 The video image display method of the corresponding embodiment, in an exemplary embodiment, determining the face of the intercom person in the decoded image may include: when the decoded image includes at least two faces, obtaining face intercom information corresponding to the decoded image, and determining the face of the intercom person in the decoded image based on the face intercom information; when the decoded image includes one face, determining the face as the face of the intercom person in the decoded image.

[0079] Among them, the face intercom information may include at least one of face position information, face posture information, face distance information, face lip information and lip sound matching confidence.

[0080] Exemplarily, the face intercom information can be obtained by performing face analysis on the collected video image by the first video intercom device. The first video intercom device can associate the face intercom information with the corresponding video image, carry it in the intercom video stream and send it to the second video intercom device. The second video intercom device decodes the received intercom video stream and can obtain the decoded image and the face intercom information corresponding to the decoded image from the decoded data of the intercom video stream.

[0081] For example, taking the first video intercom device as a face access control device and the second video intercom device as a video intercom indoor unit, the face access control device can analyze the face intercom information before starting the video intercom with the video intercom indoor unit and before sending each frame of video code stream to the video intercom indoor unit in real time, and perform face analysis such as face recognition, face distance detection and face lip movement detection on each frame of video image collected in real time to obtain the face intercom information of each frame of video image to be sent. After the video intercom is started, each frame of video image in the intercom video stream can carry the corresponding face intercom information and transmit it to the video intercom indoor unit. After the video intercom indoor unit receives the intercom video stream sent by the face access control device, it decodes the intercom video stream to obtain the decoded image and the face intercom information corresponding to the decoded image.

[0082] Exemplarily, the face intercom information can be obtained by the second video intercom device performing face analysis on the decoded image. Specifically, the first video intercom device can collect video images of the monitored area in real time, or collect video images and voice data, and generate an intercom video stream from the collected data and send it to the second video intercom device with which the video intercom is conducted. After receiving the intercom video stream, the second video intercom device decodes the intercom video stream to obtain a decoded image, and then recognizes the face in the decoded image. When it is recognized that the decoded image includes at least two faces, the decoded image is subjected to face analysis such as face recognition, face distance detection, and face lip movement detection to obtain the face intercom information corresponding to the decoded image.

[0083] Exemplarily, when the decoded image includes only one face, the target video image including the face and having a screen ratio matching that of the second video intercom device can be directly captured from the decoded image, and then displayed on the second video intercom device at the display resolution of the second video intercom device.

[0084] For example, taking the display resolution of the live video image captured by the first video intercom device as 1080*1920 and the display resolution of the live video image captured by the second video intercom device as 1024*600, when the decoded image includes a face, the second video intercom device can be centered on the face and capture the target video image containing the face and with a screen display resolution of 1024*600 in the decoded image.

[0085] based on Figure 3 The video image display method of the corresponding embodiment, in an exemplary embodiment, determining the intercom person's face in the decoded image based on the face intercom information may include: for each face in the decoded image, weighting and summing each information in the face intercom information corresponding to the face to obtain an intercom confidence score for the face; and determining the highest intercom confidence face determined based on the intercom confidence score as the intercom person's face.

[0086] Further optionally, the video image display method may also include: in a case where there is a target face whose intercom confidence score is less than a first preset confidence threshold among other human faces other than the face of the interlocutor in the decoded image, determining the target face as the face of a companion corresponding to the face of the interlocutor; or, in a case where there is a target face whose interlocutor confidence score is less than a first preset confidence threshold among other human faces other than the face of the interlocutor in the decoded image, determining the behavior association confidence between the target face and the face of the interlocutor in the decoded image, and determining the face among the target faces whose behavior association confidence is greater than a second preset confidence threshold as the face of a companion corresponding to the face of the interlocutor.

[0087] In this way, it is possible to determine whether the decoded image includes the face of the person who is talking on the other side corresponding to the face of the person who is talking on the other side based on the face intercom information. At the same time, by further judging the target face using the confidence of the behavior association between the target face and the face of the person who is talking on the other side, the accuracy of the face detection of the person who is talking on the other side can be improved. Exemplarily, when the intercom video stream includes voice data, the voice data can be subjected to voice recognition to obtain the voice text content. After obtaining the decoded image of the intercom video stream, for each face in the decoded image, the left and right rotation angle, pitch angle and position information of the face in the decoded image can be determined, and the horizontal position information and vertical position information of the face in the decoded image relative to the center position of the decoded image can be determined based on the position information; based on the size of the face and the shooting parameters of the camera that collects the decoded image, the distance of the face from the camera is determined to obtain the face distance information; the movement frequency of the lips of the face in a set time period is detected to obtain the face lip information; in the case that the intercom video stream includes voice data, the voice text content corresponding to the voice data is matched with the face to obtain the lip matching confidence of the face; at least one of the left and right rotation angle, pitch angle, horizontal position information, vertical position information, face distance information, face lip information and lip matching confidence is weighted and summed to obtain the intercom confidence score of the face. Among them, the left and right rotation angle and pitch angle represent the face posture information; the horizontal position information and vertical position information represent the face position information.

[0088] For example, based on the face intercom information, the intercom confidence score Q of the face can be determined using the following formula (1):

[0089] Q=|x-x0|×C 11 +|y-y0|×C 12 +α×C 21 +β×C 22 +S×C3+E×C4+F×C5 (1)

[0090] Where (x, y) represents the face position information in the decoded image, (x0, y0) represents the center position of the decoded image; C 11 represents the weight of the horizontal position of the face, C 12 represents the weight of the vertical position of the face; α represents the left and right rotation angle of the face relative to the front, C 21 represents the weight of the left and right rotation angle; β represents the pitch angle of the face relative to the front, C 22 represents the weight of the pitch angle; S represents the face distance information, C3 represents the weight of the face distance information; E represents the face lip information, C4 represents the weight of the face lip information; F represents the lip sound matching confidence, C5 represents the weight of the lip sound matching confidence. It can be understood that α and β represent the face posture information.

[0091] The face lip information E can reflect the movement frequency of the lips, and the higher the movement frequency, the higher E. For example, when the lips do not move, E can be set to 10, and when the lips are always moving, E can be set to 0.

[0092] It can be understood that the face intercom information includes at least one of face position information, face posture information, face distance information, face lip information and lip-sound matching confidence. When the intercom confidence score Q is determined using the above formula (1), Q is calculated based on the information items included in the face intercom information. For example, if the face intercom information only includes face position information and face posture information, then when Q is determined using formula (1), the three items S, E and F can be considered to be 0.

[0093] It can be understood that the closer the position of the face in the decoded image is to the center of the decoded image, the smaller the left and right rotation angle of the face relative to the front, and the smaller the pitch angle of the face relative to the front, the higher the intercom confidence of the face. Based on this, combined with formula (1), in the embodiment of the present invention, the higher the intercom confidence score, the lower the intercom confidence, and vice versa, the lower the intercom confidence score, the higher the intercom confidence. The degree of intercom confidence can be determined based on the intercom confidence score.

[0094] Based on this, combined with formula (1), the face with the lowest intercom confidence score in the decoded image (that is, the face with the highest intercom confidence) can be determined as the face of the intercom person. For other faces in the decoded image other than the face of the intercom person, if there is a target face with an intercom confidence score less than a first preset confidence threshold, the target face can be determined as the face of a person who is traveling with the face of the intercom person. Alternatively, the behavioral association confidence between these target faces and the face of the intercom person can be further determined, and the face with a behavioral association confidence greater than a second preset confidence threshold among the target faces can be determined as the face of a person who is traveling with the face of the intercom person. Faces other than the face of the intercom person and the faces of the traveling persons in the decoded image are faces of unrelated persons.

[0095] For example, after determining the target face in the decoded image, the feature value of at least one associated behavior feature between the target face and the interlocutor's face can be obtained, and the behavioral association confidence between the target face and the interlocutor's face can be determined based on the feature values ​​of these associated behavior features. For example, the sum of these feature values ​​can be determined as the behavioral association confidence, or these feature values ​​can be weighted and summed to obtain the behavioral association confidence. The associated behavior feature can include at least one of a face direction feature, a gaze feature, and a coordinated action feature, but is not limited thereto.

[0096] Based on the video image display method of each of the above embodiments, in an exemplary embodiment, considering that the intercom person may have companions, when displaying the video image, the surrounding companions can be included as much as possible under the premise of ensuring that the displayed video image includes the intercom person, so as to improve the effectiveness of the picture. Specifically, obtaining a target video image containing the face of the intercom person and having a screen ratio matching that of the second video intercom device from a decoded image may include: when the decoded image includes the faces of the companions corresponding to the face of the intercom person, taking the inclusion of the most faces of the companions as the goal, a video capture image containing the face of the intercom person and having a screen ratio matching that of the second video intercom device is captured in the decoded image; when the faces of the companions are not included within the peripheral preset pixel range of the video capture image in the decoded image, the video capture image is determined as the target video image.

[0097] Specifically, it can be determined based on the face intercom information whether the decoded image includes the face of the person who is intercommingling and corresponds to the face of the person who is intercommingling.

[0098] Further optionally, when the face of the companion is included in the peripheral preset pixel range of the video capture image in the decoded image, the video capture image is expanded outward to the peripheral preset pixel range to obtain the target video image.

[0099] The peripheral preset pixel range may include a longitudinal peripheral preset pixel range and / or a transverse peripheral preset pixel range of the video capture image.

[0100] For example, the live video image captured by the first video intercom device is a vertical image with a screen ratio of 9:16, and the live video image captured by the second video intercom device is a horizontal image with a screen ratio of 16:9. Figure 5 The second schematic diagram of the principle of intercepting the target video image in the decoded image is exemplified, referring to Figure 5 As shown, assuming that the display resolution of the live video image captured by the first video intercom device is 1080*1920 and the display resolution of the live video image captured by the second video intercom device is 1024*600, the second video intercom device decodes the intercom video stream sent by the first video intercom device to obtain a decoded image 501, from which the intercom person's face 51, the companion person's face 52, the companion person's face 53 and the companion person's face 54 can be determined. According to the face confirmation result, the second video intercom device can capture a 1024*600 video capture image area in the decoded image 501 that includes the intercom person's face 51 and can cover as many surrounding companion person's faces as possible as a target video image for live display, such as capturing the video capture image 502.

[0101] For example, a peripheral preset pixel range may be set, for example Figure 5 In the example of , a vertical peripheral preset pixel range P can be set. If there are faces of people who are traveling with the speaker 51 in the image area of ​​the vertical peripheral preset pixel range P of the video intercepted image 502, the video intercepted image 502 can be appropriately expanded vertically to include as many faces of these people as possible. Figure 5 If it is detected that there is a companion face 54 within the preset pixel range P in the vertical periphery, the video captured image 502 can be expanded outward by P pixels in the up and down directions to obtain image 503, and image 503 is used as the target video image.

[0102] For example, if the display resolution of the video intercepted image 502 is 1024*600, and there is at least one person's face in the range of 100 pixels above and below the vertical area of ​​the video intercepted image 502, the video intercepted image 502 can be expanded outward by 100 pixels in the vertical direction to intercept the image area of ​​1024*800, and obtain the target video image 503. The second video intercom device can scale the target video image 503 to 1024*600 for display.

[0103] Exemplarily, the direction and position of the face of the companion in the peripheral preset pixel range P relative to the video intercepted image 502 may be ignored. As long as the face of the companion in the peripheral preset pixel range P is detected, the video intercepted image 502 may be expanded outward by P pixels in both the longitudinal and lateral directions. At this time, the aspect ratio of the image formed is still 16:9, and it is directly displayed at a display resolution of 1024*600. It is understandable that in the process of outward expansion of the video intercepted image 502, if it has been expanded outward in the lateral or longitudinal direction to the boundary of the decoded image 501, the expansion in that direction is stopped, and the boundary is used as the boundary of the interception in that direction.

[0104] In an optional implementation, a video capture image containing the face of the person intercom and having a picture ratio matching that of the second video intercom device can be captured from the decoded image with the goal of containing the most faces of the companions and with the face of the person intercoming as the center.

[0105] The video image display method provided by the embodiment of the present invention can analyze face intercom information such as face position information, face posture information, face distance information, face lip information and lip sound matching confidence of the video image in real time during the video intercom process, determine the current intercom confidence score of each person in the video image based on these face intercom information, distinguish the face of the intercom person, the faces of fellow personnel and the faces of unrelated personnel based on the intercom confidence score, and use the distinguished faces to dynamically crop the optimal intercom picture to achieve the display of the adaptive optimal intercom picture, which can effectively improve the display effect of the intercom picture.

[0106] The video image display method provided by the embodiment of the present invention can be applied to video intercom fields such as access control intercom systems, video conferencing, video calls (especially mobile video calls based on small screens such as mobile phones), and has a wide range of applications.

[0107] The video image display device provided by the present invention is described below. The video image display device described below and the video image display method described above can be referred to each other.

[0108] Figure 6 The schematic diagram of the structure of the video image display device provided by the embodiment of the present invention is exemplarily shown. Figure 6 As shown, the video image display device may include: a receiving module 610, used to receive an intercom video stream sent by a first video intercom device; a decoding module 620, used to obtain a decoded image of the intercom video stream; a determination module 630, used to determine the face of the intercom person in the decoded image; an acquisition module 640, used to obtain a target video image containing the face of the intercom person and having a screen ratio matching that of the second video intercom device from the decoded image; and a display module 650, used to display the target video image on the second video intercom device at the display resolution of the second video intercom device.

[0109] In an exemplary embodiment, the acquisition module 640 includes: a capture unit, which is used to capture a video capture image containing the face of the intercom person and whose screen ratio matches that of the second video intercom device in the decoded image, with the goal of containing the most faces of the companions, when the decoded image includes the face of the companion corresponding to the face of the intercom person; and a first determination unit, which is used to determine the video capture image as the target video image, when the video capture image in the decoded image does not include the face of the companion within a peripheral preset pixel range.

[0110] In an exemplary embodiment, the acquisition module 640 also includes: a second determination unit, which is used to expand the video capture image outward to the peripheral preset pixel range of the video capture image in the decoded image to obtain the target video image when the peripheral preset pixel range of the video capture image includes the face of the companion.

[0111] In an exemplary embodiment, the determination module 630 may include: a third determination unit, which is used to determine the face as the face of the intercom person in the decoded image when the decoded image includes a human face; a fourth determination unit, which is used to obtain the face intercom information corresponding to the decoded image when the decoded image includes at least two human faces, and determine the face of the intercom person in the decoded image based on the face intercom information.

[0112] In an example embodiment, the face intercom information includes at least one of face position information, face posture information, face distance information, face lip information, and lip-sound matching confidence. The fourth determination unit may include: a first determination subunit, for each face in the decoded image, weighted summing each piece of information in the face intercom information corresponding to the face, to obtain the intercom confidence score of the face; a second determination subunit, for determining the highest intercom confidence face determined based on the intercom confidence score as the intercom face.

[0113] In an exemplary embodiment, the fourth determination unit further includes a third determination subunit and a fourth determination subunit. The third determination subunit is used to determine the target face as the face of a person traveling with the intercom person when there is a target face with a talk confidence score less than a first preset confidence threshold among other human faces other than the intercom person in the decoded image; the fourth determination subunit is used to determine the behavioral association confidence between the target face and the intercom person in the decoded image when there is a target face with a talk confidence score less than a first preset confidence threshold among other human faces other than the intercom person in the decoded image, and determine the face with a behavioral association confidence greater than a second preset confidence threshold among the target faces as the face of a person traveling with the intercom person corresponding to the intercom person.

[0114] In an exemplary embodiment, the fourth determination unit may include an acquisition subunit or an analysis subunit. The acquisition subunit is used to acquire face intercom information corresponding to a decoded image from decoded data of the intercom video stream, wherein the face intercom information is obtained by performing face analysis on the captured video image by the first video intercom device; and the analysis subunit is used to perform face analysis on the decoded image to obtain the face intercom information corresponding to the decoded image.

[0115] Figure 7 An example of a structural diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the video image display method provided by any of the above method embodiments.

[0116] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video image display method provided by any of the above method embodiments.

[0118] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video image display method provided by any of the above method embodiments is implemented.

[0119] By way of example, computer readable storage media include non-transitory computer readable storage media.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video image display method, characterized in that: include: Receiving an intercom video stream sent by a first video intercom device, and obtaining a decoded image of the intercom video stream; Determine the face of the intercom user in the decoded image, and obtain a target video image containing the face of the intercom user and having a screen ratio matching that of the second video intercom device from the decoded image; The target video image is displayed on the second video intercom device at a display resolution of the second video intercom device.

2. The video image display method according to claim 1, characterized in that: The step of obtaining a target video image containing the face of the intercom person and having a screen ratio matching that of the second video intercom device from the decoded image includes: In the case where the decoded image includes the face of the person who is traveling with the intercom user, with the goal of including the most faces of the person who is traveling with the intercom user, a video capture image including the face of the intercom user and having a screen ratio matching that of the second video intercom device is captured from the decoded image; In the case that the face of the companion is not included in the peripheral preset pixel range of the video captured image in the decoded image, the video captured image is determined as the target video image.

3. The video image display method according to claim 2, characterized in that: Also includes: In the case where the face of the companion is included in the peripheral preset pixel range of the video capture image in the decoded image, the video capture image is expanded outward to the peripheral preset pixel range to obtain the target video image.

4. The video image display method according to claim 1, characterized in that: The determining the face of the speaker in the decoded image comprises: In the case where the decoded image includes a human face, determining the human face as the face of the interlocutor in the decoded image; In the case that the decoded image includes at least two human faces, face intercom information corresponding to the decoded image is obtained, and the intercom face in the decoded image is determined based on the face intercom information.

5. The video image display method according to claim 4, characterized in that: The face intercom information includes at least one of face position information, face posture information, face distance information, face lip information, and lip-sound matching confidence; the determining the intercom person's face in the decoded image based on the face intercom information includes: For each face in the decoded image, weighted sum each piece of information in the face intercom information corresponding to the face to obtain an intercom confidence score of the face; The highest intercom confidence face determined based on the intercom confidence score is determined as the intercom person face.

6. The video image display method according to claim 5, characterized in that: Also includes: If there is a target face whose intercom confidence score is less than a first preset confidence threshold among other faces of the intercom person in the decoded image except the face of the intercom person, the target face is determined as the face of a fellow traveler corresponding to the face of the intercom person; Alternatively, when there is a target face whose intercom confidence score is less than the first preset confidence threshold among other faces other than the intercom face in the decoded image, the behavior association confidence between the target face in the decoded image and the intercom face is determined, and the face among the target faces whose behavior association confidence is greater than the second preset confidence threshold is determined as the face of the companion corresponding to the intercom face.

7. The video image display method according to claim 4, characterized in that: The obtaining of face intercom information corresponding to the decoded image includes: Acquire the face intercom information corresponding to the decoded image from the decoded data of the intercom video stream, wherein the face intercom information is obtained by performing face analysis on the collected video image by the first video intercom device; Alternatively, face analysis is performed on the decoded image to obtain the face intercom information corresponding to the decoded image.

8. A video image display device, characterized in that: include: A receiving module, used for receiving an intercom video stream sent by a first video intercom device; A decoding module, used to obtain a decoded image of the intercom video stream; A determination module, used for determining the face of the speaker in the decoded image; An acquisition module, used for acquiring a target video image containing the face of the intercom person and having a picture ratio matching that of the second video intercom device from the decoded image; A display module is used to display the target video image on the second video intercom device at a display resolution of the second video intercom device.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video image display method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video image display method according to any one of claims 1 to 7 is implemented.

Images