Close-up picture determination method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a close-up picture determination method and device, equipment and a storage medium, and the method comprises the steps: recognizing close-up boxes to which head objects in a current frame picture in video stream data belong, each close-up box corresponding to at least one main body object, and the main body object being a head object framed by the corresponding close-up box; selecting a close-up box as a current close-up box; when it is determined that other head objects which can be added into the current close-up box exist, the other head objects are determined as main body objects of the current close-up box, and close-up boxes constructed for the other head objects are deleted; selecting another close-up box to be updated as the current close-up box, and returning to execute the operation of determining that other head objects can be added into the current close-up box until all the close-up boxes are traversed; and displaying a close-up picture of each close-up frame in the video stream data. By adopting the method, the technical problem that when video communication participants are subjected to close-up composition, closer participants repeatedly appear in a plurality of close-up pictures can be solved.

Description

A close-up image determination method, device, equipment and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of video call technology, and in particular to a method, apparatus, device, and storage medium for determining a close-up image. Background Art

[0002] In the scenario where an electronic device is used to realize video communication (which can also be understood as a video call), in order to ensure that each participant in the video screen is clearly seen, the electronic device will perform a separate composition for each participant, and then integrate these compositions together for display. For example, Figure 1 is the first schematic diagram of the video screen in the video communication scenario in the related art. Referring to Figure 1, there are four participants in the current video screen. The electronic device has performed a separate composition for each of the four participants. The picture in each composition can refer to the picture selected by the close-up frame 11 of each participant in Figure 1. After that, the electronic device integrates and displays the pictures in the four close-up frames 11. Figure 2 is the first schematic diagram of the integrated picture in the video communication scenario in the related art. Referring to Figure 2, it displays the pictures in the four close-up frames 11 in Figure 1, so that the user of the electronic device can clearly see the participants in the video communication.

[0003] However, during a video communication, there are situations where two participants are very close to each other. In this case, when a separate frame is framed for one of the participants, the framed image will include the other participant who is very close to it, causing the other participant to appear in both its own framed image and the framed image of the closer participant. For example, FIG3 is a second schematic diagram of a video frame in a video communication scenario in the related art. Referring to FIG3, there are four participants in the current video frame. The electronic device has separately framed each of the four participants. The image in each frame can refer to the image in the close-up frame 12 of each participant in FIG3. At this time, based on FIG3, it can be seen that participant 13 is close to participant 14, so that participant 13 appears in participant 14's close-up frame 12. Then, when the electronic device integrates and displays the images in the four close-up frames 12, participant 13 will appear repeatedly. FIG4 is a second schematic diagram of an integrated image in a video communication scenario in the related art. Referring to FIG4, it shows the images in the four close-up frames 12 in FIG3. Based on FIG4, it can be seen that participant 13 appears in two close-up frames, which affects the user's viewing experience in the video communication and the user's communication efficiency in the video communication.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a close-up image determination method, apparatus, device, and storage medium to solve the technical problem in the related art of repeatedly appearing close participants in multiple close-up images when composing close-up images of participants in a video communication.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a close-up image, comprising:

[0007] Identifying a close-up frame to which each head object belongs in a current frame of the video stream data, wherein each close-up frame corresponds to at least one subject object, and the subject object is a head object that has been determined to be framed by the corresponding close-up frame;

[0008] Selecting one of the close-up frames as the current close-up frame;

[0009] When it is determined that there are other head objects that can be added to the current close-up frame, the other head objects are determined as the main objects corresponding to the current close-up frame, and the close-up frames constructed for the other head objects are deleted;

[0010] Selecting another close-up frame from the close-up frame to update it as the current close-up frame, and returning to perform the operation of determining that there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, until all current close-up frames are traversed;

[0011] The close-up images selected by each of the close-up frames in the video stream data are displayed.

[0012] The above-described method identifies the close-up frame to which each head object belongs in the current frame of the video stream data, wherein the close-up frame corresponds to at least one subject object, where the subject object is the head object that has been determined to be framed by the corresponding close-up frame. Then, a close-up frame is selected from the close-up frames as the current close-up frame. When it is determined that there are other head objects that can be added to the current close-up frame, the other head object is determined as the subject object corresponding to the current close-up frame, and the close-up frame to which the other head object belongs is deleted. Then, another close-up frame is selected from the close-up frames to be updated as the current close-up frame, and further determinations are made as to whether there are other head objects that can be added to the current close-up frame until all close-up frames are traversed and the close-up frames selected by each close-up frame in the video stream data are displayed. This technical means solves the technical problem in the related art of repeatedly appearing close-up participants in multiple close-up frames when framing close-up frames of participants in a video communication. When it is determined that a participant (i.e., a head object) can be added to other close-up frames, the close-up frame of the participant is deleted, so that the participant appears in only one close-up frame, thereby minimizing the participant from appearing in multiple close-up frames and improving the communication efficiency of users in video communication.

[0013] Based on the above embodiment, when it is determined that there are other head objects that can be added to the current close-up frame, the other head objects are determined as the main objects corresponding to the current close-up frame, and the close-up frame constructed for the other head objects is deleted, including:

[0014] Determining whether the main object in the current close-up frame has changed;

[0015] If the main object in the current close-up frame changes, the main object with the largest area in the current close-up frame is selected as the current main object, and the current close-up frame is deleted;

[0016] Rebuilding a close-up frame for the current subject object and using it as the current close-up frame;

[0017] Other head objects that can be added to the current close-up frame are found, the other head objects are determined as the main objects corresponding to the current close-up frame, and the close-up frames constructed for the other head objects are deleted.

[0018] As described above, by judging whether the main object in the current close-up frame has changed, and when it changes, re-finding the current main object and constructing the current close-up frame, thereafter, finding the head object that can be added to the current close-up frame, and adding the head object to the current close-up frame, it can be ensured that when the main object in the close-up frame changes, the applicable close-up frame is reconstructed for each main object, thereby ensuring the rationality of the layout of the main objects in the close-up picture.

[0019] Based on the above embodiment, the step of finding other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects includes:

[0020] Constructing a stable frame with the current subject object as the center;

[0021] Moving the stable frame, and during the movement, finding other head objects that can be added to the current close-up frame, wherein the other head objects that can be added to the current close-up frame intersect with the stable frame at a corresponding moment during the movement;

[0022] Determining the other head objects found as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects;

[0023] The main object with the largest area in the current close-up frame is selected and updated as the current main object, and the operation of constructing a stable frame with the current main object as the center is returned to execute until no other head objects that can be added to the current close-up frame are found.

[0024] As described above, by constructing a stable frame and moving the stable frame, the head object that can be added to the current close-up frame is searched, so that the head objects found are all located in the surrounding area of ​​the current main object, that is, the distance to the current main object is relatively close, which ensures the rationality of the head objects found. In addition, the main object with the largest area is updated to the current main object, which can also ensure that when searching for the head object, the surrounding area of ​​the main object with the largest area is used as a reference, further ensuring that the layout of the main objects in the close-up picture is more reasonable.

[0025] Based on the above embodiment, after determining whether the main object in the current close-up frame has changed, the following steps may be performed:

[0026] If the main object in the current close-up frame has not changed, determining whether there are other head objects in the current close-up frame;

[0027] If there are other head objects, the other head objects are determined as the main objects corresponding to the current close-up frame, and the close-up frames constructed for the other head objects are deleted, and another close-up frame is selected from the close-up frames to be updated as the current close-up frame;

[0028] If no other head object exists, an operation of selecting another close-up frame from the close-up frames and updating it as the current close-up frame is performed.

[0029] As described above, when the main object in the current close-up frame has not changed, further searching whether there are other head objects in the current close-up frame can avoid missing head objects that can be added to the current close-up frame.

[0030] Based on the above embodiment, if there are other head objects, determining the other head objects as the main object corresponding to the current close-up frame and deleting the close-up frame constructed for the other head objects includes:

[0031] If there are other head objects, the main object with the largest area in the current close-up frame is selected and updated as the current main object;

[0032] Constructing a stable frame with the current subject object as the center;

[0033] Moving the stable frame, and during the movement, finding other head objects that can be added to the current close-up frame, wherein the other head objects that can be added to the current close-up frame intersect with the stable frame at a corresponding moment during the movement;

[0034] Determining the other head objects found as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects;

[0035] The main object with the largest area in the current close-up frame is selected and updated as the current main object, and the operation of constructing a stable frame with the current main object as the center is returned to execute until no other head objects that can be added to the current close-up frame are found.

[0036] The above can avoid missing the head object that can be added to the current close-up frame, and can ensure the rationality of finding the head object.

[0037] On the basis of the above embodiment, during the movement of the stabilization frame, the position of the corresponding current subject object remains unchanged and always remains in the stabilization frame.

[0038] As mentioned above, no matter which direction the stabilization frame moves, it is guaranteed that the current subject object will not exceed the stabilization frame. In this case, the area covered by the stabilization frame during movement can be considered as the surrounding area of ​​the current subject object, thereby ensuring the accuracy of other head objects found based on the stabilization frame.

[0039] Based on the above embodiment, the size of the stabilization frame is determined by the display size of the corresponding current subject object in the current frame.

[0040] As mentioned above, as long as the display size of the current main object does not change, even if the aspect ratio of the close-up frame changes, the size of the corresponding stable frame will not change. Then, when the head object is relatively stable in the video stream data (that is, it does not move or moves very little), the head object found based on the stable frame is also relatively fixed, thereby reducing the problem of close-up picture jitter caused by changes in the aspect ratio of the close-up frame.

[0041] Based on the above embodiment, the step of moving the stabilization frame and finding other head objects that can be added to the current close-up frame during the movement includes:

[0042] Moving the stabilization frame, and during the movement, searching for other head objects that are located within the current close-up frame and intersect with the stabilization frame;

[0043] Determine a minimum rectangular area containing the other head objects found and the main object in the current close-up frame;

[0044] When the width and height of the minimum rectangular area are both smaller than the width and height of the stable frame, it is determined that the other head objects found can be added to the current close-up frame.

[0045] As described above, when searching for a head object based on a stable frame, the width and height of the minimum rectangular area containing the head object and each subject object in the current close-up frame are compared with the width and height of the stable frame to further determine whether the found head object can be added to the current close-up frame. This can ensure that the head object added to the current close-up frame is close to each subject object, further ensuring that the layout of the subject objects in the close-up picture is more reasonable.

[0046] Based on the above embodiment, the step of determining whether the main object in the current close-up frame has changed includes:

[0047] Determine whether the number of main objects in the current close-up frame has changed; or,

[0048] Determine whether the area change range of the main object with the largest area in the current close-up frame exceeds an area threshold; or

[0049] It is determined whether there is a main object in the current close-up frame that exceeds a stable frame created for the main object with the largest area.

[0050] The above can ensure that changes in the subject object can be effectively identified.

[0051] Based on the above embodiment, the displaying of the close-up picture selected by each close-up frame in the video stream data includes:

[0052] When there are multiple main objects corresponding to the close-up frame, a corresponding rectangular area is created for the close-up frame, where the rectangular area is the smallest rectangular area that includes all the main objects in the close-up frame. A virtual object is created with the center point of the rectangular area as the center. The area of ​​the virtual object is equal to the area of ​​the main object with the largest area in the close-up frame. The frame selection position of the close-up frame in the video stream data is updated based on the virtual object so that the updated close-up frame is centered on the virtual object.

[0053] When the close-up frame corresponds to only one main object, the selection position of the close-up frame in the video stream data is updated with the main object as the center;

[0054] The close-up images selected by each updated close-up frame in the video stream data are displayed.

[0055] As described above, when there are multiple main objects corresponding to the close-up frame, adjusting the selection position of the close-up frame by constructing a virtual object can ensure that the distribution of the main objects in the close-up picture is more reasonable, and no main object is highlighted as the center.

[0056] Based on the above embodiment, the step of identifying the close-up frame to which each head object belongs in the current frame of the video stream data includes:

[0057] Obtaining the most recent close-up frame recognition result obtained for the video stream data;

[0058] Determining the close-up frame to which each head object belongs in the current frame of the video stream data according to the close-up frame recognition result;

[0059] After traversing all current close-up frames, the method further includes:

[0060] The close-up frame recognition result is updated.

[0061] In a second aspect, an embodiment of the present application further provides a close-up image determination device, comprising:

[0062] a close-up frame identification unit, configured to identify a close-up frame to which each head object belongs in a current frame of the video stream data, wherein each close-up frame corresponds to at least one subject object, and the subject object is a head object that has been determined to be framed by the corresponding close-up frame;

[0063] a close-up frame selection unit, configured to select one of the close-up frames as a current close-up frame;

[0064] a first object determining unit configured to, when determining that there are other head objects that can be added to the current close-up frame, determine the other head objects as the main objects corresponding to the current close-up frame, and delete the close-up frame constructed for the other head objects;

[0065] a first close-up frame updating unit, configured to select another close-up frame from the close-up frames and update it as the current close-up frame, and return to executing an operation of determining, when determining that there are other head objects that can be added to the current close-up frame, the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, until all current close-up frames are traversed;

[0066] The close-up display unit is used to display the close-up picture selected by each close-up frame in the video stream data.

[0067] In a third aspect, an embodiment of the present application further provides a close-up image determination device, comprising:

[0068] one or more processors;

[0069] A display screen for displaying close-up images;

[0070] a memory for storing one or more programs;

[0071] When the one or more programs are executed by the one or more processors, the one or more processors implement the close-up picture determination method as described in the first aspect.

[0072] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the close-up image determination method as described in the first aspect.

[0073] The beneficial effects of the above-mentioned close-up picture determination device, equipment and storage medium can refer to the beneficial effects of the close-up picture determination method. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] FIG1 is a first schematic diagram of a video screen in a video communication scenario in the related art;

[0075] FIG2 is a first schematic diagram of an integrated screen in a video communication scenario in the related art;

[0076] FIG3 is a second schematic diagram of a video screen in a video communication scenario in the related art;

[0077] FIG4 is a second schematic diagram of an integrated screen in a video communication scenario in the related art;

[0078] FIG5 is a schematic structural diagram of a close-up image determination device provided by one embodiment of the present application;

[0079] FIG6 is a flowchart of a method for determining a close-up image according to an embodiment of the present application;

[0080] FIG7 is a schematic diagram of an arrangement of close-up images provided by an embodiment of the present application;

[0081] FIG8 is a schematic diagram of another arrangement of close-up images provided by an embodiment of the present application;

[0082] FIG9 is a flowchart of another method for determining a close-up image provided by an embodiment of the present application;

[0083] FIG10 is a schematic diagram of a close-up image and a stabilization frame provided by one embodiment of the present application;

[0084] FIG11 is a schematic diagram of another close-up image and a stabilization frame provided by an embodiment of the present application;

[0085] FIG12 is a close-up block diagram of an embodiment of the present application;

[0086] FIG13 is a schematic diagram of a virtual object provided by one embodiment of the present application;

[0087] FIG14 is another close-up block diagram provided by one embodiment of the present application;

[0088] FIG15 is a flowchart illustrating an example of a method for determining a close-up image according to an embodiment of the present application;

[0089] FIG16 is a schematic structural diagram of a close-up image determination device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0090] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended to explain the present application, not to limit it. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present application, not all structures.

[0091] It should be noted that due to space limitations, this application specification does not enumerate all optional implementation methods. After reading this application specification, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0092] Each embodiment is described in detail below.

[0093] Video communication can be understood as using the internet to facilitate video calls, allowing users in different locations to communicate face-to-face. Video communication is widely used in daily communication, meetings, and teaching scenarios.

[0094] Currently, to allow users to clearly view each participant in a video exchange, electronic devices conducting video exchanges individually frame each participant in the video to obtain a close-up of each participant. During this individual framing, a close-up frame is created for each participant. For example, referring to the close-up frames shown in Figures 1 and 3, a close-up frame can be understood as a rectangular area centered on the corresponding participant in the video. The image framed by the close-up frame in the video can be considered a close-up of the corresponding participant. When creating the close-up frames, each close-up frame has the same aspect ratio but can be of different sizes. The larger the area of ​​the corresponding participant in the video, the larger the close-up frame required to frame the participant, and the smaller the area of ​​the participant in the video, the smaller the close-up frame required to frame the participant. The electronic device then displays the close-up images in each close-up frame in a specific arrangement to allow the user of the electronic device to clearly view each participant.

[0095] However, in a video chat scenario, some participants are relatively close together. In this case, a close-up frame created for one participant will cover another participant who is also closer. Since the other participant also has a corresponding close-up frame, the other participant will appear in two close-up frames. For example, in Figure 4, participant 13 appears in both close-up frames. This affects the user's viewing experience and communication efficiency. Specifically, if a user is focusing on participant 13 during a video chat, and if participant 13 appears in both close-up frames, the user may not know which close-up frame to focus on when viewing the video, and may miss the content of the video conference due to hesitation. Alternatively, if the user first discovers participant 13 in the close-up frame in the upper right corner of Figure 4, the user will focus on participant 13 in the close-up frame for a period of time. However, since participant 13 does not occupy a large proportion of the close-up frame, the user will struggle to see participant 13, affecting the efficiency of the video conference. As can be seen, the repeated appearance of participants in multiple close-up frames will reduce the efficiency of the video conference, and the probability of this situation need to be reduced.

[0096] Based on this, a close-up picture determination method is provided in an embodiment of the present application. The close-up picture determination method can place multiple participants in video communication who are at close distance into the same close-up picture, so as to avoid the situation where participants appear repeatedly in multiple close-up pictures as much as possible and reduce the probability of such situation.

[0097] One embodiment of the present application provides a close-up image determination method that can be performed by a close-up image determination device. The close-up image determination device can be implemented in software and / or hardware and can be composed of two or more physical entities or a single physical entity. Currently, the close-up image determination device can be an electronic device capable of video communication, such as an interactive smart tablet, a tablet computer, or a laptop computer.

[0098] Figure 5 is a schematic diagram of the structure of a close-up image determination device provided in one embodiment of the present application. Referring to Figure 5 , the close-up image determination device includes a processor 21, a memory 22, and a display screen 23. The processor 21, the memory 22, and the display screen 23 may be connected via a bus or other means; Figure 5 illustrates a bus connection as an example.

[0099] There may be one or more processors 21, and one processor 21 is used as an example in Figure 5. The processor 11 may include processing units such as an application processor (AP), a graphics processing unit (GPU), and a central processing unit (CPU).

[0100] Memory 22, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the close-up image determination method in the embodiments of the present application. Memory 22 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the close-up image determination device. Furthermore, memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, memory 22 may further include memory remotely located relative to processor 21, and such remote memory may be connected to the close-up image determination device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. It is understood that when processor 21 has storage functionality, memory 22 and processor 21 may also be integrated into a single physical entity.

[0101] The number of display screens 23 (also referred to as screens or displays) is one or more, and FIG5 takes one display screen 23 as an example. The display screen 23 may be a liquid crystal display (Liquid Crystal Display), an LED display (LED display), an organic light-emitting diode (OLED) display, or a flexible light-emitting diode (FLED) display, etc., and the embodiment does not limit this. In addition, the display screen may be a straight screen or a curved screen, and the embodiment of the present application does not limit this. Based on the display screen 23, a close-up picture can be realized to determine the display function of the device. In one embodiment, the display screen 23 may also be integrated with a touch function, in which case the display screen 23 includes a display panel and a touch panel. The display panel is used to complete the visual output. The touch panel may be a touch component that supports infrared touch, electromagnetic touch, capacitive touch, or resistive touch. The touch panel may transmit the detected touch operation to the processor 21 for subsequent processing, and the display panel may provide visual output related to the touch operation.

[0102] In addition, the close-up image determination device may also include one or more communication interfaces (not shown) through which it can communicate with other electronic devices, for example, to transmit data required for video communication with other electronic devices. The type of communication interface is currently not limited.

[0103] The close-up image determination device may also include a power supply, a speaker, a physical button, and other devices, which are not limited in the embodiment.

[0104] Based on the above hardware structure, the close-up picture determination device supports at least one type of operating system, which can be an Android system, a Windows system, a Linux system, or other operating systems.

[0105] The close-up image determination device can have at least one application installed in its operating system. This application can be a native application of the operating system, an application downloaded from a backend server, or a third-party device. By running each application, the close-up image determination device can implement corresponding functions. Currently, the close-up image determination device has at least one application installed for implementing video communication and an application for implementing the current close-up image determination method. In actual applications, these two applications can also be integrated into a single application.

[0106] FIG6 is a flow chart of a close-up picture determination method provided by an embodiment of the present application. Referring to FIG6 , when a close-up picture determination device executes the close-up picture determination method, the method specifically includes steps 310 to 350:

[0107] Step 310: Identify the close-up frame to which each head object belongs in the current frame of the video stream data. Each close-up frame corresponds to at least one main object, and the main object is the head object that has been determined to be framed by the corresponding close-up frame.

[0108] Video stream data can be understood as the video data currently used for video communication. The video stream data can be data obtained by a close-up image determination device after capturing the images of local participants, or it can be data received by a close-up image determination device that is remotely transmitted and contains images of remote participants. Participants refer to people participating in the video communication. It can be understood that during the video communication, the close-up image determination device receives the video stream data in real time and obtains each frame of video images for realizing the video communication based on the video stream data. The video images contain images of the participants in the video communication. Generally speaking, the heads and faces of the participants appear in the video image. In this case, the head of each participant in the video image is regarded as a head object. In this case, a head object can represent a participant. Currently, each head object can be obtained by detecting and identifying the head or facial area of ​​each person in the video image. The detection and identification means can adopt existing image recognition means, which are not described separately at this time.

[0109] The current frame refers to a frame of video in the currently received video stream during a video exchange. A close-up frame refers to a rectangular frame generated based on the video frame, containing at least one head object within the area it encloses. Currently, a close-up frame is generated based on coordinates, for example, based on four coordinates (the coordinates of a head or face identified by head detection or face detection). A close-up frame can be visible or invisible. In this case, the area enclosed by the close-up frame in the video frame can be recorded as the close-up image selected by the close-up frame in the video stream data. The one or more head objects contained in the close-up frame can be considered the subject objects corresponding to the close-up frame. That is, each close-up frame corresponds to at least one subject object, and the subject object can be considered the head object belonging to the corresponding close-up frame. The close-up image corresponding to the close-up frame can be considered a close-up image of the subject object. In this case, determining whether a participant is enclosed by the close-up frame primarily involves determining whether the participant's head area (i.e., the head object) is enclosed by the close-up frame.

[0110] It is understood that when the number of close-up frames changes, the corresponding aspect ratio of the close-up frames also changes to ensure that when the close-up images in the close-up frames are displayed, each close-up image can be adaptively arranged to ensure the viewing effect. For example, Figure 7 is a schematic diagram of the arrangement of a close-up image provided by an embodiment of the present application, and Figure 8 is a schematic diagram of the arrangement of another close-up image provided by an embodiment of the present application. Figure 7 shows a schematic diagram of the arrangement of a close-up image when displaying a close-up image with two close-up frames, with the two close-up images arranged left to right in a 1:1 ratio. Figure 8 shows a schematic diagram of the arrangement of a close-up image when displaying a close-up image with three close-up frames, with the three close-up images arranged left, center, and right in a 1:1:1 ratio. When displaying a close-up image, the aspect ratio of the close-up image is the same as the aspect ratio of the close-up frames and is related to the number of close-up frames. For example, the aspect ratio required when there are two close-up frames is different from the aspect ratio required when there are three close-up frames. Referring to Figure 7, when there are two close-up frames, the aspect ratio of the close-up frames is 8:9, and referring to Figure 8, when there are three close-up frames, the aspect ratio of the close-up frames is 16:27. Currently, when a close-up picture determination device creates or updates a close-up frame, it can determine the aspect ratio of each close-up frame based on the number of close-up frames, and then determine the size of the close-up frame in the video picture in combination with the aspect ratio and the display size of the corresponding main object in the video picture. When there are multiple main objects in the close-up frame, the size of the close-up frame is determined based on the main object with the largest display size. Optionally, the size relationship between the close-up frame and the main object is: the height of the close-up frame = p * the height (or width) of the main object in the video picture, the width of the close-up frame = q * the height (or width) of the main object in the video picture, where p and q are coefficients, and their values ​​are related to the current aspect ratio of the close-up frame. As for whether to use the width or height of the main object in the video picture to calculate the close-up frame, it can be pre-set by the close-up picture determination device.

[0111] When the close-up image determination device receives the current frame, it identifies each head object therein and determines the close-up frame to which the head object belongs. In this case, each head object has a close-up frame to which it belongs. In one embodiment, the close-up image determination device determines the close-up frame to which each head object in the current frame belongs based on the close-up frame recognition results of the previously received video stream data. The close-up frame recognition results describe the close-up frames created (i.e., existing) in the previous video frame and the subject objects corresponding to each close-up frame. When the close-up image determination device receives the video stream data, it can track the subject objects and close-up frames corresponding to each close-up frame in each frame of the video stream data based on the previously obtained close-up frame recognition results. When the video frame changes (from the previous frame to the next frame), the close-up frame to which each head object in the current frame belongs is determined based on the tracking results. It is understandable that for the first frame of the video stream data, the previous close-up frame recognition results should be empty. In this case, the close-up image determination device can identify each head object and create a close-up frame for each head object to obtain the close-up frame to which each head object belongs. For the video screen of the current frame in the video stream data, if one or more head objects appear newly, then there is no close-up frame of the newly appearing head object in the close-up frame recognition result. At this time, a close-up frame can be created for each newly appearing head object to ensure that each head object in the video screen has its own close-up frame.

[0112] Optionally, each time the close-up picture determination device receives a frame of video, it identifies the close-up frame to which each head object belongs, and executes subsequent steps (i.e., steps 320-350), i.e., updates the close-up frame recognition result frame by frame. Alternatively, each time the close-up picture determination device receives a frame of video, it identifies the close-up frame to which each head object belongs, and displays a close-up of the corresponding main object according to the close-up picture of the close-up frame, and, at intervals of a set number of frames or a set duration, executes subsequent steps (i.e., steps 320-350) after identifying the close-up frame to which each head object belongs in the current frame, so as to update the close-up frame recognition result at intervals.

[0113] Step 320: Select a close-up frame from the close-up frames as the current close-up frame.

[0114] There are usually more than one close-up frames, and one close-up frame is selected from the more than one close-up frames as the current close-up frame.

[0115] Exemplarily, the current close-up frame refers to a close-up frame currently being updated for the subject object. The close-up image determination device may randomly select a close-up frame from the close-up frames of the current frame as the current close-up frame, or select a close-up frame in a set order (e.g., from top to bottom, from left to right) as the current close-up frame, or select a close-up frame in the order in which the close-up frames are created, where each close-up frame is created based on the order in which each face or head is recognized in head recognition or face recognition.

[0116] Step 330: When it is determined that there are other head objects that can be added to the current close-up frame, the other head objects are determined as the main objects corresponding to the current close-up frame, and the close-up frame constructed for the other head objects is deleted.

[0117] The existence of other head objects that can be added to the current close-up frame can be understood as the other head objects being included in the close-up picture of the current close-up frame. However, the close-up frame to which the other head objects belong is not the current close-up frame, that is, the other head objects are currently serving as the main objects of other close-up frames.

[0118] Optionally, the method for determining other head objects that can be added to the current close-up frame is not currently limited.

[0119] For example, when detecting that another head object appears in the close-up image of the current close-up frame, it is determined that the other head object can be added to the current close-up frame. It can be understood that when identifying the head object and creating the close-up frame, the coordinates of the head object and the coordinates of the close-up frame can be known. At this time, based on the coordinates of the two, it can be determined whether the head object appears in the close-up frame and in which close-up frame it appears.

[0120] For another example, a stabilization frame is created based on the subject object already in the current close-up frame. The stabilization frame refers to a rectangular area centered on the corresponding subject object. A stabilization frame differs from a close-up frame. A close-up frame is used to capture a close-up of the subject object, and its aspect ratio varies with the number of close-up frames. In contrast, a stabilization frame is used to determine whether there are other head objects surrounding the corresponding subject object, i.e., whether there are head objects that are relatively close. Similar to the close-up frame, a stabilization frame is also generated based on coordinates. For example, a rectangular stabilization frame is generated based on four coordinates (the coordinates of a head or face detected by head or face recognition). The stabilization frame can be visible or invisible. The size of the stabilization frame is determined based on the display size of the corresponding subject object in the video stream data. In one embodiment, the height of the stabilization frame = n * the height (or width) of the subject object in the video frame, and the width of the stabilization frame = m * the height (or width) of the subject object in the video frame. Here, n and m are coefficients with fixed values. Whether to use the width or height of the subject object in the video frame to calculate the stabilization frame can be pre-set by the close-up frame determination device. Currently, by setting reasonable coefficients for n and m, the stable frame can be made to cover as much area as possible in close-up frames of various aspect ratios, thereby ensuring the rationality and accuracy of finding head objects that are relatively close. After creating the stable frame, other head objects that can be added to the current close-up frame can be determined based on the stable frame. Specifically, if there is only one main object in the current close-up frame, a corresponding stable frame can be created based on this main object. If there are multiple main objects in the current close-up frame, a corresponding stable frame can be created based on the main object with the largest area in the video image. When determining other head objects that can be added to the current close-up frame based on the stable frame, the head objects other than the main object with the largest area covered by the stable frame can be used as head objects that can be added to the current close-up frame, or the stable frame can be moved with the main object with the largest area corresponding to the stable frame as the center. During the movement of the stable frame, other head objects covered by the stable frame can be found as head objects that can be added to the current close-up frame. Currently, other head objects covered by the stable frame are generally located within the current close-up frame.

[0121] In one embodiment, when a change is detected in the main object in the current close-up frame, such as a change in the number of main objects, a large movement of the main object resulting in a large change in the display area, or a main object moving out of a specified stable frame (such as a stable frame created based on the main object with the largest area), it indicates that the current close-up frame may no longer be applicable to the main object inside. At this time, a close-up frame can be rebuilt for each main object. During reconstruction, the current close-up frame is deleted, and a close-up frame is rebuilt for the main object with the largest area and used as the current close-up frame. Afterwards, other head objects that can be added to the current close-up frame are determined until there are no other head objects that can be added. At this time, if there are still head objects without close-up frames, a close-up frame can be created for the head object, and when the close-up frame is updated to the current close-up frame, other head objects that can be added to the close-up frame are searched. Correspondingly, when it is detected that the main object in the current close-up frame has not changed, it means that the current close-up frame is still applicable to the internal main object. At this time, it is only necessary to determine whether there are other head objects in the current close-up frame (the close-up frame to which the head object belongs is not the current close-up frame). If so, it is determined that there are other head objects that can be added to the current close-up frame. Otherwise, there is no need to update the current close-up frame and the corresponding main object.

[0122] Exemplarily, when it is determined that other head objects can be added to the current close-up frame, the other head objects are used as the main objects of the current close-up frame. At this time, the number of main objects corresponding to the current close-up frame is increased by 1. After the other head objects are used as the main objects of the current close-up frame, in order to prevent the other head objects from appearing in other close-up frames, in an embodiment, the close-up frames constructed for the other head objects are deleted. Optionally, after deleting the close-up frame, if there are other main objects in the close-up frame, close-up frames can be recreated for the other main objects. At this time, if the number of close-up frames changes, the aspect ratio of the close-up frame will also change.

[0123] Optionally, if a head object is first added to the current close-up frame as the corresponding main object, when other close-up frames are subsequently processed, if it is determined that the head object can also be added to other close-up frames that are subsequently processed, the head object will not be processed, that is, the close-up frame that was first determined to be able to be added will prevail.

[0124] Optionally, each time another head object is added to the current close-up frame, another head object that can be added to the current close-up frame is searched for until no other head object is found. At this point, the current close-up frame and the corresponding main body object are considered to be updated. Thereafter, step 340 is executed.

[0125] Step 340: Select another close-up frame from the close-up frame and update it as the current close-up frame. Return to the execution to determine that when there are other head objects that can be added to the current close-up frame, determine the other head objects as the main objects corresponding to the current close-up frame, and delete the close-up frames constructed for the other head objects until all current close-up frames are traversed.

[0126] Exemplarily, another close-up frame that has not been updated is selected as the current close-up frame, and the process returns to step 330 to update the current close-up frame and its corresponding subject object again.

[0127] It is understood that each time a current close-up frame is updated, a check is performed to determine whether there are any close-up frames that have not been set as the current close-up frame. If so, a close-up frame is selected from the close-up frames that have not been set as the current close-up frame and updated as the current close-up frame, and the process returns to step 330. If not, it is assumed that all close-up frames and their corresponding subject objects have been updated, and step 350 can then be executed.

[0128] Optionally, after traversing all close-up frames, the currently determined close-up frame and the subject object corresponding to the close-up frame may be saved, that is, the close-up frame recognition result is updated for subsequent use.

[0129] Step 350: Display the close-up images selected by each close-up frame in the video stream data.

[0130] For example, a close-up frame selected in the current frame of the video stream data is obtained, and each close-up frame has a corresponding close-up frame. Afterwards, each close-up frame is displayed on the display screen according to a corresponding arrangement (related to the number of close-up frames).

[0131] In one embodiment, when the main object in the close-up frame is updated, in order to ensure that the layout of each main object in the close-up picture is as reasonable as possible and reduce the composition differences between different close-up pictures, when displaying the close-up picture of each close-up frame, the selection area of ​​the close-up frame is adjusted based on the main object in the close-up frame. During adjustment, the smallest rectangular area encompassing all the main objects in the close-up frame is first determined. A virtual object is then created, centered around the center point of the smallest rectangular area. This virtual object is a virtual head area, with the area of ​​the virtual object equal to the area of ​​the largest main object in the close-up frame. Alternatively, a head object with the same area as the largest main object can be created as a virtual object, with the center of the smallest rectangular area being the center of the virtual object. The position of the close-up frame is then determined based on the position of the virtual object. The size and / or proportion of the close-up frame are adjusted according to the desired aspect ratio (the desired aspect ratio is pre-set, e.g., for two close-up frames, the aspect ratio is 8:9, and for three close-up frames, the aspect ratio is 16:27) and the size (determined based on the size of the virtual object, e.g., the height of the close-up frame is determined by multiplying the height of the virtual object by a preset coefficient). Each close-up frame then frames the close-up image based on the position and size of the virtual object, thereby minimizing compositional differences between different close-ups. When the close-up images are displayed, the areas of the largest main objects in each close-up image are less different. It is understandable that if there is only one main object in the close-up frame, there is no need to create a virtual object. Instead, the close-up frame position in the video stream data can be updated with the main object as the center. After that, the close-up image in each close-up frame can be displayed.

[0132] The above-described technical means solves the technical problem in the related art of repeatedly appearing in multiple close-up frames of participants in video communication when framing close-up frames of participants in video communication, thereby improving the communication efficiency of users in video communication. When it is determined that a participant (i.e., a head object) can be added to another close-up frame, the close-up frame of the participant is deleted, so that the participant appears in only one close-up frame, thereby minimizing the possibility of the participant appearing in multiple close-up frames.

[0133] FIG9 is a flowchart of another close-up image determination method provided by one embodiment of the present application. This embodiment, based on the previous embodiment, exemplifies the process of determining other head objects that can be added to the current close-up frame, determining the close-up frame to which each head object belongs within the current frame, and displaying the close-up image of each close-up frame in the video stream data. Referring to FIG9 , the close-up image determination method includes steps 410 to 4130:

[0134] Step 410: Obtain the most recent close-up frame recognition result obtained for the video stream data.

[0135] The close-up frame recognition result describes each close-up frame created in the video image and the subject object corresponding to each close-up frame. Each time the close-up frame recognition result is updated, the current updated close-up frame recognition result is saved as the most recent result.

[0136] In one embodiment, taking the example of updating the close-up frame recognition result at intervals of a set duration or a set number of frames, after the close-up frame recognition result is updated, a timer or frame count is started. When the current frame of the video stream data is obtained, it is determined whether the corresponding timer has reached the set duration or the corresponding frame count has reached the set number of frames. If so, it is determined that the close-up frame recognition result needs to be updated. In this case, the most recent close-up frame recognition result, i.e., the most recently updated close-up frame recognition result, is obtained, and step 420 is executed. If not, each close-up frame and each subject object in each frame of the video stream data is tracked according to the most recent close-up frame recognition result, and each close-up frame selected by the close-up frame is displayed.

[0137] Step 420: Determine the close-up frame to which each head object in the current frame of the video stream data belongs based on the close-up frame recognition result.

[0138] For example, after identifying each head object in the current frame, the close-up frame identification results can be used to determine which close-up frame the head object belongs to. Optionally, the position of the same head object in consecutive frames should be the same or very close. Therefore, based on the position of the head object and the position of the main object, the main object can be determined. Subsequently, based on the close-up frame identification results, the close-up frame of each head object in the current frame can be determined.

[0139] It can be understood that for the first frame in the video stream data, the most recently obtained close-up frame recognition result should be empty. In this case, a close-up frame can be created for each head object in the picture.

[0140] Step 430: Select a close-up frame from the close-up frames as the current close-up frame.

[0141] Step 440: Determine whether the main object in the current close-up frame has changed. If the main object in the current close-up frame has changed, execute step 450. If the main object in the current close-up frame has not changed, execute step 480.

[0142] For example, during the process of tracking and displaying the close-up image of the video stream data based on the close-up frame and the corresponding subject object recorded in the close-up frame recognition results, when it is necessary to update the close-up frame recognition results, it can be determined whether the subject object within the current close-up frame in the current frame has changed. If a change has occurred, it indicates that the current close-up frame may no longer be applicable to its corresponding subject object. In this case, step 450 is executed. If no change has occurred, it indicates that the current close-up frame is still applicable to its corresponding subject object. In this case, step 480 is executed.

[0143] In one embodiment, determining whether the main object in the current close-up frame has changed can be: determining whether the number of main objects in the current close-up frame has changed, or determining whether the area change range of the main object with the largest area in the current close-up frame exceeds the area threshold, or determining whether there is a main object in the current close-up frame that exceeds the stable frame created for the main object with the largest area.

[0144] When determining whether the number of subject objects in the current close-up frame has changed, the primary focus is on determining whether the number of subject objects has decreased. That is, whether any subject objects have left the current close-up frame. If the number has changed, this indicates that a subject object has left the current close-up frame. Therefore, a suitable close-up frame needs to be rebuilt for each subject object. In one embodiment, when tracking the current close-up frame and its corresponding subject object based on the close-up frame recognition results, the position of the subject object in the video stream data and the position of the current close-up frame in the video stream data can be used to determine whether the subject object has left the current close-up frame, thereby determining whether the number of subject objects in the current close-up frame has changed. If the number has changed, it is determined that the subject object has changed, and step 450 is executed. Otherwise, step 480 is executed.

[0145] When determining whether the area change range of the subject object with the largest area in the current close-up frame exceeds an area threshold, the area change range can be understood as the range of changes in the display area of ​​the subject object in the close-up frame, and the area threshold is a preset value. If the area change range of the subject object with the largest area in the current close-up frame exceeds the area threshold, it indicates that the subject object with the largest area may have changed to another subject object (i.e., the display area of ​​the original subject object with the largest area has decreased), or it may indicate that the area of ​​the subject object with the largest area has increased. The size of the close-up frame is related to the size of the subject object with the largest area. When the size of the subject object with the largest area changes, the size of the close-up frame also needs to be adjusted accordingly. Therefore, when the size is adjusted, the subject object applicable to the close-up frame may also change. Therefore, it is necessary to re-update the close-up frame based on the subject object with the largest area and re-determine the applicable subject object. In one embodiment, when tracking each subject object in the current close-up frame, the display area of ​​each subject object can be detected (derived from the width and height of the subject object). In this case, the area change range can be determined based on the display area of ​​the subject object with the largest area in the current close-up frame in the current frame and the display area of ​​the subject object when the close-up frame recognition result was previously determined. When the area change range exceeds the area threshold, it is determined that the main object has changed, and step 450 is executed; otherwise, step 480 is executed.

[0146] When determining whether a subject object in the current close-up frame extends beyond the stabilization frame created for the subject object with the largest area, a stabilization frame is created based on the subject object with the largest area in the current close-up frame. While tracking each subject object in the video stream data and its corresponding close-up frame, this stabilization frame is used to identify whether other subjects closer to the subject object have moved further away. If other subjects extend beyond the stabilization frame, this indicates that the distance between the other subjects and the subject object with the largest area may have increased, and a single close-up frame may no longer be able to simultaneously frame both subjects. In other words, the other subjects and the subject object with the largest area may need to belong to different close-up frames. Therefore, when tracking each subject object in the video stream data, a stabilization frame may be created for the subject object with the largest area in the current close-up frame, and a determination is made as to whether other subjects in the current close-up frame extend beyond this stabilization frame. If a subject object in the current close-up frame extends beyond the stabilization frame created for the subject object with the largest area, a change in the subject object is determined, and step 450 is executed. Otherwise, step 480 is executed.

[0147] In practical applications, if any of the above three solutions is met, it can be considered that the main object in the current close-up frame has changed.

[0148] Step 450: Select the main object with the largest area in the current close-up frame as the current main object, and delete the current close-up frame.

[0149] Exemplarily, based on the close-up picture selected by the current close-up frame in the current frame, the display area of ​​each subject object corresponding to the current close-up frame in the current frame can be determined. Afterwards, the subject object with the largest area is selected as the current subject object to reconstruct the applicable close-up frame based on the current subject object. In addition, the existing current close-up frame can be deleted. At this time, the subject objects corresponding to the current close-up frame will no longer be the subject objects of the current close-up frame. Optionally, in the deleted current close-up frame, in addition to the subject object with the largest area, if there are other subject objects, close-up frames can be created for the other subject objects, but they will not be used as the current close-up frame to ensure that each head object in the current frame has its own close-up frame.

[0150] It can be understood that the larger the area of ​​the main object in the current frame is, the larger the area of ​​the main object in the close-up picture will be when the close-up picture is displayed.

[0151] Step 460: Reconstruct a close-up frame for the current subject object and use it as the current close-up frame.

[0152] Exemplarily, a close-up frame is reconstructed for the current main object and used as the current close-up frame. When constructing a close-up frame, the aspect ratio of the close-up frame can be determined based on the number of close-up frames, and then the size of the close-up frame can be determined based on the aspect ratio of the close-up frame and the height (or width) of the current main object when displayed in the current frame. After that, a close-up frame is created according to the aforementioned size with the current main object as the center. It can be understood that when the number of close-up frames changes, the aspect ratio of the close-up frame also changes adaptively. For example, if the main object of a close-up frame is added to other close-up frames, the close-up frame is deleted, which in turn causes the number of close-up frames to change. At this time, the aspect ratio of the close-up frame is determined based on the latest number of close-up frames, and then the size of each current close-up frame is modified in combination with the aspect ratio of the close-up frame.

[0153] After the creation is completed, the close-up frame is used as the current close-up frame. At this time, the current close-up frame contains a subject object, that is, the current subject object is used as the subject object of the current close-up frame.

[0154] Step 470: Find other head objects that can be added to the current close-up frame, determine the other head objects as the main objects corresponding to the current close-up frame, and delete the close-up frames constructed for the other head objects. Execute step 4100.

[0155] For example, other head objects that can be added to the current close-up frame are searched. The search may be performed by identifying other head objects within the current close-up frame as head objects that can be added to the current close-up frame. Alternatively, the search may be performed by constructing a stable frame to determine head objects that are closer to the current subject, thereby obtaining head objects that can be added to the current close-up frame.

[0156] In one embodiment, the method of constructing a stable frame to search for a head object that can be added to the current close-up frame is used as an example for description. In this case, step 470 includes steps 471 to 474:

[0157] Step 471: Construct a stable frame with the current subject object as the center.

[0158] In one embodiment, the size of the stabilization frame is determined by the display size of the corresponding current subject object in the current frame. Here, the height of the stabilization frame = n * the height (or width) of the subject object in the video frame, and the width of the stabilization frame = m * the height (or width) of the subject object in the video frame. n and m are fixed coefficients. Whether to use the width or height of the subject object in the video frame to calculate the stabilization frame can be pre-set by the close-up image determination device.

[0159] For example, a pre-set coefficient for determining the size of the stabilization frame is obtained. Based on the width (or height) of the current subject in the current frame and the preset coefficient, the size of the stabilization frame can be determined. Then, a stabilization frame of the corresponding size is constructed with the current subject as the center. This stabilization frame serves as the stabilization frame of the current subject. The area where the stabilization frame is located can be considered to be the area closest to the current subject. In this case, as long as the display size of the current subject does not change, the size of the stabilization frame will not change regardless of whether the aspect ratio of the close-up frame changes.

[0160] Generally speaking, the stabilized frame is located inside the close-up frame. In rare cases, a small part of the stabilized frame exceeds the close-up frame.

[0161] Step 472: Move the stable frame, and during the movement, find other head objects that can be added to the current close-up frame. The other head objects that can be added to the current close-up frame intersect with the stable frame at the corresponding moment during the movement.

[0162] After constructing the stable frame, the stable frame is moved. In one embodiment, during the movement of the stable frame, the corresponding current subject object remains unchanged and always within the stable frame. That is, regardless of the direction of the stable frame movement, the current subject object is ensured not to exceed the stable frame, so that the stable frame always moves around the current subject object. In this case, the area covered by the stable frame during movement can be considered as the surrounding area of ​​the current subject object, thereby ensuring the accuracy of other head objects found based on the stable frame. Specifically, if another head object is covered during the movement of the stable frame, then the other head object can be considered to be closer to the current subject object and can be added to the current stable frame corresponding to the current subject object. The other head objects that can be covered by the stable frame refer to those whose display area in the current frame is completely within the stable frame. Currently, other head objects covered by the stable frame are considered to be head objects that can be added to the current close-up frame. Generally speaking, the head object is within the current close-up frame and, at a certain moment (i.e., the corresponding moment) during the movement of the stable frame, the head object intersects with the stable frame.

[0163] When moving the stabilization frame, the moving direction of the stabilization frame can be pre-set. You only need to ensure that the stabilization frame covers the surrounding area of ​​the current subject during movement.

[0164] In one embodiment, the stabilization frame may cover multiple head objects during movement. In this case, only one of the head objects is selected as the currently found head object to avoid finding too many head objects, which would cause the area formed by these multiple head objects to eventually exceed the area of ​​the stabilization frame. Optionally, among the multiple head objects covered, the head object closest to the current main object is selected as the currently found head object.

[0165] In one embodiment, during the movement of the stable frame, it may intersect with multiple head objects. However, not every head object can be added to the current close-up frame. For example, during the movement of the stable frame, it may intersect with a head object, but the head object is far away from another subject object in the current close-up frame and is therefore not suitable for addition to the current close-up frame. Accordingly, this step may further include steps 4721-4723:

[0166] Step 4721: Move the stabilization frame, and during the movement, search for other head objects that intersect with the stabilization frame within the current close-up frame.

[0167] For example, based on the coordinates of each head object and the coordinates of the current close-up frame, other head objects within the current close-up frame can be identified. The stable frame can then be used to search for these other head objects within the current close-up frame. During the movement of the stable frame, if there is an intersection with another head object within the current close-up frame, it can be considered that the other head objects found may be added to the current close-up frame.

[0168] Optionally, if there are multiple head objects that intersect during the movement process, the head object closest to the main object corresponding to the stabilization frame can be selected, or the head object found first can be selected.

[0169] Step 4722: Determine the minimum rectangular area that includes the other head objects found and the main body object in the current close-up frame.

[0170] After the stable frame finds other head objects, it can further determine whether the other head objects can be added to the current close-up frame. When determining, first determine the minimum rectangular area that includes the other head objects found and the main object in the current close-up frame.

[0171] Specifically, after the stabilization frame finds other head objects, it determines a rectangular area that encompasses the other head objects and each subject object corresponding to the current close-up frame. This rectangular area is the minimum rectangular area that contains the aforementioned objects. The minimum rectangular area can be understood as the smallest area required to contain the aforementioned objects.

[0172] Step 4723: When the width and height of the minimum rectangular area are both smaller than the width and height of the stable frame, it is determined that the other head objects found can be added to the current close-up frame.

[0173] After obtaining the minimum rectangular area, it is determined whether the width of the minimum rectangular area is less than the width of the stabilization frame, and whether the height of the minimum rectangular area is less than the height of the stabilization frame. If both are less than, it indicates that the stabilization frame can completely cover the minimum rectangular area. In other words, the other head objects found by the stabilization frame can be covered by the stabilization frame. In other words, the distance between the other head objects found and each main object corresponding to the current close-up frame is relatively close, and a single close-up frame can frame all the main objects and the found head objects. Therefore, it can be determined that the other head objects found can be added to the current close-up frame, and then step 473 is executed. If the width of the minimum rectangular area is greater than the width of the current stabilization frame, or the height of the minimum rectangular area is greater than the height of the current stabilization frame, it indicates that the distance between the other head objects found and one or more main objects corresponding to the current close-up frame is relatively far, and a single close-up frame may not be able to frame all the main objects and the found head objects. Therefore, the currently found head object is discarded. In this case, it is determined that no head objects that can be added to the current close-up frame can be found, and step 4100 can be executed.

[0174] It should be noted that as long as the display size of the current subject object does not change, even if the aspect ratio of the close-up frame changes, the size of the corresponding stable frame will not change. Therefore, when the head object is relatively stable in the video stream data (i.e., it does not move or moves very little), the head object found based on the stable frame is also relatively fixed, thereby reducing the problem of close-up image jitter caused by changes in the aspect ratio of the close-up frame. For example, Figure 10 is a schematic diagram of a close-up image and a stable frame provided by an embodiment of the present application, and Figure 11 is a schematic diagram of another close-up image and a stable frame provided by an embodiment of the present application. Currently, the two close-up images shown in Figure 10 are changed to three close-up images shown in Figure 11. At this time, the aspect ratio of the close-up image has changed significantly, but the size of the stable frame 41 has not changed. Therefore, the subject object corresponding to the close-up frame finally found based on the stable frame 41 will not change significantly, that is, the subject object in the close-up image will not change significantly, thereby effectively avoiding close-up image jitter caused by changes in the aspect ratio of the close-up frame.

[0175] Step 473: Determine the other head object found as the main object corresponding to the current close-up frame, and delete the close-up frame constructed for the other head object.

[0176] When another head object is found that can be added to the current close-up frame, the other head object is used as the main object of the current close-up frame. At this time, the number of main objects corresponding to the current close-up frame is increased by 1. After the other head object is used as the main object of the current close-up frame, in order to prevent the other head object from appearing in other close-up frames, in this embodiment, the close-up frame constructed for the other head object is deleted. Optionally, after deleting the close-up frame, if there are other main objects in the close-up frame besides the other head object, a new close-up frame can be created for the other main objects.

[0177] Step 474: Select the main object with the largest area in the current close-up frame and update it as the current main object, and return to execute the operation of constructing a stable frame with the current main object as the center until no other head objects that can be added to the current close-up frame are found.

[0178] Exemplarily, when the found head object is added to the current close-up frame, the main object in the current close-up frame will be updated. At this time, the main object with the largest area may change. Based on the principle of finding the head object around the main object with the largest area, when the main object in the current close-up frame is updated, the main object with the largest area is reselected to be updated as the current main object, and the process returns to step 471, i.e., a stable frame is re-created for the current main object to find other head objects that are closer to the current main object and add them to the current close-up frame. After the stable frame is created, no other head objects that can be added to the current close-up frame are found. At this time, if it is determined that no other head objects can be found through the stable frame, another close-up frame can be selected as the current close-up frame, and it can be re-determined whether other head objects can be added to the current close-up frame, i.e., step 4100 can be executed.

[0179] Step 480: Determine whether there are other head objects in the current close-up frame. If there are other head objects, execute step 490; if not, execute step 4100.

[0180] For example, if the main object in the current close-up frame has not changed, it can be further determined whether there are other head objects in the current close-up frame, that is, whether there is a head object that is not a main object entering the selection area corresponding to the current close-up frame.

[0181] When there are other head objects in the current close-up frame, it can be considered that the display areas of the other head objects in the current frame are completely located in the current close-up frame.

[0182] In one embodiment, if there are other head objects in the current close-up frame, step 490 is executed to determine the other head objects as the main objects of the current close-up frame. Otherwise, step 4100 is executed.

[0183] Step 490: Determine the other head objects as the main objects corresponding to the current close-up frame, and delete the close-up frames constructed for the other head objects. Execute step 4100.

[0184] For example, if another head object exists within the current close-up frame, the other head object is used as the main object of the current close-up frame. In this case, the number of main objects corresponding to the current close-up frame is incremented by 1. After the other head object is used as the main object of the current close-up frame, in order to prevent the other head object from appearing in other close-up frames, in an embodiment, the close-up frame constructed for the other head object is deleted. Optionally, after deleting the close-up frame, if there are other main objects in the close-up frame besides the other head object, a new close-up frame can be created for the other main objects.

[0185] In one embodiment, there are other head objects in the current close-up frame that are located at the edge of the current close-up frame and are relatively far away from the main objects in the current close-up frame. In this case, adding the other head object to the current close-up frame will cause the layout of the main objects in the close-up image within the current close-up frame to be poor, which may make it difficult for the user to see the head object located at the edge. It is better to place the other head object in another close-up frame (the other head object is located in a more central position in the other close-up frame). In this case, although the other head object may appear in two close-up frames, it is located in the center of one close-up frame and at the edge of the other close-up frame (perhaps only a portion of the area is located in the close-up frame). Therefore, the user's perception of the close-up image is less affected, and the head object will not be seen as having a larger area in both close-up frames. Based on this, if there are other head objects in the current close-up frame, it can be further determined whether the other head object can be added to the current close-up frame. At this time, if there are other head objects, the other head objects are determined as the main objects corresponding to the current close-up frame, and the close-up frames constructed for the other head objects are deleted. Specifically: if there are other head objects, the main object with the largest area in the current close-up frame is selected and updated as the current main object; a stable frame is constructed with the current main object as the center; the stable frame is moved, and during the movement, other head objects that can be added to the current close-up frame are found, and the other head objects that can be added to the current close-up frame have an intersection with the stable frame at the corresponding moment in the movement process; the other head objects found are determined as the main objects corresponding to the current close-up frame, and the close-up frames constructed for the other head objects are deleted; the main object with the largest area in the current close-up frame is selected and updated as the current main object, and the operation of constructing a stable frame with the current main object as the center is returned to execute until no other head objects that can be added to the current close-up frame are found.

[0186] For example, if other head objects exist within the current close-up frame, the main body object with the largest area within the current close-up frame is selected and updated as the current main body object. A stable frame is then created, and objects that can be added to the current close-up frame are searched based on the stable frame. For details on this, see the descriptions of steps 471-474. This process continues until no other head objects exist within the current close-up frame, or until every other head object within the current close-up frame has been determined to be eligible as the main body object corresponding to the current close-up frame, i.e., until no other main body objects can be added to the current close-up frame, then step 4100 is executed.

[0187] Step 4100: Determine whether there are any close-up frames that have not been traversed. If all close-up frames in the current video stream data have been traversed, execute step 4110. If not all close-up frames in the current video stream data have been traversed, execute step 4130.

[0188] For example, if every close-up frame in the current frame is selected as the current close-up frame and the subject object therein is confirmed, it can be determined that each currently required close-up frame and its corresponding subject object have been obtained, and therefore step 4100 can be executed. If there are close-up frames in the current frame that have not been traversed to be the current close-up frame, the untraversed close-up frames are updated as the current close-up frame, that is, step 4130 is executed to continue determining the subject object that the current close-up frame should correspond to.

[0189] Step 4110: Update the close-up frame recognition result and proceed to step 4120.

[0190] Exemplarily, the currently obtained close-up frames and the subject objects corresponding to each close-up frame are used as the currently obtained close-up frame recognition results, replacing the previously obtained close-up frame recognition results to achieve an update of the close-up frame recognition results. The close-up frame recognition results are used to track each head object and the close-up frame to which the head object belongs in the video stream data, and a close-up image of each close-up frame is displayed. When the close-up frame recognition results need to be updated next time, the currently obtained close-up frame recognition results can be used to determine the close-up frame to which each head object belongs in the video image.

[0191] Step 4120: Display the close-up image of each close-up frame in the video stream data.

[0192] In one embodiment, step 4120 includes: when there are multiple main objects corresponding to the close-up frame, creating a corresponding rectangular area for the close-up frame, the rectangular area is the minimum rectangular area that contains all the main objects in the corresponding close-up frame, creating a virtual object with the center point of the rectangular area as the center, the area of ​​the virtual object is equal to the area of ​​the main object with the largest area in the corresponding close-up frame, updating the frame selection position of the close-up frame in the video stream data based on the virtual object, so that the updated close-up frame is centered on the virtual object; when there is only one main object corresponding to the close-up frame, updating the frame selection position of the close-up frame in the video stream data with the main object as the center; and displaying the close-up pictures framed by each updated close-up frame in the video stream data.

[0193] For example, when there are multiple main objects in a close-up frame, in order to ensure that each main object is located in the center area as much as possible and the distribution of each main object in the close-up image is more reasonable, a rectangular area is created. This rectangular area is the minimum rectangular area that contains each main object (primarily the head area of ​​the main object). This rectangular area can be used to clearly determine the area in which each main object is concentrated. Then, a virtual object is created with the center point of the rectangular area as the center, currently recorded as the virtual object. The area of ​​the virtual object is equal to the area of ​​the main object with the largest area in the close-up frame. In this case, the virtual object can be considered to be a simulated main object with the largest area displayed at the center of the close-up frame. Then, the position of the close-up frame in the video stream data is updated with the virtual object as the center. Currently, the position of the close-up frame in the video stream data is recorded as the frame selection position, and the center of the frame selection position is the virtual object. In this case, updating the frame selection position of the close-up frame in the video stream data based on the virtual object includes: using the virtual object as the center of the close-up frame and updating the frame selection position of the close-up frame in the video stream data according to the size of the close-up frame. The size of the close-up frame is related to the aspect ratio of the close-up frame and the maximum area of ​​each subject object, so that each subject object can be distributed in the center of the close-up frame as much as possible. For example, Figure 12 is a schematic diagram of a close-up frame provided by an embodiment of the present application. Referring to Figure 12, there are two subject objects in the close-up frame 51, which are currently recorded as subject object 52 and subject object 53. At this time, based on the head area of ​​the subject object 52 and the head area of ​​the subject object 53, a minimum rectangular area 54 containing the two head areas is created. Afterwards, a virtual object is created with the center point of the minimum rectangular area 54 as the center, and the display area of ​​the virtual object is equal to the display area of ​​the subject object 52. Figure 13 is a schematic diagram of a virtual object provided by an embodiment of the present application. Referring to Figure 13, a virtual head area is created as a virtual object 55 in the minimum rectangular area 54 shown in Figure 12. Next, the position of the close-up frame is changed based on the position of the virtual object 55. FIG14 is a schematic diagram of another close-up frame according to one embodiment of the present application. Referring to FIG14 , a close-up frame 56 centered on the virtual object 55 and a stable frame 57 centered on the virtual object 55 are shown. Subsequently, when determining whether the main object in the close-up frame 5 has changed, the close-up frame 56 and the stable frame 57 can be used as a reference.

[0194] When there is a main object in the close-up frame, the main object is used as the center, and the frame selection position of the close-up frame in the video stream data is updated so that the main object is located at the center of the video stream data.

[0195] After that, the close-up images in each close-up frame can be displayed according to the set size. The set size is related to the arrangement of the close-up images. For example, referring to Figure 7, when the close-up images are arranged in a 1:1 ratio, the size of the current close-up image display area can be determined. After that, the close-up images in each close-up frame are displayed in each display area.

[0196] Step 4130: Select another close-up frame from the close-up frames and update it as the current close-up frame. Return to step 440.

[0197] In the above, by determining whether the subject object in the current close-up frame has changed, and if so, re-finding the current subject object and constructing the current close-up frame, then finding a head object that can be added to the current close-up frame and adding the head object to the current close-up frame, it is possible to ensure that when the subject object in the close-up frame changes, a suitable close-up frame is re-constructed for each subject object, thereby ensuring the rationality of the layout of the subject objects in the close-up image. Furthermore, by constructing a stable frame and moving the stable frame to search for head objects that can be added to the current close-up frame, the head objects found can all be located in the surrounding area of ​​the current subject object, that is, close to the current subject object, ensuring the rationality of the found head objects. Furthermore, by updating the subject object with the largest area as the current subject object, it is also possible to ensure that when searching for head objects, the surrounding area of ​​the subject object with the largest area is used as a reference, further ensuring a more rational layout of the subject objects in the close-up image. Furthermore, when a head object is found within the stable frame, the width and height of the smallest rectangular area encompassing the head object and each subject object in the current close-up frame are compared with the width and height of the stable frame to determine whether the found head object can be added to the current close-up frame. This ensures that the head object added to the current close-up frame is close to each subject object, further ensuring a more balanced layout of the subject objects in the close-up image. Furthermore, when the subject object in the current close-up frame has not changed, the system determines whether other head objects exist within the current close-up frame. If so, it further determines whether other head objects can be added to the current close-up frame. This prevents missing eligible head objects. Furthermore, by detecting changes in the number and area of ​​subject objects, the system determines whether the subject object in the current close-up frame has changed, ensuring effective recognition of changes in subject objects. Furthermore, when the close-up frame corresponds to multiple subject objects, the close-up frame's selection position is adjusted by constructing virtual objects, ensuring a more balanced distribution of the subject objects in the close-up image, preventing any single subject from being overly prominent.

[0198] The following is an exemplary description of a close-up frame determination method provided by an embodiment of the present application. In this example, the close-up frame recognition result is updated once per second. FIG15 is an example flow chart of a close-up frame determination method provided by an embodiment of the present application.

[0199] 15 , the close-up image determination method includes steps 510 to 5170:

[0200] Step 510: Track each head object in the video stream data according to the close-up frame recognition result to display a close-up picture in the close-up frame to which each head object belongs.

[0201] Step 520: After an interval of 1 second, identify the close-up frame to which each head object belongs in the current frame of the video stream data according to the close-up frame recognition result.

[0202] Step 530: Select a close-up frame from the close-up frames as the current close-up frame.

[0203] Step 540: Determine whether the main object in the current close-up frame has changed. If so, proceed to step 550. If not, proceed to step 5120.

[0204] Step 550: The subject object with the largest display area in the current close-up frame is taken as the current subject object, and the current close-up frame is deleted.

[0205] Step 560: Construct a close-up frame and a stable frame for the current subject object, and use the constructed close-up frame as the current close-up frame.

[0206] Step 570: Move the stabilization frame and, during the movement, search for other head objects within the current close-up frame that intersect with the stabilization frame. If no other head objects are found, proceed to step 5130. If other head objects are found, proceed to step 580.

[0207] Step 580: Determine whether the height and width of the smallest rectangular area containing the other head objects found and the main body objects corresponding to the current close-up frame are smaller than the height and width of the stabilization frame. If so, execute step 590; otherwise, execute step 5110.

[0208] Step 590: Determine the other head objects found as the main objects corresponding to the current close-up frame, and delete the close-up frames constructed for the other head objects. Execute step 5100.

[0209] Step 5100 : Select the subject object with the largest area in the current close-up frame and update it as the current subject object. Build a stable frame for the current subject object, and then return to step 570 .

[0210] Step 5110: Abandon determining the other head objects currently found as the main body objects corresponding to the current close-up frame, and execute step 5130.

[0211] Step 5120: Determine whether there are other head objects in the current close-up frame. If there are other head objects, execute step 5100. If there are no other head objects, execute step 5130.

[0212] Step 5130: Determine whether there are any untraversed close-up frames. If there are any untraversed close-up frames, proceed to step 5140. If there are no untraversed close-up frames, proceed to step 5150.

[0213] Step 5140: Select another close-up frame from the close-up frames and update it as the current close-up frame. Return to step 540.

[0214] Step 5150: When there are multiple main objects corresponding to the close-up frame, create a corresponding rectangular area for the close-up frame, create a virtual object with the center point of the rectangular area as the center, and update the frame selection position of the close-up frame in the video stream data based on the virtual object; when there is only one main object corresponding to the close-up frame, update the frame selection position of the close-up frame in the video stream data with the main object as the center.

[0215] Step 5160: Display the close-up images of each updated close-up frame in the video stream data.

[0216] That is, the close-up frame recognition result is updated, and then the process returns to step 510. This process continues until the current video exchange ends or the close-up frame is displayed.

[0217] Through the above example, it is possible to avoid a head object from appearing repeatedly in multiple close-up images as much as possible, and the layout of the head objects in each close-up image is more reasonable, thereby improving the user's viewing experience of the close-up images.

[0218] One embodiment of the present application further provides a close-up image determination device. FIG16 is a schematic diagram of the structure of a close-up image determination device provided in one embodiment of the present application. Referring to FIG16 , the close-up image determination device includes: a close-up frame recognition unit 601, a close-up frame selection unit 602, a first object determination unit 603, a first close-up frame update unit 604, and a close-up display unit 605.

[0219] Among them, the close-up frame identification unit 601 is used to identify the close-up frame to which each head object in the current frame of the video stream data belongs, each close-up frame corresponding to at least one main object, and the main object is the head object that has been determined to be framed by the corresponding close-up frame; the close-up frame selection unit 602 is used to select a close-up frame from the close-up frames as the current close-up frame; the first object determination unit 603 is used to, when determining that there are other head objects that can be added to the current close-up frame, determine the other head objects as the main objects corresponding to the current close-up frame, and delete the close-up frames constructed for the other head objects; the first close-up frame updating unit 604 is used to select another close-up frame from the close-up frames and update it to the current close-up frame, and return to executing the operation of determining that there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, until all current close-up frames are traversed; the close-up display unit 605 is used to display the close-up picture selected by each close-up frame in the video stream data.

[0220] On the basis of the above embodiment, the first object determination unit 603 includes: a change judgment subunit, used to judge whether the main object in the current close-up frame has changed; a first object selection subunit, used to select the main object with the largest area in the current close-up frame as the current main object if the main object in the current close-up frame has changed, and delete the current close-up frame; a close-up frame reconstruction subunit, used to reconstruct a close-up frame for the current main object and use it as the current close-up frame; a second object determination subunit, used to find other head objects that can be added to the current close-up frame, determine the other head objects as the main object corresponding to the current close-up frame, and delete the close-up frames constructed for the other head objects.

[0221] Based on the above embodiment, the second object determination subunit includes: a first stable frame construction grandchild unit, which is used to construct a stable frame with the current main object as the center; a first stable frame movement grandchild unit, which is used to move the stable frame and, during the movement, find other head objects that can be added to the current close-up frame, and the other head objects that can be added to the current close-up frame have an intersection with the stable frame at the corresponding moment during the movement; a third object determination grandchild unit, which is used to determine the other head objects found as the main object corresponding to the current close-up frame, and delete the close-up frame constructed for the other head objects; a first object update grandchild unit, which is used to select the main object with the largest area in the current close-up frame and update it to the current main object, and return to execute the operation of constructing a stable frame with the current main object as the center until no other head objects that can be added to the current close-up frame are found.

[0222] On the basis of the above embodiment, the close-up picture determination device also includes: an object existence determination unit, which is used to determine whether the main object in the current close-up frame has changed, and if the main object in the current close-up frame has not changed, determine whether there are other head objects in the current close-up frame; a fourth object determination unit, which is used to determine the other head objects as the main object corresponding to the current close-up frame if there are other head objects, delete the close-up frames constructed for the other head objects, and execute the operation of selecting another close-up frame from the close-up frames to update to the current close-up frame; a second close-up frame updating unit, which is used to select another close-up frame from the close-up frames to update to the current close-up frame if there are no other head objects.

[0223] Based on the above embodiment, the fourth object determination unit includes: a second object selection subunit, which is used to select the main object with the largest area in the current close-up frame and update it as the current main object if there are other head objects; a second stable frame construction subunit, which constructs a stable frame with the current main object as the center; a second stable frame movement subunit, which is used to move the stable frame and, during the movement, find other head objects that can be added to the current close-up frame, where the other head objects that can be added to the current close-up frame have an intersection with the stable frame at the corresponding moment during the movement; a fifth object determination subunit, which is used to determine the other head objects found as the main object corresponding to the current close-up frame and delete the close-up frames constructed for the other head objects; and a second object update subunit, which is used to select the main object with the largest area in the current close-up frame and update it as the current main object, and return to executing the operation of constructing a stable frame with the current main object as the center until no other head objects that can be added to the current close-up frame are found.

[0224] On the basis of the above embodiment, during the movement of the stabilization frame, the position of the corresponding current subject object remains unchanged and always remains in the stabilization frame.

[0225] Based on the above embodiment, the size of the stabilization frame is determined by the display size of the corresponding current subject object in the current frame.

[0226] Based on the above embodiment, the first stable frame moving sub-unit and the second stable frame moving sub-unit are specifically used to: move the stable frame, and during the movement, search for other head objects located in the current close-up frame that intersect with the stable frame; determine the minimum rectangular area containing the other head objects found and the main object in the current close-up frame; when the width and height of the minimum rectangular area are both smaller than the width and height of the stable frame, determine that the other head objects found can be added to the current close-up frame.

[0227] Based on the above embodiment, the change judgment subunit is specifically used to: judge whether the number of subject objects in the current close-up frame has changed; or, judge whether the area change range of the subject object with the largest area in the current close-up frame exceeds the area threshold; or, judge whether there is a subject object in the current close-up frame that exceeds the stable frame created for the subject object with the largest area.

[0228] On the basis of the above embodiment, the close-up display unit 605 includes: a first framing position updating sub-unit, which is used to create a corresponding rectangular area for the close-up frame when there are multiple main objects corresponding to the close-up frame, and the rectangular area is the minimum rectangular area containing all the main objects in the corresponding close-up frame. A virtual object is created with the center point of the rectangular area as the center, and the area of ​​the virtual object is equal to the area of ​​the main object with the largest area in the corresponding close-up frame. The framing position of the close-up frame in the video stream data is updated based on the virtual object so that the updated close-up frame is centered on the virtual object; a second framing position updating sub-unit, which is used to update the framing position of the close-up frame in the video stream data with the main object as the center when there is only one main object corresponding to the close-up frame; a close-up picture display sub-unit, which is used to display the close-up pictures framed by each updated close-up frame in the video stream data.

[0229] Based on the above embodiment, close-up frame identification unit 601 includes a result acquisition subunit for obtaining the most recent close-up frame identification result for the video stream data; and a close-up identification subunit for determining, based on the close-up frame identification result, the close-up frame to which each head object in the current frame of the video stream belongs. Accordingly, the close-up frame identification device further includes a result update unit for updating the close-up frame identification result after traversing all current close-up frames.

[0230] The above-mentioned close-up picture determination device can be used to execute the close-up picture determination method provided by any of the above-mentioned embodiments, and has corresponding functions and beneficial effects.

[0231] It is worth noting that in the embodiment of the above-mentioned close-up image determination device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0232] One embodiment of the present application further provides a close-up image determination device. Referring to Figure 5 , the close-up image determination device includes a processor 21, a memory 22, and a display screen 23. Display screen 23 is configured to display close-up images, and memory 22 is configured to store one or more programs. When one or more programs are executed by one or more processors 21, one or more processors 21 implement the close-up image determination method described in any of the aforementioned embodiments. For details on each component, refer to the aforementioned description.

[0233] The above-mentioned close-up picture determination device includes a close-up picture determination device, which can be used to execute any close-up picture determination method and has corresponding functions and beneficial effects. For specific details not currently described, please refer to the relevant description of the above-mentioned close-up picture determination method.

[0234] One embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a processor of a close-up image determination device, are used to perform relevant operations in the close-up image determination method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0235] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0236] Therefore, the application can adopt the form of complete hardware embodiment, complete software embodiment, or the embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The application is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow chart and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processing module of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instruction executed by the processing module of the computer or other programmable data processing device produces a device for realizing the function specified in one flow chart flow or multiple flows and / or one block or multiple blocks of the block diagram. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device that implements the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram.

[0237] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0238] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity, or device. Without further restriction, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, commodity, or device comprising the element. Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for determining a close-up view, wherein, Including: Identifying a close-up frame to which each head object in the current frame of the video stream data belongs, each of the close-up frames corresponding to at least one main object, and the main object being a head object that has been determined to be framed by the corresponding close-up frame; Selecting one of the close-up frames as the current close-up frame; When it is determined that there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects; Selecting another close-up frame from the close-up frames to update it as the current close-up frame, and returning to execute the operation of determining that when there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, until all the current close-up frames are traversed; Displaying the close-up images framed by the respective close-up frames in the video stream data.

2. The method for determining a close-up picture according to claim 1, wherein, When it is determined that there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, includes: Judging whether the main object in the current close-up frame has changed; If the main object in the current close-up frame has changed, selecting the main object with the largest area in the current close-up frame as the current main object, and deleting the current close-up frame; Reconstructing a close-up frame for the current main object and using it as the current close-up frame; Searching for other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects.

3. The method for determining a close-up picture according to claim 2, wherein, Searching for other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, includes: Constructing a stable frame centered on the current main object; Moving the stable frame, and during the moving process, searching for other head objects that can be added to the current close-up frame, and there is an intersection between the other head objects that can be added to the current close-up frame and the stable frame at the corresponding moment during the moving process; Determining the found other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects; Selecting the main object with the largest area in the current close-up frame to update it as the current main object, and returning to execute the operation of constructing a stable frame centered on the current main object until no other head objects that can be added to the current close-up frame are found.

4. The method for determining a close-up picture according to claim 2, wherein, After judging whether the main object in the current close-up frame has changed, includes: If the main object in the current close-up frame has not changed, determining whether there are other head objects in the current close-up frame; If there are other head objects, determine the other head object as the main object corresponding to the current close-up frame, delete the close-up frame constructed for the other head object, and perform the operation of selecting another close-up frame from the close-up frames and updating it as the current close-up frame; If there are no other head objects, perform the operation of selecting another close-up frame from the close-up frames and updating it as the current close-up frame.

5. The method for determining a close-up picture according to claim 4, wherein, The "If there are other head objects, determine the other head object as the main object corresponding to the current close-up frame, and delete the close-up frame constructed for the other head object" includes: If there are other head objects, select the main object with the largest area in the current close-up frame and update it as the current main object; Construct a stable frame centered on the current main object; Move the stable frame, and during the movement, find other head objects that can be added to the current close-up frame, where the other head objects that can be added to the current close-up frame have an intersection with the stable frame at the corresponding moment during the movement; Determine the other head object found as the main object corresponding to the current close-up frame, and delete the close-up frame constructed for the other head object; Select the main object with the largest area in the current close-up frame and update it as the current main object, and return to perform the operation of constructing a stable frame centered on the current main object until no other head objects that can be added to the current close-up frame are found.

6. The method for determining a close-up picture according to claim 3 or 5, wherein, During the movement of the stable frame, the position of the corresponding current main object remains unchanged and always stays within the stable frame.

7. The method for determining a close-up picture according to claim 3 or 5, wherein, The size of the stable frame is determined by the display size of the corresponding current main object in the current frame of the video.

8. The method for determining a close-up picture according to claim 3 or 5, wherein The "Move the stable frame, and during the movement, find other head objects that can be added to the current close-up frame" includes: Move the stable frame, and during the movement, find other head objects located within the current close-up frame that have an intersection with the stable frame; Determine the smallest rectangular area that contains the other head objects found and the main object in the current close-up frame; When both the width and height of the smallest rectangular area are smaller than the width and height of the stable frame, determine that the other head objects found can be added to the current close-up frame.

9. The method for determining a close-up view according to claim 2, wherein, The "Judge whether the main object in the current close-up frame has changed" includes: Judge whether the number of main objects in the current close-up frame has changed; or, Judge whether the area change range of the main object with the largest area in the current close-up frame exceeds the area threshold; or, Judge whether there are main objects in the current close-up frame that exceed the stable frame created for the main object with the largest area.

10. The method for determining a close-up picture according to claim 1, wherein, The "Display the close-up images framed by each of the close-up frames in the video stream data" includes: When there are multiple main objects corresponding to a close-up frame, create a corresponding rectangular area for the close-up frame. The rectangular area is the smallest rectangular area that contains all the main objects within the corresponding close-up frame. Create a virtual object centered on the center point of the rectangular area. The area of the virtual object is equal to the area of the main object with the largest area within the corresponding close-up frame. Update the selected position of the close-up frame in the video stream data based on the virtual object, so that the updated close-up frame is centered on the virtual object; When there is one main object corresponding to a close-up frame, update the selected position of the close-up frame in the video stream data with the main object as the center; Display the close-up images selected by each updated close-up frame in the video stream data.

11. The method for determining a close-up view according to claim 1, wherein, Identifying the close-up frame to which each head object in the current frame image in the video stream data belongs includes: Obtain the close-up frame recognition result obtained for the video stream data most recently; Determine the close-up frame to which each head object in the current frame image in the video stream data belongs according to the close-up frame recognition result; After traversing all the current close-up frames, it further includes: Update the close-up frame recognition result.

12. A close-up view determination device, wherein, It includes: A close-up frame recognition unit for identifying the close-up frame to which each head object in the current frame image in the video stream data belongs. Each close-up frame corresponds to at least one main object, and the main object is a head object that has been determined to be framed by the corresponding close-up frame; A close-up frame selection unit for selecting one of the close-up frames as the current close-up frame; A first object determination unit for determining that when there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects; A first close-up frame update unit for selecting another close-up frame from the close-up frames to update as the current close-up frame, and returning to execute the operation of determining that when there are other head objects that can be added to the current close-up frame, determining the other head objects as the main objects corresponding to the current close-up frame, and deleting the close-up frames constructed for the other head objects, until all the current close-up frames are traversed; A close-up display unit for displaying the close-up images selected by each of the close-up frames in the video stream data.

13. A close-up image determination device, wherein, It includes: One or more processors; A display screen for displaying close-up images; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the close-up image determination method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the close-up image determination method according to any one of claims 1-11.

Images