Image processing method, electronic equipment and storage medium

Abstract

The invention discloses an image processing method, electronic equipment and a storage medium, and belongs to the field of image data. The method comprises the steps of loading and displaying videos and pictures in a picture library in response to a first operation of a user; in response to the fact that the user selects the image materials in the image library and confirms to execute cutting operation, the image materials are cut in batches according to the first picture proportion, so that the picture proportion of the image materials is the first picture proportion; wherein the image materials comprise videos and / or pictures; in the process of cutting the image material in batches according to the first picture proportion, after a first image frame sequence in the image material is processed, the first image frame sequence with the first picture proportion obtained through processing is previewed and displayed, and the first image frame sequence is any batch of image frames processed in batches in the image material. In this way, the situation that black edges appear in the processed video can be avoided, and real-time preview of the cutting processing result can be achieved.

Description

Technical Field

[0001] The present application relates to the field of image data, and in particular to an image processing method, electronic equipment and storage medium. Background Art

[0002] At present, in order to achieve better display effects of images or videos captured by electronic devices, users can import image materials (image materials can be images, videos, or both) into electronic devices or image processing software provided by third parties for processing, for example, splicing the image materials. However, when multiple image materials with different aspect ratios are imported into the image processing software for processing, black edges may appear in the processed image materials that are finally saved. Moreover, the image processing software can only export and display the processed image materials after all the imported image materials have been processed, which results in low flexibility. Summary of the invention

[0003] The present application provides an image processing method, an electronic device and a storage medium, which can avoid the appearance of black edges in the processed video, and can realize real-time preview of the image processing results, thereby improving the flexibility of image processing display. The technical solution is as follows:

[0004] In a first aspect, an image processing method is provided, the method comprising:

[0005] The user performs a first operation in the gallery interface, and in response to the first operation, the videos and pictures in the gallery are loaded and displayed. Afterwards, the user selects an image material in the gallery and confirms to perform cropping. In response to the operation of selecting the image material and confirming to perform cropping, the image material is cropped in batches according to the first aspect ratio, so that the aspect ratio of the image material is the first aspect ratio. Moreover, in the process of cropping the image material in batches according to the first aspect ratio, after processing any batch of image frames in the image material, i.e., the first image frame sequence, is completed, the first image frame sequence obtained by the processing with the first aspect ratio is previewed and displayed, such as displaying a preview video of the first image frame sequence with the first aspect ratio.

[0006] By cropping the image material so that the image material's aspect ratio is the first aspect ratio, the black edges in the processed video due to different aspect ratios can be avoided. By cropping the image material in batches, and after any batch of image frames processed in the image material are processed, the processed image frames are previewed, so that the cropping results can be previewed while the image material is cropped, so that the cropping results can be previewed in real time, which improves the flexibility of image processing display.

[0007] In combination with the first aspect, in some implementations of the first aspect, the image material may be cropped in batches according to the first frame ratio and the subject position in the image material. In this way, not only can the appearance of black edges in the final processed video be avoided, but also the subject loss or missing may be avoided, thereby improving the user experience.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the process of cropping image materials in batches according to the first frame ratio includes: sampling multiple image frames included in the first image frame sequence to obtain multiple sampling frames, the number of the multiple sampling frames is less than the number of the multiple image frames; performing subject detection on the multiple sampling frames to obtain subject position information of each sampling frame in at least some of the multiple sampling frames, the subject position information of each sampling frame is used to indicate the position of the target subject in each sampling frame in the corresponding sampling frame; determining cropping frame position information of each image frame in the multiple image frames according to the subject position information of at least some of the sampling frames and the first frame ratio, the cropping frame position information is used to indicate the position of the cropping frame in the corresponding image frame, and the frame ratio of the cropping frame is the first frame ratio; cropping each image frame according to the cropping frame position information of each image frame in the multiple image frames to obtain multiple first image frames corresponding to the multiple image frames one by one, and the frame ratio of the first image frames is the first frame ratio.

[0009] By first sampling the first image frame sequence to obtain a sampling frame, and then performing subject detection on the sampling frame to obtain subject position information of the sampling frame, the cropping frame position information of the first image frame sequence is determined according to the subject position information of the sampling frame and the first frame ratio, and then each image frame is cropped according to the cropping frame position information of each image frame to obtain the first image frame corresponding to this batch of image frames, the complexity of frame cropping can be reduced, the efficiency of frame cropping can be improved, and strong support can be provided for the real-time preview of the frame cropping result. In addition, the accuracy of frame cropping is also improved to ensure that the subject is not lost or missing.

[0010] In combination with the first aspect, in some implementations of the first aspect, multiple image frames in the first image frame sequence may be grouped to obtain multiple groups of image frames, and then each group of sampled frames in the multiple groups of image frames may be sampled to extract at least one sampled frame from each group of sampled frames, thereby obtaining multiple sampled frames. In this way, the balance of sampling may be improved, thereby improving the efficiency and accuracy of frame cropping.

[0011] In combination with the first aspect, in certain implementations of the first aspect, a sliding time window can be used to slide on image frames in the image material in sequence, and according to the sliding time window, a sequence of image frames to be processed is extracted from the image material in sequence as a first image frame sequence.

[0012] In combination with the first aspect, in certain implementations of the first aspect, a sliding time window is used to slide on image frames in the image material in sequence. In the process of sliding on image frames in the image material in sequence using the sliding time window, if the sliding time window slides to a first continuous frame sequence in which all aspect ratios in the image material are not the first aspect ratio, then, according to the sliding time window, the image frame sequence to be processed is extracted from the first continuous image frame sequence in sequence as the first image frame sequence. If the sliding time window slides to a second continuous frame sequence in which all aspect ratios in the image material are the first aspect ratio, then, according to the sliding time window, the image frame sequence is extracted from the second continuous image frame sequence in sequence, and the extracted image frame sequence is previewed and displayed.

[0013] Wherein, when there is an image frame before the first continuous image frame sequence, the aspect ratio of the previous image frame of the first continuous image frame sequence is the first aspect ratio, and when there is an image frame after the first continuous image frame sequence, the aspect ratio of the next image frame of the first continuous image frame sequence is the first aspect ratio. When there is an image frame before the second continuous image frame sequence, the aspect ratio of the previous image frame of the second continuous image frame sequence is not the first aspect ratio, and when there is an image frame after the second continuous image frame sequence, the aspect ratio of the next image frame of the second continuous image frame sequence is not the first aspect ratio.

[0014] In this way, for some image frames whose original aspect ratio in the image material is the first aspect ratio, frame cropping is not required and they can be directly previewed and displayed. For continuous image frames whose aspect ratio in the image material is not the first aspect ratio, the sampling sliding time window is used to perform cropping in batches, which further improves the cropping efficiency and provides support for real-time preview of the frame cropping results.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the process of performing subject detection on multiple sampling frames to obtain subject position information of each sampling frame in at least some of the multiple sampling frames includes: performing subject detection on the first sampling frame to obtain a subject detection result, and the first sampling frame is any one of the multiple sampling frames. In the case where the subject detection result indicates that there is no subject in the first sampling frame, that is, no subject is detected when subject detection is performed on the first sampling frame, it is determined that there is no target subject in the first sampling frame. In this way, for the sampling frame without the target subject, when subsequently determining its cropping frame position information, the cropping frame position information of the sampling frame can be determined based on the position information of the preset position of the sampling frame and the first frame ratio. Among them, the preset position can be the center position of the sampling frame, etc., which is not limited in the embodiments of the present application.

[0016] In addition, when the subject detection result includes subject position information of at least one subject, the spatial dimension information and the temporal dimension information of each subject in the at least one subject are determined. According to the spatial dimension information and the temporal dimension information of the at least one subject, a target subject is determined from the at least one subject, and the position information of the target subject is used as the subject position information of the first sampling frame.

[0017] The spatial dimension information includes the area ratio and center distance of the corresponding subject. The area ratio refers to the ratio between the area of ​​the corresponding subject and the image area of ​​the first sampling frame, and the center distance refers to the distance between the corresponding subject and the center of the picture of the first sampling frame. In addition, the center distance can also be replaced by the edge distance, which refers to the distance between the corresponding subject and the edge of the picture of the first sampling frame.

[0018] The time dimension information includes the appearance duration information of the corresponding subject in the first image frame sequence, such as the appearance duration information of the corresponding subject in the sliding time window. The appearance duration information can be the appearance duration of the corresponding subject in the first image frame sequence, or can be the ratio between the appearance duration of the corresponding subject in the first image frame sequence and the total duration of the first image frame sequence. As an example, the appearance duration of the corresponding subject in the first image frame sequence can be represented by the appearance duration of the corresponding subject in multiple sampling frames.

[0019] By determining the target subject from at least one subject based on the spatial dimension information and temporal dimension information of at least one subject in the sampling frame, the accuracy of determining the significant subject in the sampling frame can be improved, thereby improving the accuracy of frame cropping and avoiding the situation where the subject is lost in the frame cropping result.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the process of determining a target subject from at least one subject based on the spatial dimension information and temporal dimension information of at least one subject includes: if there is a subject in at least one subject whose appearance duration information meets a first preset condition, then the subject whose appearance duration information meets the first preset condition is determined as the target subject.

[0021] Among them, the first preset condition may be that the appearance duration in the first image frame sequence is greater than the first preset duration, or the appearance duration ratio is greater than the first preset ratio, and the appearance duration ratio refers to the ratio between the appearance duration of the corresponding subject in the first sampling frame and the total duration of the first sampling frame. Of course, the first preset condition may also be other conditions related to the appearance duration, which is not limited in the embodiment of the present application.

[0022] By identifying a subject whose appearance duration information meets the first preset condition as a target subject, a subject that frequently appears in the first image frame sequence can be identified as a target subject, which facilitates subsequent frame cropping based on the frequently appearing subjects, so that the frequently appearing subjects can be included in the frame cropping result, thereby improving the accuracy of frame cropping.

[0023] In combination with the first aspect, in certain implementations of the first aspect, if there is no subject in at least one subject whose appearance duration information satisfies the first preset condition, the target subject is determined from at least one subject based on the area ratio and center distance of each subject in at least one subject.

[0024] In combination with the first aspect, in certain implementations of the first aspect, when at least one subject includes multiple subjects, a target subject is determined from at least one subject based on the area ratio and center distance of each subject in the at least one subject, including: if the maximum distance difference between the center distances of the multiple subjects is less than or equal to a distance difference threshold, then the subject whose area ratio meets the second preset condition among the multiple subjects is determined as the target subject; if the maximum area difference between the area ratios of the multiple subjects is less than or equal to the area difference threshold, then the subject whose center distance meets the third preset condition among the at least one subject is determined as the target subject.

[0025] In this way, when the distances between multiple subjects and the center of the picture are close, the subject with a larger area ratio can be identified as the target subject. When the area ratios of multiple subjects are similar, the subject with a smaller center distance can be identified as the target subject.

[0026] In combination with the first aspect, in certain implementations of the first aspect, when at least one subject includes a subject, if the area ratio of the subject is greater than or equal to a second preset ratio, and / or the center distance of the subject is less than or equal to a first distance threshold, the subject is determined as a target subject.

[0027] In combination with the first aspect, in certain implementations of the first aspect, if the area ratio of at least one subject is less than the second preset ratio threshold, and the center distance of at least one subject is greater than the first distance threshold, it is determined that there is no target subject in the first sampling frame.

[0028] That is, when there is no obvious subject (the area of ​​multiple subjects is small and they are far from the center of the picture), it is determined that the first sampling frame has no target subject. For the sampling frame without the target subject, when determining its cropping frame position information later, the cropping frame position information of the sampling frame can be determined according to the position information of the preset position of the sampling frame and the first frame ratio.

[0029] In one example, before subject detection is performed on multiple sampling frames, each sampling frame may be subjected to resolution reduction processing to reduce the resolution of the sampling frame to obtain a sampling frame with a lower resolution, and subject detection is then performed on the sampling frame after the resolution reduction processing. The resolution reduction processing may be downsampling, etc.

[0030] In combination with the first aspect, in certain implementations of the first aspect, determining the cropping frame position information of each image frame in a plurality of image frames based on the subject position information of at least a portion of the sampling frames and the first frame ratio includes: determining the cropping frame position information of the plurality of sampling frames based on the subject position information of at least a portion of the sampling frames and the first frame ratio; and interpolating the cropping frame position information of the plurality of sampling frames to obtain the cropping frame position information of the plurality of image frames.

[0031] By interpolating the cropping frame position information of multiple sampling frames to obtain the cropping frame position information of multiple image frames, the complexity of determining the cropping frame position can be reduced, the efficiency of determining the cropping frame position can be improved, and the efficiency of frame cropping can be improved, thereby providing support for real-time preview of frame cropping results.

[0032] In combination with the first aspect, in certain implementations of the first aspect, when a single subject appears in multiple sampling frames, the cropping frame position information of the multiple sampling frames can be determined based on the motion trajectory of the single subject and the first frame ratio, and the cropping frames of the multiple sampling frames include the motion trajectory of the single subject.

[0033] For example, if it is determined based on the subject position information of at least some of the sampling frames that the target subjects in at least some of the sampling frames are all the first target subject, or if the appearance duration of the first target subject in at least some of the sampling frames meets the fourth preset condition, it is determined that a single subject appears in multiple sampling frames, and then the first target subject is subject tracked based on the multiple sampling frames to obtain the motion trajectory of the first target subject, and then the cropping frame position information of the multiple sampling frames is determined based on the motion trajectory and the first frame ratio. Wherein, the cropping frames of the multiple sampling frames include the motion trajectory of the first target subject.

[0034] Among them, the fourth preset condition may be that the appearance duration is greater than the second preset duration, or that the ratio between the appearance duration of the first target subject in multiple sampling frames and the total duration of multiple sampling frames is greater than or equal to the fifth preset ratio. Of course, it may also be other preset conditions related to the appearance duration.

[0035] By determining the cropping frame position information of multiple sampling frames according to the motion trajectory of a single target and the first frame ratio, it can be ensured that the cropping frames of the multiple sampling frames can contain the motion trajectory of the single target, so that the frame cropping result can reflect the motion trajectory of the single target, thereby improving the accuracy of frame cropping and avoiding the loss of the subject.

[0036] In combination with the first aspect, in certain implementations of the first aspect, determining the cropping frame position information of multiple sampling frames according to the motion trajectory and the first frame ratio includes: if the first target subject in the second sampling frame is within the first cropping frame, determining the position information of the first cropping frame as the cropping frame position information of the second sampling frame, the position of the first cropping frame in the second sampling frame is the same as the position of the second cropping frame in the third sampling frame, the second cropping frame is the cropping frame of the third sampling frame, and the third sampling frame is a sampling frame in which the first target subject exists before the second sampling frame; wherein, if the third sampling frame is the first sampling frame in which the first target subject exists among the multiple sampling frames, the position information of the second cropping frame is determined according to the position information of the first target subject in the third sampling frame and the first frame ratio; if the first target subject in the second sampling frame is not within the first cropping frame, determining the cropping frame position information of the second sampling frame according to the position information of the first target subject in the second sampling frame and the first frame ratio.

[0037] That is, in the process of determining the cropping frame position information of multiple sampling frames according to the motion trajectory of the first target subject and the first frame ratio, if the motion range of the first target subject is within the range of the cropping frame, the cropping frame position will not be changed; if the motion range exceeds the range of the cropping frame, the cropping frame position information will be determined according to the current position information of the first target subject and the first frame ratio. In this way, the frame cropping result can reflect the motion trajectory of the target subject, and the smoothness of the frame cropping can be guaranteed to avoid severe shaking of the subject position.

[0038] In combination with the first aspect, in certain implementations of the first aspect, if there is a fifth sampling frame including the first target subject after the fourth sampling frame including the first target subject, and the first target subject does not exist in the sampling frame between the fourth sampling frame and the fifth sampling frame, and the time interval between the fourth sampling frame and the fifth sampling frame is less than or equal to the duration threshold, then the cropping frame position information of the fifth sampling frame is determined based on the cropping frame position information of the fourth sampling frame, and the position of the cropping frame of the fourth sampling frame in the fourth sampling frame is the same as the position of the cropping frame of the fifth sampling frame in the fifth sampling frame.

[0039] That is, in the process of determining the cropping frame position information of multiple sampling frames according to the motion trajectory of the first target subject and the first frame ratio, if the first target subject disappears for a short time and then recovers, the frame cropping position remains unchanged. In this way, the smoothness of the frame cropping can be ensured and the violent shaking of the subject position can be avoided.

[0040] In combination with the first aspect, in certain implementations of the first aspect, determining the cropping frame position information of multiple sampling frames based on the subject position information and the first frame ratio of at least part of the sampling frames includes: if it is determined based on the subject position information of at least part of the sampling frames that the target subjects in at least part of the sampling frames are not the same target subject, or the appearance duration of the same target subject in at least part of the sampling frames does not meet the fourth preset condition, then determining the cropping frame position information of each sampling frame based on the subject position information and the first frame ratio of each sampling frame in at least part of the sampling frames.

[0041] In combination with the first aspect, in certain implementations of the first aspect, at least some of the sampling frames include a sixth sampling frame, and there are multiple target subjects in the sixth sampling frame; determining the cropping frame position information of each sampling frame based on the subject position information and the first frame ratio of each sampling frame in at least some of the sampling frames includes: if the subject aggregation of the multiple target subjects meets the fifth preset condition, then determining the cropping frame position information of the sixth sampling frame based on the center position information and the first frame ratio of the multiple target subjects; if the subject aggregation of the multiple target subjects does not meet the fifth preset condition, then determining the significant subject from the multiple target subjects based on the area ratio, center distance or appearance duration information of the multiple target subjects in multiple sampling frames, and determining the cropping frame position information of the sixth sampling frame based on the subject position information and the first frame ratio of the significant subject.

[0042] In this way, when multiple target subjects appear in a certain sampling frame, the cropping frame position information can be determined according to the aggregation characteristics of the multiple target subjects. If the aggregation of the multiple target subjects meets the aggregation conditions, the cropping frame position information is determined according to the center position information of the multiple target subjects and the first frame ratio.

[0043] In combination with the first aspect, in certain implementations of the first aspect, before the image material is cropped in batches according to the first aspect ratio, the image material can also be parsed through a parsing node in a preview pipeline to obtain image frames in the image material, and the preview pipeline includes a parsing node, a rendering node, and a display node connected in sequence; then, the image frames in the image material are cropped in batches through a frame cropping service pipeline, and the frame cropping service pipeline is located after the parsing node and before the rendering node; after the first image frame sequence in the image material is processed through the frame cropping service pipeline, the first image frame sequence with the first aspect ratio obtained by the processing is rendered through the rendering node; the rendered image frame sequence with the first aspect ratio is displayed through the display node, so as to preview and display the processed first image frame sequence with the first aspect ratio.

[0044] By constructing a preview pipeline and a frame cropping business pipeline to crop image materials in batches, the efficiency and reliability of frame cropping can be improved.

[0045] In combination with the first aspect, in some implementations of the first aspect, the preview pipeline also includes a cache node, which is located after the parsing node and before the rendering node, and the frame cropping service pipeline is located after the cache node and before the rendering node; after parsing the image material through the parsing node in the preview pipeline, the image frame cache node can also be used to extract a to-be-processed image frame sequence from the image frames in the image material in sequence according to a sliding time window as a first image frame sequence, and the extracted first image frame sequence is cached; then, the first image frame sequence cached by the cache node is obtained through the frame cropping service pipeline, and each time a group of first image frame sequences is obtained, the first image frame sequence is cropped so that the frame ratio of the first image frame sequence is the first frame ratio.

[0046] In combination with the first aspect, in certain implementations of the first aspect, in response to a user selecting an image material in a gallery and confirming an operation to perform cropping, a first interface is displayed, the first interface including multiple aspect ratios; in response to the user selecting a first aspect ratio, the image material is cropped in batches according to the first aspect ratio.

[0047] That is, the embodiment of the present application also supports the user to select an aspect ratio from multiple aspect ratio options, and the image materials are cropped in batches according to the aspect ratio selected by the user. In this way, the personalization and flexibility of the aspect ratio cropping can be improved.

[0048] In a second aspect, an image processing device is provided, which has the function of implementing the image processing method in the first aspect. The image processing device includes at least one module, and the at least one module is used to implement the image processing method provided in the first aspect.

[0049] In a third aspect, an image processing device is provided, wherein the structure of the image processing device includes a processor and a memory, wherein the memory is used to store a program that supports the image processing device to execute the image processing method provided in the first aspect, and to store data involved in implementing the image processing method described in the first aspect. The processor is configured to execute the program stored in the memory. The image processing device may also include a communication bus, wherein the communication bus is used to establish a connection between the processor and the memory.

[0050] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the image processing method described in the first aspect.

[0051] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the image processing method described in the first aspect.

[0052] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

[0055] Figure 3 is a schematic diagram of a software structure of another electronic device provided in an embodiment of the present application;

[0056] Figure 4 is an example diagram of video processing provided by an embodiment of the present application;

[0057] Figure 5 This is an example diagram of a black-bordered screen provided in an embodiment of the present application;

[0058] Figure 6 It is a schematic diagram of a user interface for splicing multiple videos using an image processing software provided in the related art;

[0059] Figure 7 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0060] Figure 8 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0061] Fig. 9 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0062] Fig.10 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0063] Fig.11 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0064] Fig.12 It is a schematic diagram of a process of cropping a batch of image frames processed in batches provided by an embodiment of the present application;

[0065] Fig.13This is an example diagram of a sliding time window sliding on an image material provided by an embodiment of the present application;

[0066] Fig.14 is a schematic diagram of a sampling frame provided in an embodiment of the present application;

[0067] Fig.15 is a schematic diagram of another sampling frame provided in an embodiment of the present application;

[0068] Fig.16 is a schematic diagram of another sampling frame provided in an embodiment of the present application;

[0069] Fig.17 is a schematic diagram of the position of a cropping frame of a group of sampling frames provided in an embodiment of the present application;

[0070] Fig.18 It is a schematic diagram of cropping image frames in a video in batches provided by an embodiment of the present application;

[0071] Fig.19 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0072] Fig. 20 It is a schematic flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0074] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different. In the description of this embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0075] The image processing method provided in the embodiment of the present application is applied to electronic devices. The electronic device may also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device may be an electronic device (mobile phone), a smart TV, a wearable device, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the electronic device.

[0076] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 1 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0079] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0080] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0081] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0082] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0083] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The USB interface 130 can also be used to connect other electronic devices, such as AR devices, etc.

[0084] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0085] The electronic device 100 can realize the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0086] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0087] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0088] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0089] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is an integer greater than 1.

[0090] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0091] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0092] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0093] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0094] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0095] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 and the application processor.

[0096] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0097] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0098] Understandably, Figure 1 The components included in the hardware structure shown do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0099] Next, the software system of the electronic device 100 will be described.

[0100] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile terminal (such as a mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.

[0101] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0102] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0103] Figure 2 1 is a schematic diagram of the software structure of an electronic device 100 provided in an embodiment of the present application. The software structure includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software structure is divided from top to bottom into an application layer, a media middle platform framework layer, a smart middle platform framework layer, a smart engine framework layer, and an application framework layer.

[0104] The application layer can include a series of application packages, such as camera, gallery, calendar, map, navigation, etc. When these application packages are running, they can access the various service modules provided by the media platform framework layer through the application programming interface (API) and execute corresponding services.

[0105] In some embodiments, the camera is used to take photos, videos, slow-motion images, panoramic images, etc. in response to user operations. After these images are taken by the camera, or after the user triggers the electronic device to take a screenshot, or after the user triggers the electronic device to record the screen, or after the electronic device downloads images from other devices, the electronic device can save these images in a gallery, so that the user can perform video editing operations on the images in the gallery, such as smart cropping operations.

[0106] The embodiment of the present application divides the gallery into: a business layer, an application function layer and a basic function layer from top to bottom.

[0107] Among them, the business layer provides a variety of services such as one-click filming, wonderful moments, and free creation. These services are presented in the form of controls in the user interface (UI) of the gallery. Users can trigger the gallery to perform corresponding video processing actions by operating a certain control. For example, after the user selects one or more image materials and clicks on the free creation control of the gallery, one or more image materials can be intelligently cropped, text processed, and special effects processed to obtain a created video. Among them, after the image material is intelligently cropped, the black edges in the created video caused by the inconsistent aspect ratio of the image material can be avoided. In addition, in the process of intelligent cropping of the image material, the image material can also be cropped in batches, and the cropping results can be previewed and displayed to achieve real-time preview of the intelligent cropping results and improve user experience. The aspect ratio can be the aspect ratio of the image material.

[0108] The application function layer includes a clipping framework. Each business in the business layer can call the clipping framework to provide clipping services for pictures and videos. Exemplarily, the clipping framework may include functional modules such as intelligent cropping, text, and special effects, wherein intelligent cropping includes configuration modules such as specifying the subject and the aspect ratio, the specified subject is used to specify the subject in the cropped image material, and the aspect ratio is used to specify the aspect ratio of the cropped image material. The text module is used to add a text description to the subject in the image material. The special effects module is used to add beautification effects to the subject in the image material, such as adjusting the brightness of the subject, beautifying the face of the subject (such as a person), etc. Exemplarily, the intelligent cropping module is used to detect the subject in the image material, generate the position information of the cropping frame according to the subject position and the selected aspect ratio, and crop the image material according to the position information of the cropping frame.

[0109] The basic function layer is used to perform basic function processing on the edited video clips after the editing framework edits multiple image materials. Exemplarily, the basic function layer may include basic function modules such as video parsing, synthesis preservation, video scaling, and rendering. Among them, video parsing is used to parse the image materials to obtain image frame data. For example, the data of the streaming media protocol is parsed into standard corresponding encapsulation format data, and the image frame data is obtained from the encapsulation format data. When the video is transmitted on the network, various streaming media protocols are often used, such as Hyper Text Transfer Protocol (HTTP), Real Time Messaging Protocol (RTMR), etc. These protocols transmit some signaling data while transmitting video data. These signaling data include playback control (play, pause, stop), or description of network status, etc. In the parsing process, the signaling data will be removed and only the video data will be retained. For example, the data transmitted using the RTMP protocol outputs the data of the flash video format (FLash Video, FLV) after the protocol decomposition operation, and the image frame data is obtained from the FLV data. Synthetic preservation is used to cache the image frame data obtained by parsing, or to store the cropping frame position determined after cropping, the cropped image material, etc. For example, synthetic preservation can cache the image frame data obtained by parsing in batches. For example, synthetic preservation can divide the image frame data obtained by parsing into multiple batches of image frame data according to certain rules, obtain a batch of image frame data from these multiple batches of image frame data in turn and cache them, send the cached batch of image frame data to the media middle platform framework layer for processing, and then obtain the next batch of image frame data. Video scaling is used to scale the image material obtained after cropping so that the cropped image material can completely fill the entire display screen. The rendering module is used to render the image material obtained after cropping to preview and display the cropped image material. In addition, the basic function layer may also include video effects, which are used to add video effects to the image material obtained after cropping, such as adding style filters to the image material.

[0110] The media middle platform framework layer, the smart middle platform framework layer and the smart engine framework layer are software layers set between the application layer and the application framework layer. The media middle platform framework layer may include a decoding module, a frame cropping algorithm interface and an encoding module. The decoding module is used to decode the image frame data, such as sampling, downsampling and reducing the resolution of the image frame data. For example, the decoding module is used to obtain the image frame data cached by the merged storage module, sample the image frame data, obtain the sampled frame data, downsample and reduce the resolution of the sampled frame data, and obtain the processed sampled frame data. In addition, the decoding module can also be used to decode the video compression encoding data into uncompressed video raw data. Through decoding, the compressed encoded video data is output as uncompressed color data, such as color data (LuminanceChrominance, YUV), color data (Red Green Blue, RGB), etc. The frame cropping algorithm interface is used to call the frame cropping algorithm of the smart engine framework layer according to the image frame data, and the image frame data is cropped by the frame cropping algorithm. For example, the frame cropping algorithm of the intelligent engine framework layer can be called according to the sampled frame data processed by the decoding module, and the processed sampled frame data can be cropped by the frame cropping algorithm. The encoding module is used to encode the image frame data. For example, the cropping frame positions of some image frames in the image frame sequence can be interpolated to obtain the cropping frame positions of all image frames in the image frame sequence. For example, the cropping frame positions of the sampled frame data in the image frame sequence can be interpolated to obtain the cropping frame positions of all image frame data in the image frame sequence.

[0111] The smart middle platform framework layer includes a software development kit (SDK), which is a collection of related binary files, documents, examples, and tools that assist in the development of application software. The software development kit is used to call the algorithm modules in the smart engine framework layer to implement specific functions of the algorithm modules. For example, the frame cropping algorithm interface can call the frame cropping algorithm in the smart engine framework layer through the SDK in the smart middle platform framework layer.

[0112] The smart engine framework layer includes a cropping algorithm, which is used to crop the image frame according to a specific aspect ratio so that the aspect ratio of the image frame is the specific aspect ratio. The cropping algorithm may include algorithms such as subject detection, subject tracking, and subject segmentation. Subject detection is used to detect the subject in the image material, subject tracking is used to track the specified subject, and subject segmentation is used to generate the position information of the cropping frame for cropping the video subject.

[0113] The application framework layer, referred to as the framework layer for short, can be used to support the operation of various modules in the media middle platform framework layer and the smart middle platform framework layer.

[0114] Understandably, Figure 2 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components. For example, the media platform framework layer may also include a decapsulation module, an encapsulation module, etc. For another example, the software structure may also include a hardware abstraction layer, a system library, a kernel layer, etc., and the present application does not limit this.

[0115] The hardware abstraction layer is a package of the Linux kernel driver, providing an interface to the upper layer. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms. The kernel layer uses file descriptors to access files. File descriptors are non-negative integers. When opening an existing file or creating a new file, the kernel returns a file descriptor. Reading and writing files also requires the use of file descriptors to specify the files to be read and written.

[0116] It is understandable that in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0117] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0118] The following is combined with the above Figure 2 The software architecture shown briefly summarizes the image processing method provided in the embodiment of the present application.

[0119] In an embodiment of the present application, the free creation service is located in the business layer of the application layer, and is a sub-service of video editing. The image material to be cropped is selected in the business layer, and the image material is sent to the application function layer, and the application function layer sends the image material to the basic function layer. The basic function layer can send the image frames in the image material to the media middle platform framework layer in batches. For example, in the basic function layer, the image material can be parsed, and then the parsed image frames can be cached in batches, and the cached image frames can be sent to the media middle platform framework layer in sequence. In the application function layer, the free creation service is implemented based on the editing framework, which includes functional modules such as intelligent cropping, wherein intelligent cropping includes configuration modules such as specifying the subject and the aspect ratio. In the application function layer, the subject in the cropped image material and the aspect ratio of the cropped image material can be specified, and the specified subject information and the specified aspect ratio and other specified parameters can be generated, and then the specified parameters are sent to the media middle platform framework layer, such as the aspect cropping algorithm interface sent to the media middle platform framework layer.

[0120] The media platform framework layer is mainly used to send the image frame sequence and specified parameters sent by the upper layer to the smart platform framework layer. For example, the media platform framework layer can decode the image frame sequence sent by the basic function layer, generate a decoded image frame sequence, and then send the decoded image frame sequence and specified parameters to the smart platform framework layer through the frame cropping algorithm interface.

[0121] The smart middle platform framework layer calls the algorithm module in the smart engine framework layer through the SDK, and calculates the cropping frame position information of the image frame sequence through the algorithm module. For example, the algorithm module detects the subject in the image frame sequence, tracks the subject to obtain the subject's motion trajectory, and determines the cropping frame position information based on the motion trajectory and the specified frame ratio. Afterwards, the smart engine framework layer returns the calculated cropping frame position information to the smart middle platform framework layer, the smart middle platform framework layer returns it to the media middle platform framework layer, and finally returns it to the application layer. The application layer performs cropping processing, basic function processing, and preview display on the image frame sequence according to the cropping frame position information, thereby realizing real-time preview of the frame cropping results of the image material.

[0122] In another embodiment, a video editing preview pipeline can be constructed in the basic function layer of the application layer based on the capabilities of the basic function layer, and a frame cropping service pipeline can be constructed in the media middle platform framework layer based on the capabilities of the media middle platform framework layer, so as to achieve real-time preview of the frame cropping results of the image material through the video editing preview pipeline and the frame cropping service pipeline.

[0123] Please refer to Figure 3 , Figure 3 1 is a schematic diagram of the software structure of another electronic device 100 provided in an embodiment of the present application. Figure 3As shown, the software structure is divided from top to bottom into application layer, media middle platform framework layer, smart middle platform framework layer, smart engine framework layer, and application framework layer.

[0124] The application layer includes the gallery, which is divided into the business layer, application function layer and basic function layer from top to bottom. Figure 2 The relevant description of the embodiment of the present application will not be repeated here. The basic function layer includes a video clip preview pipeline built based on the capabilities of the basic function layer, and the video clip preview pipeline includes a parsing node, a cache node, a rendering node and a display node connected in sequence. The pipeline is a linear communication model and can be regarded as a pipeline. In short, it is a comprehensive solution for a system, providing a one-stop service for obtaining results. The parsing node is used to parse the image material sent by the application function layer. The cache node is used to obtain and cache the parsed image frame data in batches. For example, the parsed image frame data can be divided into multiple batches of image frame data according to certain rules, and a batch of image frame data is obtained from these multiple batches of image frame data in turn and cached, and the cached batch of image frame data is sent to the decoding node of the frame cropping service pipeline for processing, and then the next batch of image frame data is obtained. The rendering node is used to crop the image frame data according to the image frame data and the corresponding cropping frame position, and render the cropped image frame data. The display node is used to display the rendering results of the rendering node. In addition, the application function layer can also send specified parameters, such as specified frame ratio or specified subject, to the frame cropping algorithm node of the frame cropping business pipeline in the media middle platform framework layer through intelligent cropping.

[0125] Among them, the media middle platform framework layer includes a frame cropping business pipeline built based on the capabilities of the media middle platform framework layer, and the frame cropping business pipeline is located after the cache node and before the rendering node of the video clip preview pipeline. The frame cropping business pipeline includes a decoding node, a frame cropping algorithm node, and a frame interpolation node connected in sequence. The decoding node is used to decode a batch of image frame data sent by the cache node, such as sampling, downsampling, and reducing the resolution of the image frame data. The frame cropping algorithm node calls the frame cropping algorithm of the intelligent engine framework layer according to the decoded image frame data and specified parameters, and determines the cropping frame position information of the decoded image frame data through the frame cropping algorithm. The frame interpolation node is used to interpolate the cropping frame position information output by the frame cropping algorithm node to obtain a cropping frame position sequence. In addition, the basic function layer also includes an engineering data model, which can store the cropping frame position sequence output by the frame cropping business pipeline, so that it can be directly obtained from the engineering data model when the cropping frame position corresponding to the image frame needs to be obtained later.

[0126] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0127] The following is combined with the above Figure 3 The software architecture shown briefly summarizes the image processing method provided in the embodiment of the present application.

[0128] In the embodiment of the present application, the free creation service is located in the business layer of the application layer, which belongs to the sub-service of video editing. The image material to be cropped is selected in the business layer, and the image material is sent to the application function layer, such as the parsing node of the video clip preview pipeline sent to the basic function layer. In the application function layer, the free creation service is implemented based on the clipping framework, which includes functional modules such as intelligent cropping, wherein the intelligent cropping includes configuration modules such as the specified subject and the frame ratio. In the application function layer, the subject in the cropped image material and the frame ratio of the cropped image material can be specified, and the specified subject information and the specified frame ratio and other specified parameters can be generated, and then the specified parameters are sent to the media middle platform framework layer, such as the frame cropping algorithm node of the frame cropping service pipeline sent to the media middle platform framework layer. After confirming the execution of the cropping operation through the business layer, the parsing node in the video clip preview pipeline parses the image material to obtain the image frame. The cache node caches the parsed image frames in batches, and sequentially sends the cached image frame sequence to the decoding node of the frame cropping service pipeline for processing. For example, the cache node can divide the image frames obtained by parsing into multiple groups of image frame sequences according to certain rules, obtain a group of image frame sequences from the multiple groups of image frame sequences in turn and cache them, send the cached group of image frame sequences to the decoding node of the frame cropping service pipeline for processing, and then obtain the next group of image frame sequences. For example, the cache node can obtain the image frame sequence to be processed from the image frames obtained by parsing in turn according to the sliding time window and cache them, and send the cached image frame sequence to the decoding node for processing. The decoding node decodes the image frame sequence sent by the cache node, such as sampling multiple image frames in the image frame sequence to obtain sample frames, and downsampling and reducing the frame rate of the sample frames. The frame cropping algorithm node calls the frame cropping algorithm of the intelligent engine framework layer according to the sample frames output by the decoding node and the specified parameters, and determines the cropping frame position of the sample frame data through the frame cropping algorithm. The frame interpolation node interpolates according to the cropping frame position information of the sample frame data to obtain the cropping frame position information of multiple image frames in the image frame sequence, thereby obtaining a cropping frame position sequence. The frame interpolation node sends the cropping frame position sequence to the engineering data model of the basic function layer, and the engineering data model stores the cropping frame position sequence. The engineering data model sends the cropping frame position corresponding to the image frame data to the rendering node, and the rendering node crops the image frame data according to the cropping frame position corresponding to the image frame data, and renders the cropped image frame data. The display node then displays the rendering result of the rendering node, thereby realizing batch processing of image materials and real-time preview of the cropping results of the batch processing.

[0129] The execution subject of the image processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the image processing method, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. The following takes an electronic device as an example, and combines the accompanying drawings to exemplarily illustrate the image processing method provided in the embodiment of the present application.

[0130] The technical problems to be solved by this application and the inventive concept of this application are introduced below.

[0131] Currently, in order to make the images or videos taken by electronic devices display better, users can import image materials into electronic devices or image processing software provided by third parties for processing, such as splicing multiple videos into one video through image processing software. However, when multiple image materials with different aspect ratios are imported into image processing software for splicing, black edges may appear in the processed image materials that are finally saved. For example, Figure 4 This is an example diagram of video processing provided in an embodiment of the present application. Figure 4 The video shown in Figure (a) is synthesized from two videos with different aspect ratios. The first video includes 3 image frames, and the second video includes 2 image frames. Due to the different aspect ratios of the first video and the second video, black edges appear in the second video in the processed image material.

[0132] In view of the above problems, the embodiment of the present application provides an image processing method. In order to avoid the appearance of black edges in the final processed video, the present application can crop the image material according to a uniform aspect ratio, that is, crop the image material so that the aspect ratio of the cropped image material is the same. For example, the image processing method provided in the embodiment of the present application can be used to obtain Figure 4 The processing result is shown in Figure (b). Figure 4 As can be seen from the video shown in (b), the image processing method provided in the embodiment of the present application can ensure that the aspect ratio of the processed image materials is the same, avoid the appearance of black edges in the processed video, and improve the user experience.

[0133] It should be noted that the black borders mentioned above may be videos or images with black borders on the top and bottom; or videos or images with black borders on the left and right; or videos or images with black borders on the top, bottom, left and right. For example, Figure 5 is an example diagram of a black-bordered screen provided in an embodiment of the present application. Figure 5 (a) refers to videos or images with black edges on the top and bottom. Figure 5 (b) refers to videos or images with black edges on the left and right. Figure 5 (c) in the figure refers to a video or image with black edges on the top, bottom, left, and right sides.

[0134] In the embodiment of the present application, the image material can also be cropped in batches according to the uniform frame ratio and the subject position in the image material. In this way, not only can the black edges in the final processed video be avoided, but also the subject loss or missing can be ensured, thereby improving the user experience.

[0135] It should be noted that the subject mentioned in the embodiments of the present application refers to the object that is emphasized in the video or image. For example, the subject can be a person, an animal, a landscape, etc., or a part of a person or an animal, such as a head or a hand, etc., which is not limited in the embodiments of the present application. The image material can contain only images, only videos, or both images and videos, which is not limited in the embodiments of the present application.

[0136] At present, after the image material is imported into an electronic device or image processing software provided by a third party for processing, the electronic device or image processing software needs to finish processing all the imported image materials before exporting and displaying the processed image materials, resulting in low flexibility in processing and display, and users cannot see the processing results in time, resulting in poor user experience. For example, after all the image materials are processed, users can see the exported and displayed image materials, and then judge whether the processing results meet the user's needs. If they do not meet the user's needs, the image materials need to be processed again, which may result in the need to process the image materials multiple times, wasting processing resources and having low processing efficiency.

[0137] For example, please refer to Figure 6 , Figure 6 It is a schematic diagram of a user interface for stitching multiple videos using an image processing software provided in the related art. Figure 6 The user interface shown in Figure (a) includes multiple videos and pictures in the gallery, and the user can Figure 6 In the user interface shown in FIG. (a), two videos are selected and then the stitching control is clicked. In response to the user's operation, the electronic device displays Figure 6 The user interface shown in FIG. (b) includes a save control. The user clicks the save control, and in response to the user's operation, the electronic device begins to splice and save the two selected videos, and displays Figure 6 The user interface shown in FIG. (c) includes a saving progress. After the two selected videos are stitched together and successfully saved, the display Figure 6The user interface shown in FIG. (d) in FIG. 1 includes a video icon of the processed video and a prompt message “Save successfully, click to play”. When the user clicks the video icon of the processed video, the electronic device can play the processed video in response to the user’s operation and display the video. Figure 6 The playback interface of the processed video shown in Figure (e).

[0138] Depend on Figure 6 It can be seen that the image processing software needs to save the processed image materials locally after processing all the image materials selected by the user. After the user performs a play operation on the processed image materials saved locally, the electronic device can play the processed image materials, and then the user can see the processed image materials. Therefore, the flexibility of processing and displaying image materials is low, the user cannot see the processing results in time, and the user experience is poor.

[0139] In the embodiment of the present application, on the basis of avoiding the occurrence of black edges in the processed video, in order to improve the flexibility of processing and displaying the image material and ensure that the user can see the image processing results in time, the image material can be cropped in batches according to a unified aspect ratio, and after any batch of image frame sequences in the image material is processed, the processed image frame sequence with the aspect ratio is previewed and displayed, such as displaying a preview video of the processed image frame sequence with the aspect ratio. In this way, the image material can be cropped while the cropping result is previewed and displayed, realizing a real-time preview of the cropping result, thereby improving the flexibility of processing and displaying the image material, ensuring that the user can see the cropping result in time, and improving the user experience.

[0140] the following Figures 7 to 11 It is a set of example diagrams of a graphical user interface (GUI) provided in an embodiment of the present application.

[0141] In one application scenario, for example, a user uses an electronic device to pre-shoot multiple image materials, and the electronic device stores the multiple image materials in a gallery. Figure 7 As shown in Figure (a) of FIG. 1 , an icon of a gallery is displayed on the desktop of the electronic device. When the user wants the electronic device to perform frame cropping based on the image material, the user can click the icon of the gallery on the desktop of the electronic device. In response to the user's click operation on the icon, the electronic device displays the following Figure 7 The gallery interface shown in Figure (b) in FIG. 1 provides a "free creation" option, such as Figure 7 As shown in Figure (c) of FIG. 1 , the user can click on the “free creation” option. In response to the user clicking on the “free creation” option, the electronic device displays the following Figure 7The gallery interface shown in Figure (d) in FIG. 1 may include multiple image materials taken recently, such as multiple image materials taken on October 1, 2020. Figure 7 As shown in FIG. 5( e ), the user can click on any one of the multiple image materials. In response to the user's click operation on the any one of the image materials, the electronic device displays the following Figure 7 The gallery interface shown in Figure (f) includes all the materials in the gallery. Figure 7 Select the image material to be cropped in the gallery interface shown in Figure (f), for example Figure 7 As shown in Figure (f) in , assume that the user selects 3 image materials. Figure 7 The gallery interface shown in Figure (f) also includes confirmation options, such as Figure 7 As shown in Figure (f), the confirmation option is checked.

[0142] like Figure 8 As shown in FIG. 1 (a), after selecting the image material to be cropped, the user can click a confirmation option, such as clicking a check mark. In response to the user's click operation on the check mark option, the electronic device displays the following Figure 8 The gallery interface shown in FIG. (b) includes a "clip" option. When the user wants to perform intelligent cropping on the image material to be cropped, the user can click the "clip" option in the gallery interface. In response to the user's click operation on the "clip" option, the electronic device displays the following Figure 8 The gallery interface shown in FIG. (c) includes a "smart crop" option, and the user can click the "smart crop" option. In response to the user clicking the "smart crop" option, the electronic device displays the following Figure 8 The gallery interface shown in Figure (d) includes multiple different aspect ratio options, such as "9:16", "16:9", "1:1", "4:3", "3:4", etc. Users can Figure 8 In the gallery interface shown in Figure (d), select the aspect ratio. For example, if the user selects the "16:9" option, Figure 8 The gallery interface shown in Figure (d) also includes a cancel option, for example, the cancel option is a cross, and the user clicks the cross option to cancel the selected aspect ratio. The electronic device uses the preset aspect ratio when cropping. Figure 8 The gallery interface shown in Figure (d) also includes a confirmation option, such as a check mark. Figure 8 As shown in Figure (d) in FIG. 1 , the user can click the check mark option after selecting the aspect ratio “9:16” option. In response to the user’s click operation on the check mark option, the electronic device displays the following Figure 8The gallery interface shown in Figure (e) in FIG. 1 includes an option to specify a subject, such as a check mark, and an option not to specify a subject, such as a cross. The user can Figure 8 In the gallery interface shown in FIG. (e), a user selects a designated subject option, for example, the user clicks the "check mark" option. In response to the user's click operation on the "check mark" option, the electronic device displays the following Figure 8 In the gallery interface shown in Figure (f), the user can specify the subject by manually selecting the subject in the gallery interface, such as Figure 8 In the gallery interface shown in Figure (f), the user can manually select the entire man as the designated subject in the preview box. Figure 8 The gallery interface shown in Figure (f) also includes a "reset" option. When the user is not satisfied with the currently selected subject or wants to reselect the subject, the user can click the "reset" option. The gallery interface also includes a confirmation option, such as a check mark. When the user has determined that the currently selected subject is the specified subject, the user can click the check mark.

[0143] In response to the user clicking the check mark, the electronic device begins to crop the image material in batches according to the preset or specified aspect ratio and the position information of the specified subject in the image material. During the cropping process of the image material in batches, the electronic device may display the following information: Fig. 9 The preview interface shown in Figure (a) includes a preview frame, a play button, and a preview progress bar. The preview frame is used to display the part of the image material that has been cropped, and the preview progress bar is used to control the preview progress of the image material that has been cropped. The user can click Fig. 9 In response to the user clicking the play button in FIG. (a), the electronic device displays the following Fig. 9 The preview interface shown in Figure (b) in the figure is displayed, and the part of the image material that has been cropped is displayed in the preview box of the preview interface. The user can drag Fig. 9 The preview progress bar in (b) of FIG. 1 is used to adjust the preview progress. For example, dragging the preview progress bar forward adjusts the preview progress forward, and dragging the preview progress bar backward adjusts the preview progress backward, so that the user can flexibly view the image material at any time point after the cropping process is completed. Fig. 9 As shown in FIG. 1( b), the user can drag the preview progress bar backwards. In response to the user dragging the preview progress bar backwards, the electronic device displays the following Fig. 9 The preview interface is shown in Figure (c), and the image material after the current preview progress is displayed on the preview interface.

[0144] In addition, you can also limit the dragging range of the preview progress bar, for example, limit the preview progress bar to be dragged only within the preview progress range corresponding to the part of the image material that has been cropped, that is, limit the preview progress bar to be dragged only within the preview progress range. In this way, you can avoid the user dragging the preview progress bar to the preview progress corresponding to the part of the image material that has not been cropped, causing the preview box to be unable to display the preview screen or display an error, thereby improving the user experience.

[0145] In addition, after the electronic device completes the cropping of all image materials, it can automatically export the processed video and store the video in the gallery, so that the user can view the video clip from the gallery. Figure 8 As shown in Figure (a), the user can also click on the "Export" option. In response to the user's click on the video export option, the electronic device exports the generated video after all the image materials are cropped and processed, and stores the video in the gallery. In this way, the user can view the video from the gallery.

[0146] In another example, a user Figure 8 After selecting the aspect ratio in the gallery interface shown in Figure (d) and clicking the check mark option, the electronic device will display the following Fig.10 The gallery interface shown in (a) of FIG. 1 includes multiple different subject options, such as a "head" option, a "body" option, a "hand" option, a "flower and grass" option, etc. The user can Fig.10 In the designated subject in the gallery interface shown in (a), for example, the user can click the "head" option to select the "head" as the designated subject. Fig.10 Figure (a) in FIG. 1 also includes a cancel option, such as a cross. When the user does not want to specify a subject, the user can click the cross. For example, Fig.10 As shown in Figure (b), the user can cancel the specified subject by clicking the "cross" option. Fig.10 Figure (a) in FIG. 1 also includes a confirmation option, such as a check mark. When the user has determined that the currently selected "head" is the specified subject, the user can click the check mark, such as Fig.10 As shown in Figure (c), the user clicks the "check mark" option, and the electronic device responds to the user's click of the check mark operation, and starts to detect the subject in the image material, matches the detected subject with the specified subject, and determines whether the specified subject exists in the detected subject. If it is determined that the detected subject is the specified subject, then the cropping frame is determined according to the position information of the detected subject and the aspect ratio selected by the user, and the image material is cropped according to the cropping frame. Fig.10In Figure (c), after the electronic device has cropped the image material, the user can click the "Export" option. In response to the user's click operation on the video export option, the electronic device exports the video clip and stores the video clip in the gallery. In this way, the user can view the video clip from the gallery.

[0147] In another example, the user may not specify a subject, such as when the user clicks Figure 8 In the gallery interface shown in FIG. (e), in response to a user clicking the “cross” option, the electronic device begins to crop the image materials in batches according to a preset or specified aspect ratio and displays the image materials. Fig. 9 The preview interface is shown.

[0148] In one example, after the electronic device crops the image material according to a uniform aspect ratio, it may also scale the cropped image material so that the cropped image material can completely fill the entire display screen.

[0149] In one example, Figure 8 The gallery interface displayed in (b) or (c) also includes other functional options, which may include but are not limited to filter, music, text, special effects, picture-in-picture, etc. After the electronic device crops the image material to generate a video, the user can perform operations such as filter processing, add music, add text, add special effects, etc. on the generated video based on these functional options.

[0150] Fig.11 1 is a set of example diagrams of GUI provided in an embodiment of the present application. In another application scenario, for example, a user uses an electronic device to pre-shoot multiple image materials, and the electronic device stores the multiple image materials in a gallery. In this way, the user can select image materials in the gallery, so that the electronic device generates a video clip without black edges based on the image materials.

[0151] For example, Fig.11 As shown in Figure (a) of FIG. 1 , after entering the gallery interface, the gallery interface includes multiple image materials that have been taken recently. The user can click on any one of the multiple image materials, such as the first image material. In response to the user's click operation on the any one material, the electronic device displays the following Fig.11 The gallery interface shown in (b) includes all the image materials in the gallery. Fig.11 Select the image material to be cropped in the gallery interface shown in (b) of Figure 1. Fig.11 As shown in Figure (b), it is assumed that the user selects 3 image materials to be cropped. Fig.11The gallery interface shown in (b) of FIG. 1 further includes a "Create" option. When the user wants to crop the image material to be cropped, the user can click the "Create" option. In response to the user clicking the "Create" option, the electronic device displays the following Fig.11 The gallery interface shown in (c) includes a "free creation" option, and the user can choose to click the "free creation" option to achieve cropping of the image material to be cropped.

[0152] In one example, after the user clicks on the "free creation" option, the electronic device responds to the user clicking on the "free creation" option and displays a gallery interface that can refer to Figure 8 The interfaces shown in Figures (b) to (f) in FIG. In this example, the user can click Figure 8 Click the "Clip" option shown in (b) in the figure, and then click Figure 8 The "Smart Crop" option shown in (c) in the figure, then Figure 8 Select the aspect ratio in (d) and click to confirm the option. For example, click the "16:9" option and click the check mark. Figure 8 In (e) of the , select the specified subject option, such as the check mark option, and then Figure 8 Specify the subject in (f) and click Confirm. For example, Figure 8 In the preview box in the gallery interface shown in (f), the subject is manually selected by framing the subject and the check mark option is clicked after the subject is specified. At this time, the electronic device begins to crop the image material according to the specified aspect ratio and the specified subject, and displays Fig. 9 The preview interface is shown.

[0153] In another example, after the user clicks on the "free creation" option, the electronic device responds to the user clicking on the "free creation" option and displays a gallery interface that can refer to Figure 8 Figures (b), (c), (d) and Fig.10 Figures (a) and (c) show the interfaces in Figure 1. In this example, the user can click Figure 8 Click the "Clip" option shown in (b) in the figure, and then click Figure 8 The "Smart Crop" option shown in (c) in the figure, then Figure 8 Select the aspect ratio in the (d) picture and click to confirm the option. For example, click the "16:9" option and click the check mark. Fig.10 In (a) of Figure 1, for example, the user can click the "head" option to select "head" as the specified subject, and then Fig.10In the figure (c) in the figure, click the confirmation option. For example, after the user clicks the "head" option, click the check option. At this time, the electronic device begins to crop the image material according to the specified aspect ratio and the specified subject, and displays Figure 8 The preview interface is shown.

[0154] It should be noted that in the above application scenarios, the user's operation of selecting the aspect ratio and the operation of specifying the subject are both optional operations.

[0155] Next, the cropping processing algorithm involved in the embodiment of the present application is described in detail.

[0156] In order to improve the cropping efficiency of image materials, realize real-time preview of cropping results, and avoid the loss of subjects in the cropped video, in the process of cropping image frames in the image material in batches, for any batch of image frames in the batch processing, the batch of image frames can be sampled to obtain sampling frames, and then the sampling frames can be subject detected to obtain subject position information of the sampling frames, and the cropping frame position information of the batch of image frames can be determined according to the subject position information of the sampling frames and the first frame ratio, and then each image frame is cropped according to the cropping frame position information of each image frame to obtain a first image frame (frame cropping result) corresponding to the batch of image frames one by one, and the frame ratio of the first image frame is the first frame ratio. The first frame ratio is the frame ratio of the first frame in the image material, a preset frame ratio, or a frame ratio selected from multiple frame ratio options.

[0157] Please refer to Fig.12 , Fig.12 FIG. 1 is a flow chart of cropping a batch of image frames processed in batches provided by an embodiment of the present application, such as Fig.12 As shown, the frame cutting process includes the following steps A1 to A5:

[0158] A1: sampling a plurality of image frames included in a first image frame sequence to obtain a plurality of sampled frames, the number of the plurality of sampled frames being less than the number of the plurality of image frames, and the first image frame sequence being any batch of image frames processed in batches in the image material.

[0159] As an example, multiple image frames may be sampled according to a preset sampling rule, wherein the preset sampling rule may be to extract n sampling frames every m sampling frames, where m and n are both positive integers and may be pre-set as required.

[0160] As an example, multiple image frames may be grouped to obtain multiple groups of image frames, and then each group of sampling frames in the multiple groups of image frames may be sampled to extract at least one sampling frame from each group of sampling frames, thereby obtaining multiple sampling frames.

[0161] In one example, a sliding time window may be used to slide on image frames in the image material in sequence, and a sequence of image frames to be processed may be extracted from the image material in sequence according to the sliding time window as the first sequence of image frames.

[0162] The sliding time window may be preset, for example, the sliding time window may be 3 seconds, so as to sequentially obtain 3 seconds of image frames from the image material. The duration of the image frame may be determined according to the frame rate of the image frame, and the frame rate may be the transmission frame rate, the analysis frame rate, and the display frame rate of the image frame, etc., which is not limited in the embodiments of the present application.

[0163] In addition, considering that the original aspect ratio of some image frames in the image material is the first aspect ratio, in order to improve the efficiency of cropping processing, these image frames may not be cropped, but directly previewed. For example, in one example, the process of sliding the image frames in the image material in sequence using a sliding time window includes the following two implementation methods:

[0164] The first implementation method: if the sliding time window slides to the first continuous frame sequence whose aspect ratios in the image material are not the first aspect ratio, then according to the sliding time window, the image frame sequence to be processed is extracted from the first continuous image frame sequence in sequence as the first image frame sequence.

[0165] Among them, when there is an image frame before the first continuous image frame sequence, the aspect ratio of the previous image frame of the first continuous image frame sequence is the first aspect ratio, and when there is an image frame after the first continuous image frame sequence, the aspect ratio of the next image frame of the first continuous image frame sequence is the first aspect ratio.

[0166] The second implementation method: if the sliding time window slides to a second continuous frame sequence in which the aspect ratios of the image material are all the first aspect ratio, then according to the sliding time window, the image frame sequence is extracted from the second continuous image frame sequence in sequence, and the extracted image frame sequence is previewed and displayed.

[0167] Among them, when there is an image frame before the second continuous image frame sequence, the aspect ratio of the previous image frame of the second continuous image frame sequence is not the first aspect ratio, and when there is an image frame after the second continuous image frame sequence, the aspect ratio of the next image frame of the second continuous image frame sequence is not the first aspect ratio.

[0168] Please refer to Fig.13 , Fig.13 is an example diagram of a sliding time window sliding on an image material provided by an embodiment of the present application. Fig.13As shown, if the image material includes video 1 and video 2, the aspect ratios of the multiple image frames included in video 1 are all the first aspect ratio, and the aspect ratios of the multiple image frames included in video 2 are all the second aspect ratio, and the first aspect ratio and the second aspect ratio are different. Assuming that the sliding time window is 3 seconds, in the process of sliding the sliding time window on video 1 and video 2, if the sliding time window slides to video 1, a 3-second image frame sequence can be extracted from video 1 in sequence according to the sliding time window, and the extracted image frame sequence can be directly previewed and displayed. If the sliding time window slides to video 2, a 3-second image frame sequence can be extracted from video 2 in sequence according to the sliding time window, and the extracted image frame sequence is used as the first image frame sequence for cropping processing, and previewed after the first image frame sequence is processed. Among them, when the sliding time window slides to the end of video 1 or video 2, if the remaining image frames at the end are less than 3 seconds, the remaining image frames are directly extracted as the image frames within the sliding time window.

[0169] It should be noted that in the embodiment of the present application, there is no overlap in the image frame sequences acquired by adjacent sliding time windows, so that repeated processing of image frames can be avoided and processing efficiency can be improved.

[0170] A2: Perform subject detection on multiple sampling frames to obtain subject position information of each sampling frame in at least some of the multiple sampling frames, where the subject position information of each sampling frame is used to indicate the position of the target subject in each sampling frame in the corresponding sampling frame.

[0171] As an example, the following operations may be performed on each of the multiple sampling frames:

[0172] 1) Perform subject detection on a first sampling frame to obtain a subject detection result, where the first sampling frame is any one of the multiple sampling frames.

[0173] 2) When the subject detection result indicates that there is no subject in the first sampling frame, that is, no subject is detected during subject detection on the first sampling frame, it is determined that there is no target subject in the first sampling frame.

[0174] For a sample frame without a target subject, when subsequently determining its cropping frame position information, the cropping frame position information of the sample frame can be determined according to the position information of the preset position of the sample frame and the first frame ratio. The preset position can be the center position of the sample frame, etc., which is not limited in the embodiment of the present application.

[0175] 3) When the subject detection result includes the subject position information of at least one subject, determine the spatial dimension information and temporal dimension information of each subject in the at least one subject, determine the target subject from the at least one subject based on the spatial dimension information and temporal dimension information of the at least one subject, and use the position information of the target subject as the subject position information of the first sampling frame.

[0176] The spatial dimension information includes the area ratio and center distance of the corresponding subject. The area ratio refers to the ratio between the area of ​​the corresponding subject and the image area of ​​the first sampling frame, and the center distance refers to the distance between the corresponding subject and the center of the picture of the first sampling frame. In addition, the center distance can also be replaced by the edge distance, which refers to the distance between the corresponding subject and the edge of the picture of the first sampling frame.

[0177] The time dimension information includes the appearance duration information of the corresponding subject in the first image frame sequence, such as the appearance duration information of the corresponding subject in the sliding time window. The appearance duration information can be the appearance duration of the corresponding subject in the first image frame sequence, or can be the ratio between the appearance duration of the corresponding subject in the first image frame sequence and the total duration of the first image frame sequence. As an example, the appearance duration of the corresponding subject in the first image frame sequence can be represented by the appearance duration of the corresponding subject in multiple sampling frames.

[0178] By determining the target subject from at least one subject based on the spatial dimension information and temporal dimension information of at least one subject in the sampling frame, the accuracy of determining the significant subject in the sampling frame can be improved, thereby improving the accuracy of frame cropping and avoiding the situation where the subject is lost in the frame cropping result.

[0179] As an example, determining the target subject from at least one subject according to the spatial dimension information and the temporal dimension information of at least one subject includes the following two possible implementation situations:

[0180] A first possible implementation scenario: if there is a subject whose appearance duration information meets the first preset condition among at least one subject, the subject whose appearance duration information meets the first preset condition is determined as the target subject.

[0181] Among them, the first preset condition may be that the appearance duration in the first image frame sequence is greater than the first preset duration, or the appearance duration ratio is greater than the first preset ratio, and the appearance duration ratio refers to the ratio between the appearance duration of the corresponding subject in the first sampling frame and the total duration of the first sampling frame. Of course, the first preset condition may also be other conditions related to the appearance duration, which is not limited in the embodiment of the present application.

[0182] Among them, the first preset ratio and the first preset duration can be preset, for example, the first preset ratio can be 50% or 60%, etc., which is not limited in the embodiment of the present application.

[0183] By identifying a subject whose appearance duration information meets the first preset condition as a target subject, a subject that frequently appears in the first image frame sequence can be identified as a target subject, which facilitates subsequent frame cropping based on the frequently appearing subjects, so that the frequently appearing subjects can be included in the frame cropping result, thereby improving the accuracy of frame cropping.

[0184] A second possible implementation scenario: if there is no subject in at least one subject whose appearance duration information meets the first preset condition, a target subject is determined from at least one subject based on the area ratio and center distance of each subject in at least one subject.

[0185] In one example, in the case of a single subject, that is, in the case where at least one subject includes one subject, the subject can be directly determined as the target subject, or the target subject can be determined based on the spatial dimension information and time dimension information of the subject.

[0186] As an example, determining the target subject according to the spatial dimension information and the temporal dimension information of the subject may include: if the area ratio of the subject is greater than or equal to a second preset ratio, and / or the center distance of the subject is less than or equal to a first distance threshold, then determining the subject as the target subject. If the area ratio of the subject is less than the second preset ratio, and the center distance of the subject is greater than the first distance threshold, then determining that there is no target subject in the first sampling frame.

[0187] In another example, in the case of multiple subjects, that is, in the case where at least one subject includes multiple subjects, determining the target subject from the multiple subjects according to the spatial dimension information and the time dimension information of the multiple subjects may include the following cases:

[0188] The first case: When multiple subjects are close to the center of the picture, the subject with the largest area is identified as the target subject.

[0189] For example, if the maximum distance difference between the center distances of multiple subjects is less than or equal to the distance difference threshold, it is determined that the multiple subjects are close to the screen, and the subject whose area ratio meets the second preset condition among the multiple subjects is determined as the target subject.

[0190] The second preset condition may be that the area ratio is greater than the third preset ratio, or that the first t entities are sorted in descending order of area ratio. t is a positive integer, and t can be preset, for example, t is 1.

[0191] Please refer to Fig.14 , Fig.14is a schematic diagram of a sampling frame provided in an embodiment of the present application. Fig.14 As shown in FIG. 1 , the original aspect ratio of the sample frame is 16:9 (aspect ratio), the first aspect ratio is 9:16, and the sample frame includes two subjects: subject 1 and subject 2. The two subjects are close to the center of the picture, so subject 1 with a larger area can be identified as the target subject, so that the cropping frame with an aspect ratio of 9:16 can be determined based on the first aspect ratio and subject 1.

[0192] The second case: The area proportions of multiple subjects are similar, and the subject with a small center distance is identified as the target subject.

[0193] For example, if the maximum area difference between the area proportions of multiple subjects is less than or equal to the area difference threshold, the area proportions of the multiple subjects are determined to be similar, and at least one subject whose center distance meets the third preset condition is determined as the target subject.

[0194] The third preset condition may be that the center distance is less than the second distance threshold, or that the first s entities are sorted in ascending order of center distance. s is a positive integer, and s may be preset, for example, s is 1.

[0195] Please refer to Fig.15 , Fig.15 is a schematic diagram of another sampling frame provided in an embodiment of the present application. Fig.15 As shown, the original aspect ratio of the sample frame is 16:9, the first aspect ratio is 9:16, and the sample frame includes two subjects: subject 3 and subject 4. The area ratios of the two subjects are similar, so subject 3 with a smaller center distance can be identified as the target subject, so that a cropping frame with an aspect ratio of 9:16 can be determined according to the first aspect ratio and subject 3.

[0196] The third case: there is no obvious subject (the area of ​​multiple subjects is relatively small and they are far away from the center of the picture), and it is determined that there is no target subject in the first sampling frame.

[0197] For example, if the area proportions of the multiple subjects are all smaller than the second preset ratio, and the center distances of the multiple subjects are all greater than the first distance threshold, it is determined that there is no obvious subject in the first sampling frame, and further determined that there is no target subject in the first sampling frame.

[0198] For a sample frame without a target subject, when subsequently determining its cropping frame position information, the cropping frame position information of the sample frame can be determined according to the position information of the preset position of the sample frame and the first frame ratio. The preset position can be the center position of the sample frame, etc., which is not limited in the embodiment of the present application.

[0199] The above-mentioned preset ratio and distance threshold can be set in advance as needed. As an example, a parameter setting entrance can be provided for the user, and the user sets the above-mentioned preset ratio or distance threshold through the parameter setting entrance. For example, a parameter setting interface can be displayed, and the parameter setting interface includes a parameter setting entrance, and the parameter setting entrance is used to set the above-mentioned preset ratio or distance threshold.

[0200] Please refer to Fig.16 , Fig.16 is a schematic diagram of another sampling frame provided in an embodiment of the present application. Fig.16 As shown, the original aspect ratio of the sampling frame is 16:9, the first aspect ratio is 9:16, and the sampling frame includes 4 subjects. The area ratios of the 4 subjects are all smaller than the second preset ratio and the center distances are all greater than the first distance threshold. It is determined that the sampling frame has no target subject, so that a cropping frame with an aspect ratio of 9:16 can be subsequently determined according to the center position of the sampling frame and the first aspect ratio.

[0201] In addition, before performing subject detection on multiple sampling frames, each sampling frame may be subjected to resolution reduction processing to reduce the resolution of the sampling frame, obtain a sampling frame with a smaller resolution, and then subject detection is performed on the sampling frame after the resolution reduction processing. The resolution reduction processing may be downsampling, etc., which is not limited in the embodiment of the present application.

[0202] A3: Determine cropping frame position information of each of the multiple sampling frames according to the subject position information of at least part of the sampling frames and the first frame ratio.

[0203] In the embodiment of the present application, according to the subject position information and the first frame ratio of at least part of the sampled frames, determining the cropping frame position information of each of the multiple sampled frames may include the following three situations:

[0204] The first sampling frame is any one of the plurality of sampling frames, and the preset position can be the center position of the first sampling frame.

[0205] The first case: a single subject appears in multiple sampling frames.

[0206] When a single subject appears in multiple sampling frames, cropping frame position information of the multiple sampling frames may be determined according to the motion trajectory of the single subject and the first frame ratio, and the cropping frames of the multiple sampling frames include the motion trajectory of the single subject.

[0207] For example, if it is determined based on the subject position information of at least some of the sampling frames that the target subjects in at least some of the sampling frames are all the first target subject, or if the appearance duration of the first target subject in at least some of the sampling frames meets the fourth preset condition, it is determined that a single subject appears in multiple sampling frames, and then the first target subject is subject tracked based on the multiple sampling frames to obtain the motion trajectory of the first target subject, and then the cropping frame position information of the multiple sampling frames is determined based on the motion trajectory and the first frame ratio. Wherein, the cropping frames of the multiple sampling frames include the motion trajectory of the first target subject.

[0208] Among them, the fourth preset condition may be that the appearance duration is greater than the second preset duration, or that the ratio between the appearance duration of the first target subject in multiple sampling frames and the total duration of multiple sampling frames is greater than or equal to the fifth preset ratio. Of course, it may also be other preset conditions related to the appearance duration.

[0209] In one example, when the target subject is a person, the corresponding cropping frame includes the face of the person, so that the face of the person can be ensured to be included in the cropping range of the frame.

[0210] By determining the cropping frame position information of multiple sampling frames according to the motion trajectory of a single target and the first frame ratio, it can be ensured that the cropping frames of the multiple sampling frames can contain the motion trajectory of the single target, so that the frame cropping result can reflect the motion trajectory of the single target, thereby improving the accuracy of frame cropping and avoiding the loss of the subject.

[0211] As an example, in the process of determining the cropping frame position information of multiple sampling frames according to the motion trajectory of the first target subject and the first frame ratio, if the motion range of the first target subject is within the range of the cropping frame, the cropping frame position is not changed, and if the motion range exceeds the range of the cropping frame, the cropping frame position information is determined according to the current position information of the first target subject and the first frame ratio. In this way, the frame cropping result can reflect the motion trajectory of the target subject, and the smoothness of the frame cropping can be guaranteed to avoid severe shaking of the subject position.

[0212] For example, if the first target subject in the second sampling frame is within the first cropping frame, the position information of the first cropping frame is determined as the cropping frame position information of the second sampling frame, the position of the first cropping frame in the second sampling frame is the same as the position of the second cropping frame in the third sampling frame, the second cropping frame is the cropping frame of the third sampling frame, and the third sampling frame is the sampling frame in which the first target subject exists before the second sampling frame; wherein, if the third sampling frame is the first sampling frame in which the first target subject exists among multiple sampling frames, the position information of the second cropping frame is determined based on the position information of the first target subject in the third sampling frame and the first frame ratio. If the first target subject in the second sampling frame is not within the first cropping frame, the cropping frame position information of the second sampling frame is determined based on the position information of the first target subject in the second sampling frame and the first frame ratio.

[0213] In one example, the first target subject within the first cropping frame may include the first target subject being entirely within the first cropping frame, or most of the first target subject being within the first cropping frame, for example, a ratio of an area of ​​the first target subject within the first cropping frame to a total area of ​​the first target subject is greater than a fourth preset ratio.

[0214] Please refer to Fig.17 , Fig.17 Schematic diagram of the position of the cropping frame of a set of sampling frames provided in an embodiment of the present application. Fig.17 As shown, the group of sampling frames includes sampling frame 1, sampling frame 2, sampling frame 3 and sampling frame 4. The target subject in this group of sampling frames is a person, and the person is swinging on a swing. Fig.17 In the scene, if the motion range of the person swinging on the swing is within the range of the cropping frame, the position of the cropping frame is not changed. If the motion range exceeds the range of the cropping frame, the position information of the cropping frame is determined according to the position information of the target subject in the current frame and the first frame ratio. For example, cropping frame 1 is determined according to the position of the person in sampling frame 1 and the first frame ratio. If the motion range of the person in sampling frame 2 does not exceed cropping frame 2, which has the same position as cropping frame 1, cropping frame 2 is used as the cropping frame of sampling frame 2. Similarly, if the motion range of the person in sampling frame 3 does not exceed cropping frame 3, which has the same position as cropping frame 2, cropping frame 3 is used as the cropping frame of sampling frame 2. However, if the motion range of the person in sampling frame 4 exceeds cropping frame 4, which has the same position as cropping frame 3, cropping frame 4 is re-determined as the cropping frame of sampling frame 4 according to the position of the person in sampling frame 4 and the first frame ratio, so that the cropping frame moves smoothly with the target subject as the center.

[0215] As another example, in the process of determining cropping frame position information of multiple sampling frames according to the motion trajectory of the first target subject and the first frame ratio, if the first target subject disappears briefly and then recovers, the frame cropping position remains unchanged.

[0216] For example, if there is a fifth sampling frame including the first target subject after the fourth sampling frame including the first target subject, and the first target subject does not exist in the sampling frame between the fourth sampling frame and the fifth sampling frame, and the time interval between the fourth sampling frame and the fifth sampling frame is less than or equal to the duration threshold, then the cropping frame position information of the fifth sampling frame is determined according to the cropping frame position information of the fourth sampling frame, and the position of the cropping frame of the fourth sampling frame in the fourth sampling frame is the same as the position of the cropping frame of the fifth sampling frame in the fifth sampling frame.

[0217] Among them, the duration threshold can be pre-set, for example, it can be 1 second or 2 seconds, etc., and the embodiment of the present application does not limit this.

[0218] The second case: multiple sampling frames contain non-single subject.

[0219] In the case where non-single subjects appear in multiple sampling frames, for example, it is determined based on the subject position information of at least some of the sampling frames that the target subjects in at least some of the sampling frames are not the same target subject, or the appearance duration of the same target subject in at least some of the sampling frames does not meet the fourth preset condition, then the cropping frame position information of each sampling frame is determined based on the subject position information of each sampling frame in at least some of the sampling frames and the first frame ratio.

[0220] In addition, when multiple target subjects appear in a certain sampling frame, the cropping frame position information can also be determined based on the aggregation characteristics of the multiple target subjects. If the aggregation of the multiple target subjects meets the aggregation conditions, the cropping frame position information is determined based on the center position information of the multiple target subjects and the first frame ratio.

[0221] For example, at least some of the sampling frames include a sixth sampling frame, and there are multiple target subjects in the sixth sampling frame. If the subject aggregation of the multiple target subjects meets the fifth preset condition, the cropping frame position information of the sixth sampling frame is determined according to the center position information of the multiple target subjects and the first frame ratio; if the subject aggregation of the multiple target subjects does not meet the fifth preset condition, a significant subject is determined from the multiple target subjects according to the area ratio, center distance or appearance duration information of the multiple target subjects in multiple sampling frames, and the cropping frame position information of the sixth sampling frame is determined according to the subject position information of the significant subject and the first frame ratio.

[0222] In one example, the fifth threshold condition may be that a subject aggregation parameter is greater than a parameter threshold, where the subject aggregation parameter is used to indicate aggregation between subjects, and a larger subject aggregation parameter indicates greater aggregation between subjects.

[0223] The third case: there is no target subject in the sampling frame.

[0224] When there is no target subject in the first sampling frame, the cropping frame position information of the first sampling frame may be determined according to the position information of the preset position of the first sampling frame and the first frame ratio, and the cropping frame includes the preset position.

[0225] A4: Interpolate the cropping frame position information of the multiple sampling frames to obtain the cropping frame position information of the multiple image frames.

[0226] For example, the cropping frame position information of the plurality of sampling frames may be interpolated according to the number of the plurality of image frames to obtain the cropping frame position information of the plurality of image frames.

[0227] That is, the cropping frame position information of the other image frames except the multiple sampling frames can be determined by an interpolation algorithm based on the cropping frame position information of the multiple sampling frames, thereby obtaining the cropping frame position information of all the image frames in the multiple image frames.

[0228] In an example, the interpolation algorithm used to interpolate the cropping frame position information of multiple sampling frames may be a linear interpolation algorithm, etc., which is not limited in the embodiment of the present application.

[0229] A5: Crop each image frame according to the cropping frame position information of each image frame in the multiple image frames to obtain multiple first image frames corresponding to the multiple image frames one by one, and the aspect ratio of the first image frame is the first aspect ratio.

[0230] In one example, each image frame may be cropped according to the cropping frame position information of each image frame, and the cropped image frame may be scaled to obtain a first image frame corresponding to each image frame, so that the obtained first image frame can fill the entire display screen.

[0231] In another example, after subject detection is performed on multiple sampling frames to obtain subject position information of each sampling frame in at least some of the multiple sampling frames, interpolation can be performed based on the subject position information of at least some of the sampling frames to obtain subject position information of multiple image frames. Then, based on the subject position information of the multiple image frames and the first frame ratio, cropping frame position information of the multiple image frames is determined.

[0232] At present, when cropping image materials, cropping is usually performed at a fixed position of the image material, such as cropping at the center position of the image material, so that the aspect ratio of the cropped image material is consistent. However, the position of the subject in the image material is usually uncertain. For example, the subject in the video is located in the lower left corner of the video screen in one frame, and in the upper right corner of the video screen in another frame. If cropping is performed at the center position of the image material, the cropped image material may contain the subject, may not contain the subject, or may only contain part of the subject, resulting in the missing subject in the final processed image material, affecting the user experience.

[0233] In the embodiment of the present application, by performing subject detection on a batch of image frames to be processed during the process of cropping the image material in batches, and determining the cropping frame position information of the image frame based on the subject position information and the first frame ratio, the accuracy of frame cropping can be improved, the situation where the subject may be missing in the image material can be avoided, and the user experience can be improved. In addition, by first sampling a batch of image frames to be processed, and then performing subject detection on the sampled frames, and determining the cropping frame position information of all image frames based on the subject position information of the sampled frames and the first frame ratio, the efficiency of frame cropping can be improved, and the data source can be increased for realizing real-time preview of the frame cropping effect.

[0234] Please refer to Fig.18 , Fig.18 Schematic diagram of cropping image frames in a video in batches provided by an embodiment of the present application. Fig.18 As shown, the video includes multiple image frames, and a 3-second sliding time window can be used to extract image frames within 3 seconds from the video in sequence. Then, the image frames within 3 seconds are sampled. For example, the image frames within 3 seconds are divided into 3 groups, and the image frames within 1 second are divided into one group. Then, each group of image frames is sampled, for example, 2 frames of image frames are extracted from each group of image frames, so as to obtain 6 frames of image frames. After that, the sampled frames are positioned frame by frame through algorithms such as subject detection, subject tracking and subject segmentation to determine the cropping frame position information of the sampled frames, and linear interpolation is performed according to the cropping frame position information of the sampled frames to obtain the cropping frame position information of all image frames within these 3 seconds. Then, each image frame is cropped according to the cropping frame position information of each image frame to obtain the cropped image frame corresponding to each image frame.

[0235] It should be noted that in the process of cropping the image material in batches according to the first frame ratio in the embodiment of the present application, the cropping processes of different batches can be executed in parallel or in series, and the embodiment of the present application does not limit this. For example, after processing a batch of image frames, the next batch of image frames can be processed.

[0236] Next, combine Figure 2 In the software architecture, taking the first aspect ratio as the aspect ratio selected by the user as an example, the image processing method provided in the embodiment of the present application is described in detail.

[0237] Fig.19 is a schematic flow chart of an image processing method provided in an embodiment of the present application. The method can be executed by an electronic device, or by a processor or chip in the electronic device, and the present application embodiment does not make any limitation. For the convenience of description, the method is described in detail by taking an electronic device as an example. Fig.19 As shown, the method comprises the following steps:

[0238] S201. The business layer receives the user's operation on the free creation option.

[0239] It should be understood that the business layer includes a free creation application. The free creation application is correspondingly presented as a free creation option on the interface. When the user clicks the free creation option, the business layer will enable the free creation application.

[0240] S202: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0241] Exemplarily, a free creation option is displayed in the UI interface of the gallery application of the electronic device. When the user clicks on the free creation option, multiple pictures and multiple videos are displayed in the gallery interface, and a confirmation control for the free creation (such as a check mark) is displayed.

[0242] S203: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0243] In one example, after executing S202, the business layer can receive the image material to be cropped selected by the user and receive the user's confirmation to perform the free creation operation. Figure 6 In the gallery interface shown in Figure (c), click the Free Creation option. Figure 6 In the gallery interface shown in (f) of the figure, select one or more videos and one or more pictures as the image materials to be cropped, and then click the √ control to confirm the execution of the free creation operation. The business layer can receive the image materials selected by the user and receive the user's confirmation to execute the free creation operation.

[0244] In another example, the business layer may directly receive the image material to be cropped selected by the user and receive the user's confirmation to perform the free creation operation without executing steps S201 and S202. Fig.10In the gallery interface shown in (b) in FIG. 1 , select one or more videos and one or more pictures as the image materials to be cropped, then click the Create option and Fig.10 After clicking the free creation option in the gallery interface shown in (c), the business layer can receive the image material selected by the user and receive the user's confirmation to perform the free creation operation.

[0245] It should be understood that after the user confirms to execute free creation, the modules at different layers interact to complete the algorithm initialization so as to use the initialized algorithm to crop the image material. The embodiment of the present application does not limit the method of algorithm initialization.

[0246] S204. In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the application function layer.

[0247] S205: The application function layer displays the image material.

[0248] It should be understood that after the user clicks the √ control, the graphical interface displays the interface of the image processing software, for example, Figure 7 Figures (b) to (f) show the graphical interface. Figure 7 Figure (b) to Figure 7 The image material is displayed in the preview box in the graphical interface shown in Figure (f).

[0249] S206: The application function layer receives the user's operation on the smart cropping option and the operation of selecting the aspect ratio 1.

[0250] For example, after clicking the "Smart Crop" option, the user can select the aspect ratio to crop the image material. Figure 7 After selecting the "Smart Crop" option shown in (c) in the figure, Figure 7 Select the aspect ratio in the gallery interface of Figure (d) in the figure. For example, click the "16:9" option and then click the check mark option. Figure 7 Click the check mark option in the gallery interface (e).

[0251] S207: In response to the user's operation on the smart cropping option, the application function layer sends the image material to the basic function layer.

[0252] S208. In response to the user selecting the aspect ratio 1, the application function layer sends the selected aspect ratio 1 to the aspect ratio cropping algorithm interface of the media platform framework layer.

[0253] S209: The basic function layer analyzes the image material and generates an analysis file.

[0254] The parsed file includes image frames in the image material.

[0255] S210: The basic function layer obtains an image frame sequence within a sliding time window from the parsed file, and caches the image frame sequence.

[0256] S211. The basic function layer sends the cached image frame sequence to the decoding module of the media platform framework layer.

[0257] S212 . The decoding module samples multiple image frames 1 in the image frame sequence to obtain multiple sample frames 1 .

[0258] S213 . The decoding module downsamples each sampling frame 1 in the multiple sampling frames 1 to obtain multiple sampling frames 2 .

[0259] The plurality of sampling frames 2 correspond one to one with the plurality of sampling frames 1. By downsampling the sampling frame 1, a corresponding sampling frame 2 having a resolution smaller than that of the sampling frame 1 can be obtained.

[0260] S214. The decoding module sends a plurality of sampling frames 2 to the frame cropping algorithm interface.

[0261] S215, the frame cropping algorithm interface sends multiple sampling frames 2 and frame ratio 1 to the subject detection module in the intelligent engine framework layer through the software development kit in the smart middle platform framework layer.

[0262] For example, the frame cropping algorithm interface can send multiple sampling frames 2 and frame ratio 1 to the software development kit in the smart middle platform framework layer. After sending multiple sampling frames 2 and frame ratio 1 to the software development kit in the smart middle platform framework layer, the software development kit can call the algorithm module in the intelligent engine framework layer to crop the multiple sampling frames 2 according to the frame ratio 1, so that the frame ratios of the multiple sampling frames 2 are all the selected frame ratio 1.

[0263] S216. The subject detection module in the intelligent engine framework layer performs subject detection on multiple sampling frames 2 to obtain subject position information of at least some of the sampling frames 2.

[0264] The position information of the subject of the sampling frame 2 refers to the position information of the target subject in the sampling frame 2 in the image frame 2. Some of the multiple sampling frames 2 may not have the target subject, and thus these sampling frames 2 have no subject position information.

[0265] It should be understood that the subject detection module performs subject detection on multiple sampling frames 2, and obtaining subject position information of at least some of the sampling frames 2 can refer to the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0266] S217 . The subject detection module sends subject position information and frame ratio 1 of at least part of the sampling frame 2 to the subject tracking module and the subject segmentation module.

[0267] S218 , if the subject tracking module determines that there is a single target subject 1 in multiple sampling frames 2 according to the subject position information of at least part of the sampling frames 2 , the subject tracking module tracks the target subject 1 according to the multiple sampling frames 2 to obtain a motion trajectory of the target subject 1 .

[0268] Among them, the existence of a single target subject 1 in multiple sampling frames 2 means that based on the subject position information of at least part of the sampling frames 2, it is determined that the target subjects in at least part of the sampling frames 2 are all target subjects 1, or, based on the subject position information of at least part of the sampling frames 2, it is determined that the appearance duration information of the target subject 1 in multiple sampling frames 2 meets a preset condition, such as the appearance duration of the target subject 1 is greater than a duration threshold or the proportion of the appearance duration in the total duration of multiple sampling frames 2 is greater than a proportion threshold.

[0269] When there is a single target subject 1 in multiple sampling frames 2, the cropping frame position information of the multiple sampling frames 2 can be determined according to the motion trajectory of the target subject 1. Therefore, the target subject 1 can be first tracked in multiple sampling frames 2 through the subject tracking module to obtain the motion trajectory of the target subject 1.

[0270] S219. The subject tracking module sends the motion trajectory of the target subject 1 to the subject segmentation module.

[0271] S220 , when receiving the motion trajectory of the target subject 1 , the subject segmentation module determines the cropping frame position information of the plurality of sampling frames 2 according to the motion trajectory of the target subject 1 and the frame ratio 1 .

[0272] That is, when a single target subject 1 exists in multiple sampling frames 2 , the cropping frame position information of the multiple sampling frames 2 can be determined according to the motion trajectory of the target subject 1 and the frame ratio 1 .

[0273] It should be understood that the specific implementation of determining the cropping frame position information of multiple sampling frames 2 according to the motion trajectory of the target subject 1 and the frame ratio 1 can refer to the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0274] S221 , when the subject segmentation module does not receive the motion trajectory of the target subject 1 , the subject segmentation module determines the cropping frame position information of multiple sampling frames 2 according to the subject position information and the frame ratio 1 of at least part of the sampling frames 2 .

[0275] That is, when there is no single target subject 1 in multiple sampling frames 2, that is, when there are multiple target subjects, the cropping frame position information of the multiple sampling frames 2 can be determined based on the subject position information and the frame ratio 1 of at least part of the sampling frames 2. For each sampling frame 2 in at least part of the sampling frames 2, the cropping frame position information of each sampling frame 2 can be determined based on the subject position information and the frame ratio of each sampling frame 2, and the cropping frame of each sampling frame 2 includes the target subject in the corresponding sampling frame 2. For other sampling frames 2 in the multiple sampling frames 2 except at least part of the sampling frames 2, since these sampling frames 2 do not have target subjects, the cropping frame position information of the corresponding sampling frames can be determined based on the position information of the preset position in the corresponding sampling frame and the frame ratio 1. The preset position can be the center of the picture of the corresponding sampling frame, etc.

[0276] S222. The subject segmentation module sends the cropping box position information of the plurality of sampling frames 2 to the encoding module.

[0277] S223 . The encoding module performs linear interpolation according to the cropping frame position information of the plurality of sampling frames 2 to obtain the cropping frame position information of the image frame sequence.

[0278] That is, the cropping frame position information of the other sampling frames in the image frame sequence except the multiple sampling frames 2 can be generated according to the cropping frame position information of the multiple sampling frames 2 through linear interpolation. The cropping frame position information of these image frames can be directly obtained through interpolation without the need to be obtained through subject detection. Therefore, the efficiency of determining the cropping frame position information can be improved, thereby improving the frame cropping efficiency and preview efficiency.

[0279] S224. The encoding module sends the cropping frame position information of the image frame sequence to the basic function layer.

[0280] S225 . The basic function layer performs cropping processing on the image frame sequence according to the cropping frame position information of the image frame sequence to obtain multiple image frames 2 , and the aspect ratio of the multiple image frames 2 is aspect ratio 1 .

[0281] S226 , the basic function layer renders multiple image frames 2 .

[0282] S227, the basic function layer sends the rendered multiple image frames 2 for display.

[0283] S228. The business layer previews and displays multiple image frames 2.

[0284] In addition, after the basic function layer sends the cached image frame sequence to the decoding module of the media middle platform framework layer, it can also obtain the image frame sequence in the next sliding time window from the parsed file, cache the image frame sequence, and then crop the image frame sequence through S210-S228, and preview and display the image frame sequence with an aspect ratio of 1 obtained after cropping, until all the image frames in the parsed file are cropped and previewed.

[0285] It should be understood that after sending the cached image frame sequence to the decoding module of the media middle platform framework layer, the basic function layer can be triggered to obtain the image frame sequence in the next sliding time window from the parsed file to start cropping the image frame sequence in the next sliding time window. The basic function layer can also be triggered to obtain the image frame sequence in the next sliding time window from the parsed file at other times to start cropping the image frame sequence in the next sliding time window. For example, after determining the cropping frame position information of the image frame sequence, or after previewing and displaying an image frame 2, the embodiment of the present application does not limit the timing of triggering the basic function layer to obtain the image frame sequence in the next sliding time window from the parsed file.

[0286] It should be noted that the embodiment of the present application is only to illustrate the example of taking the image frame sequence acquired in each sliding time window as the image frame sequence to be cropped, while in other embodiments, if the original aspect ratio of the image frame sequence acquired in a certain time window is aspect ratio 1, the acquired image frame sequence may not be taken as the image frame sequence to be cropped, but the acquired image frame sequence may be directly previewed and displayed. For example, after acquiring such an image frame sequence, the basic function layer renders and displays the acquired image frame sequence, so that the business layer can preview and display these image frame sequences.

[0287] It should also be noted that the embodiments of the present application are only described by taking the user selecting the aspect ratio as an example. In other embodiments, if the user does not select the aspect ratio, the application function layer does not send the selected aspect ratio to the aspect cropping algorithm interface, and accordingly, the aspect cropping algorithm interface does not send the selected aspect ratio to the subject segmentation module. In this case, the subject segmentation module in the intelligent engine framework layer can determine the cropping frame position information of multiple sampling frames 2 according to the preset aspect ratio or the aspect ratio of the first frame image of the image material. Among them, the preset aspect ratio can be configured by the subject segmentation module during initialization, or it can be sent by other modules. The aspect ratio of the first frame image of the image material can be determined by the subject segmentation module, or it can be sent by other modules, and the embodiments of the present application do not limit this.

[0288] Next, combine Figure 3In the software architecture, taking the first aspect ratio as the aspect ratio selected by the user as an example, the image processing method provided in the embodiment of the present application is described in detail.

[0289] Fig. 20 is a schematic flow chart of an image processing method provided in an embodiment of the present application. The method can be executed by an electronic device, or by a processor or chip in the electronic device, and the present application embodiment does not make any limitation. For the convenience of description, the method is described in detail by taking an electronic device as an example. Fig. 20 As shown, the method comprises the following steps:

[0290] S301. The business layer receives the user's operation on the free creation option.

[0291] S302: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0292] S303: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0293] S304: In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the application function layer.

[0294] S305: The application function layer displays the image material.

[0295] S306: The application function layer receives the user's operation on the smart cropping option and the operation of selecting the aspect ratio 1.

[0296] S307: In response to the user's operation on the smart cropping option, the application function layer sends the image material to the parsing node of the video clip preview pipeline in the basic function layer.

[0297] S308. In response to the user selecting the aspect ratio 1, the application function layer sends the selected aspect ratio 1 to the aspect ratio algorithm node of the aspect ratio cropping service pipeline in the media platform framework layer.

[0298] S309: The parsing node parses the image material to generate a parsing file.

[0299] The parsed file includes image frames in the image material.

[0300] S310: The parsing node sends the parsing file to the cache node.

[0301] S311. The cache node obtains the image frame sequence within the sliding time window from the parsed file, and caches the image frame sequence.

[0302] S312: The cache node sends the cached image frame sequence to the decoding node of the frame cropping service pipeline.

[0303] S313 . The decoding node samples multiple image frames 1 in the image frame sequence to obtain multiple sample frames 1 .

[0304] S314 . The decoding node downsamples each sampling frame 1 in the multiple sampling frames to obtain multiple sampling frames 2 .

[0305] The plurality of sampling frames 2 correspond one to one with the plurality of sampling frames 1. By downsampling the sampling frame 1, a corresponding sampling frame 2 having a resolution smaller than that of the sampling frame 1 can be obtained.

[0306] S315. The decoding node sends multiple sampling frames 2 to the frame cropping algorithm node.

[0307] S316. The frame cropping algorithm node sends multiple sampling frames 2 and frame ratio 1 to the subject detection module in the intelligent engine framework layer.

[0308] The frame cropping algorithm node can send multiple sampling frames 2 and frame ratio 1 to the subject detection module through the software development kit in the smart middle platform framework layer.

[0309] For example, the frame cropping algorithm node sends multiple sampling frames 2 and frame ratio 1 to the software development kit in the smart middle platform framework layer. After sending multiple sampling frames 2 and frame ratio 1 to the software development kit in the smart middle platform framework layer, the software development kit calls the algorithm module in the intelligent engine framework layer to crop the multiple sampling frames 2 according to the frame ratio 1, so that the frame ratios of the multiple sampling frames 2 are all the selected frame ratios.

[0310] S317. The subject detection module in the intelligent engine framework layer performs subject detection on multiple sampling frames 2 to obtain subject position information of at least some of the sampling frames 2.

[0311] It should be understood that the subject detection module performs subject detection on multiple sampling frames 2, and obtaining subject position information of at least some of the sampling frames 2 can refer to the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0312] The position information of the subject of the sampling frame 2 refers to the position information of the target subject in the sampling frame 2 in the image frame 2 .

[0313] It should be understood that some of the multiple sampling frames 2 may not have a target subject, and thus these sampling frames 2 do not have subject position information.

[0314] S318. The subject detection module sends subject position information and frame ratio 1 of at least part of the sampling frame 2 to the subject segmentation module and the subject tracking module.

[0315] S319 , if the subject tracking module determines that there is a single target subject 1 in multiple sampling frames 2 according to the subject position information of at least part of the sampling frames 2 , the target subject 1 is tracked according to the multiple sampling frames 2 to obtain the motion trajectory of the target subject 1 .

[0316] S320. The subject tracking module sends the motion trajectory of the target subject 1 to the subject segmentation module.

[0317] S321 , when receiving the motion trajectory of the target subject 1 , the subject segmentation module determines the cropping frame position information of the plurality of sampling frames 2 according to the motion trajectory of the target subject 1 and the frame ratio 1 .

[0318] S322 , when the subject segmentation module does not receive the motion trajectory of the target subject 1 , the subject segmentation module determines the cropping frame position information of the plurality of sampling frames 2 according to the subject position information of the plurality of sampling frames 2 , at least part of the sampling frames 2 , and the frame ratio 1 .

[0319] S323. The subject segmentation module sends the cropping frame position information of the plurality of sampling frames 2 to the frame interpolation node.

[0320] S324 , the frame interpolation node performs linear interpolation according to the cropping frame position information of the plurality of sampling frames 2 to obtain the cropping frame position information of the image frame sequence, that is, the cropping frame position sequence.

[0321] S325: The frame interpolation node sends the cropping frame position information of the image frame sequence to the rendering node of the video clip preview pipeline in the basic function layer.

[0322] S326 . The rendering node crops the image frame sequence according to the cropping box position information of the image frame sequence to obtain multiple image frames 2 , and renders the multiple image frames 2 . The aspect ratio of the multiple image frames 2 is aspect ratio 1 .

[0323] S327, the rendering node sends the rendered multiple image frames 2 to the display sending node.

[0324] S328, the display sending node sends the multiple image frames 2 for display.

[0325] S329: The business layer previews and displays multiple image frames 2.

[0326] For example, a preview video of multiple image frames 2 is displayed.

[0327] In addition, after the basic function layer sends the cached image frame sequence to the decoding module of the media middle platform framework layer, it can also obtain the image frame sequence in the next sliding time window from the parsed file, cache the image frame sequence, and then crop the image frame sequence through S311-S329, and preview and display the image frame sequence with an aspect ratio of 1 obtained after cropping, until all the image frames in the parsed file are cropped and previewed.

[0328] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0329] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: In response to the user's first operation, loading and displaying videos and pictures in the gallery; In response to a user selecting an image material in the gallery and confirming to perform a cropping operation, the image materials are cropped in batches according to a first aspect ratio, so that the aspect ratios of the image materials are all the first aspect ratio; wherein the image materials include videos and / or pictures; In the process of cropping the image material in batches according to the first aspect ratio, after the processing of the first image frame sequence in the image material is completed, the first image frame sequence with the first aspect ratio obtained by processing is previewed and displayed, and the first image frame sequence is any batch of image frames processed in batches in the image material.

2. The method according to claim 1, characterized in that The cropping of the image materials in batches according to the first frame ratio includes: Sampling the plurality of image frames included in the first image frame sequence to obtain a plurality of sampling frames, wherein the number of the plurality of sampling frames is less than the number of the plurality of image frames; Performing subject detection on the multiple sampling frames to obtain subject position information of each sampling frame in at least some of the multiple sampling frames, wherein the subject position information of each sampling frame is used to indicate the position of the target subject in each sampling frame in the corresponding sampling frame; Determine, according to the subject position information of at least part of the sampling frames and the first frame ratio, cropping frame position information of each image frame in the multiple image frames, the cropping frame position information is used to indicate the position of the cropping frame in the corresponding image frame, and the frame ratio of the cropping frame is the first frame ratio; Each image frame is cropped according to the cropping frame position information of each image frame in the multiple image frames to obtain multiple first image frames corresponding to the multiple image frames one by one, and the aspect ratio of the first image frames is the first aspect ratio.

3. The method according to claim 2, characterized in that The performing subject detection on the multiple sampling frames to obtain subject position information of each sampling frame in at least some of the multiple sampling frames includes: Performing subject detection on a first sampling frame to obtain a subject detection result, wherein the first sampling frame is any one of the multiple sampling frames; In a case where the subject detection result includes subject position information of at least one subject, determining spatial dimension information and temporal dimension information of each subject in the at least one subject; wherein the spatial dimension information includes an area ratio and a center distance of the corresponding subject, the area ratio refers to a ratio between an area of ​​the corresponding subject and an image area of ​​the first sampling frame, the center distance refers to a distance between the corresponding subject and an image center of the first sampling frame, and the temporal dimension information includes appearance duration information of the corresponding subject in the first image frame sequence; According to the spatial dimension information and the temporal dimension information of the at least one subject, a target subject is determined from the at least one subject, and the position information of the target subject is used as the subject position information of the first sampling frame.

4. The method according to claim 3, characterized in that The step of determining a target subject from the at least one subject according to the spatial dimension information and the temporal dimension information of the at least one subject comprises: If there is a subject whose appearance duration information satisfies a first preset condition among the at least one subject, determining the subject whose appearance duration information satisfies the first preset condition as the target subject; If there is no subject whose appearance duration information satisfies the first preset condition in the at least one subject, the target subject is determined from the at least one subject according to the area ratio and center distance of each subject in the at least one subject.

5. The method according to claim 4, characterized in that The appearance duration information is the appearance duration ratio of the corresponding subject, and the appearance duration ratio refers to the ratio between the appearance duration of the corresponding subject in the first sampling frame and the total duration of the first sampling frame; The method further comprises: If the appearance duration ratio of the first subject is greater than the first preset ratio, it is determined that the appearance duration information of the first subject meets the first preset condition, and the first subject is any one of the at least one subject.

6. The method according to claim 4 or 5, characterized in that The at least one subject includes a plurality of subjects; and determining the target subject from the at least one subject according to the area ratio and the center distance of each subject in the at least one subject comprises: If the maximum distance difference between the center distances of the multiple subjects is less than or equal to the distance difference threshold, determining the subject whose area ratio meets the second preset condition among the multiple subjects as the target subject; If the maximum area difference between the area proportions of the multiple subjects is less than or equal to the area difference threshold, then the subject whose center distance among the at least one subject meets the third preset condition is determined as the target subject.

7. The method according to claim 4 or 5, characterized in that The at least one subject includes a subject; and determining the target subject from the at least one subject according to the area ratio and the center distance of each subject in the at least one subject, comprises: If the area ratio of the subject is greater than or equal to a second preset ratio, and / or the center distances of the subjects are less than or equal to a first distance threshold, the subject is determined as the target subject.

8. The method according to claim 6 or 7, characterized in that The method further comprises: If the area ratio of the at least one subject is smaller than the second preset ratio threshold, and the center distance of the at least one subject is greater than the first distance threshold, it is determined that there is no target subject in the first sampling frame.

9. The method according to any one of claims 2 to 8, characterized in that: The determining, according to the subject position information of at least part of the sampling frames and the first frame ratio, the cropping frame position information of each of the multiple image frames comprises: Determining cropping frame position information of the plurality of sampling frames according to the subject position information of at least part of the sampling frames and the first frame ratio; Interpolation is performed according to the cropping frame position information of the multiple sampling frames to obtain the cropping frame position information of the multiple image frames.

10. The method according to claim 9, characterized in that The determining, according to the subject position information of at least part of the sampling frames and the first frame ratio, the cropping frame position information of the plurality of sampling frames comprises: If it is determined according to the subject position information of at least some of the sampling frames that all the target subjects in the at least some of the sampling frames are the first target subject, or if the appearance duration of the first target subject in the at least some of the sampling frames meets a fourth preset condition, subject tracking is performed on the first target subject based on the multiple sampling frames to obtain a motion trajectory of the first target subject; The cropping frame position information of the plurality of sampling frames is determined according to the motion trajectory and the first frame ratio.

11. The method according to claim 10, characterized in that The step of determining the cropping frame position information of the plurality of sampling frames according to the motion trajectory and the first frame ratio includes: If the first target subject in the second sampling frame is within the first cropping frame, the position information of the first cropping frame is determined as the cropping frame position information of the second sampling frame, the position of the first cropping frame in the second sampling frame is the same as the position of the second cropping frame in the third sampling frame, the second cropping frame is the cropping frame of the third sampling frame, and the third sampling frame is a sampling frame in which the first target subject exists before the second sampling frame; wherein, if the third sampling frame is the first sampling frame in which the first target subject exists among the multiple sampling frames, the position information of the second cropping frame is determined according to the position information of the first target subject in the third sampling frame and the first frame ratio; If the first target subject in the second sampling frame is not within the first cropping frame, the cropping frame position information of the second sampling frame is determined according to the position information of the first target subject in the second sampling frame and the first frame ratio.

12. The method according to claim 11, characterized in that The method further comprises: If there is a fifth sampling frame including the first target subject after the fourth sampling frame including the first target subject, and the first target subject does not exist in the sampling frame between the fourth sampling frame and the fifth sampling frame, and the time interval between the fourth sampling frame and the fifth sampling frame is less than or equal to the duration threshold, then the cropping frame position information of the fifth sampling frame is determined according to the cropping frame position information of the fourth sampling frame, and the position of the cropping frame of the fourth sampling frame in the fourth sampling frame is the same as the position of the cropping frame of the fifth sampling frame in the fifth sampling frame.

13. The method according to any one of claims 9 to 12, characterized in that: The determining, according to the subject position information of at least part of the sampling frames and the first frame ratio, the cropping frame position information of the plurality of sampling frames comprises: If it is determined based on the subject position information of the at least part of the sampling frames that the target subjects in the at least part of the sampling frames are not the same target subject, or the appearance duration of the same target subject in the at least part of the sampling frames does not meet the fourth preset condition, then the cropping frame position information of each sampling frame in the at least part of the sampling frames is determined based on the subject position information of each sampling frame in the at least part of the sampling frames and the first frame ratio.

14. The method according to claim 13, characterized in that The at least part of the sampling frames includes a sixth sampling frame, and a plurality of target subjects exist in the sixth sampling frame; The step of determining the cropping frame position information of each sampling frame according to the subject position information of each sampling frame in the at least part of the sampling frames and the first frame ratio includes: If the subject aggregation of the multiple target subjects meets the fifth preset condition, determining the cropping frame position information of the sixth sampling frame according to the center position information of the multiple target subjects and the first frame ratio; If the subject aggregation of the multiple target subjects does not meet the fifth preset condition, then the significant subject is determined from the multiple target subjects according to the area ratio, center distance or appearance duration information of the multiple target subjects in the multiple sampling frames, and the cropping frame position information of the sixth sampling frame is determined according to the subject position information of the significant subject and the first frame ratio.

15. The method according to any one of claims 1 to 14, characterized in that: The method further comprises: According to the sliding time window, the image frame sequence to be processed is extracted from the image material in sequence as the first image frame sequence, the starting point of the sliding time window is the first image frame in the image material, and the first image frame sequence includes at least one image frame.

16. The method according to any one of claims 1 to 14, characterized in that: The method further comprises: Using a sliding time window to slide on the image frames in the image material in sequence; In the process of the sliding time window sliding on the image frames in the image material in sequence, if the sliding time window slides to a first continuous frame sequence in the image material, then according to the sliding time window, image frame sequences to be processed are extracted in sequence from the first continuous image frame sequence as the first image frame sequence, and the aspect ratios of the first continuous image frame sequences are not the first aspect ratio; Among them, when there is an image frame before the first continuous image frame sequence, the aspect ratio of the previous image frame of the first continuous image frame sequence is the first aspect ratio; when there is an image frame after the first continuous image frame sequence, the aspect ratio of the next image frame of the first continuous image frame sequence is the first aspect ratio.

17. The method according to claim 16, characterized in that The method further comprises: If the sliding time window slides to a second continuous frame sequence in the image material, image frame sequences are extracted from the second continuous image frame sequence in sequence according to the sliding time window, and the extracted image frame sequences are previewed and displayed, and the aspect ratios of the second continuous image frame sequences are all the first aspect ratio; Among them, when there is an image frame before the second continuous image frame sequence, the aspect ratio of the previous image frame of the second continuous image frame sequence is not the first aspect ratio; when there is an image frame after the second continuous image frame sequence, the aspect ratio of the next image frame of the second continuous image frame sequence is not the first aspect ratio.

18. The method according to any one of claims 1 to 17, characterized in that: Before the image materials are cropped in batches according to the first frame ratio, the method further includes: The image material is parsed through a parsing node in a preview pipeline to obtain an image frame in the image material, wherein the preview pipeline includes the parsing node, the rendering node and the display sending node connected in sequence; The cropping of the image materials in batches according to the first frame ratio includes: The image frames in the image material are cropped in batches through a frame cropping service pipeline, wherein the frame cropping service pipeline is located after the parsing node and before the rendering node; After the first image frame sequence in the image material is processed, previewing and displaying the first image frame sequence having the first aspect ratio obtained by the processing, comprises: After the first image frame sequence in the image material is processed by the frame cropping service pipeline, the first image frame sequence having the first frame ratio obtained by the processing is rendered by the rendering node; The rendered image frame sequence having the first aspect ratio is displayed through the display node, so as to preview and display the processed first image frame sequence having the first aspect ratio.

19. The method according to claim 18, characterized in that The preview pipeline further includes a cache node, which is located after the parsing node and before the rendering node, and the frame cropping service pipeline is located after the cache node and before the rendering node; After parsing the image material through the parsing node in the preview pipeline, the method further includes: Extracting a to-be-processed image frame sequence from the image frames in the image material in sequence according to the sliding time window as the first image frame sequence through the image frame cache node, and caching the extracted first image frame sequence; The cropping process of the image frames in the image material in batches through the frame cropping service pipeline includes: The first image frame sequence cached by the cache node is obtained through the frame cropping service pipeline, and each time a group of the first image frame sequences is obtained, the first image frame sequence is cropped so that the frame ratios of the first image frame sequences are all the first frame ratio.

20. The method according to any one of claims 1 to 19, characterized in that: In response to the user selecting an image material in the gallery and confirming to perform a cropping operation, the image material is cropped in batches according to a first frame ratio, including: In response to the user selecting an image material in the gallery and confirming to perform a cropping operation, displaying a first interface, wherein the first interface includes a plurality of aspect ratios; In response to the user selecting the first aspect ratio, the image materials are cropped in batches according to the first aspect ratio.

21. An electronic device, characterized in that: The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 20 when executed by the processor.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 20.

Images