Image processing method, electronic equipment, readable storage medium and chip

Abstract

The invention provides an image processing method, electronic equipment, a readable storage medium and a chip. According to the method, when the N to-be-cut image materials are cut, the picture proportions of the N image materials are all the first picture proportion according to the first picture proportion, black edges can be prevented from appearing in the finally stored processed image materials, and in addition, the image quality is improved. According to the embodiment of the invention, under the condition that main body detection is carried out on an image frame to obtain first position information used for indicating the position of a target main body in the image frame, second position information used for indicating the position of a first cutting frame in the image frame is obtained based on the first position information and a first picture proportion; the image frame is cut according to the second position information, the first image frame is obtained, the first image frame contains the target body due to the fact that the first cut frame contains the target body, it is guaranteed that the finally stored processed image material contains the target body, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of image data, and in particular to an image processing method, an electronic device, a readable storage medium and a chip. Background Art

[0002] At present, in order to make the images or videos taken by electronic devices display better, users can import image materials (image materials can be images, videos, or both) into electronic devices or image processing software provided by third parties for processing, for example, splicing the image materials. However, when multiple image materials with different aspect ratios are imported into the image processing software for processing, black edges may appear in the processed image materials that are finally saved.

[0003] In order to avoid the appearance of black edges in the processed image material that is finally saved, a common solution is to crop the image material at a fixed position of the imported image material so that the aspect ratios of multiple image materials with different aspect ratios imported into the image processing software are consistent.

[0004] However, the position of the subject in the image material imported into the image processing software is unconfirmed. After the image material is cropped at a fixed position, the fixed position may contain the subject, or may not contain the subject, or may only contain a part of the subject, resulting in the subject being missing or lost in the final saved processed image material, affecting the user experience. Summary of the invention

[0005] The present application provides an image processing method, an electronic device, a readable storage medium and a chip, which can avoid the appearance of black edges in the processed image material finally saved, while ensuring that the processed image material finally saved contains the subject, avoiding the absence or loss of the subject, and improving the user experience.

[0006] In a first aspect, an image processing method is provided, the method comprising: displaying a first interface including N image materials to be cropped, at least two of the N image materials having different aspect ratios, N being an integer greater than or equal to 2; in response to a user's operation of cropping the N image materials, processing each image frame of each image material in at least some of the N image materials according to a first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio, wherein when at least some of the image materials include some of the N image materials, the aspect ratios of the image materials other than the some of the N image materials are the first aspect ratio; displaying a second interface including the N image materials having the first aspect ratio;

[0007] The process of processing the image frame includes:

[0008] When subject detection is performed on an image frame to obtain first position information, second position information for indicating a position of a first cropping frame in the image frame is obtained based on the first position information and the first frame ratio, the first position information being used to indicate a position of a target subject in the image frame, the first cropping frame including the target subject, and the frame ratio of the first cropping frame being the first frame ratio; the image frame is cropped according to the second position information to obtain a first image frame, the first image frame including the target subject, and the frame ratio of the first image frame being the first frame ratio.

[0009] Usually, since at least two of the N image materials have different aspect ratios, black edges may appear in the spliced ​​N image materials. Therefore, in the embodiment of the present application, when cropping the N image materials to be cropped, each image frame of each image material in at least part of the N image materials may be processed so that the aspect ratios of the N image materials are the same, that is, the aspect ratios of the N image materials are all the first aspect ratio. Since the aspect ratios of the processed N image materials are all the same, black edges may be avoided in the processed image materials that are finally saved.

[0010] Furthermore, since the embodiment of the present application can perform subject detection on the image frame when cropping the N image materials to be cropped, when the position information of the target subject in the image frame can be detected, the position information of the first cropping frame is generated according to the position information of the target subject and the first frame ratio, and then the image frame is cropped according to the position information of the first cropping frame, that is, when the subject is detected on the image frame to obtain the first position information indicating the position of the target subject in the image frame, based on the first position information and the first frame ratio, the second position information indicating the position of the first cropping frame in the image frame is obtained, wherein the first cropping frame includes the target subject, and then the image frame is cropped according to the second position information to obtain the first image frame, wherein the first image frame includes the target subject. Therefore, compared with the cropping scheme that usually cropping the image material at a fixed position will result in missing or lost subject, since the first image frame includes the target subject, it is ensured that the processed image material finally saved includes the target subject, avoiding the occurrence of missing or lost subject, thereby improving the user experience.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the image frame includes multiple subjects; and the method also includes: performing subject detection on the image frame to obtain multiple third position information corresponding to the multiple subjects, each third position information is used to indicate the position of the corresponding subject in the image frame; based on the multiple third position information, the first position information is obtained, and the target subject includes at least part of the multiple subjects.

[0012] Typically, when an electronic device performs subject detection on an image frame, regardless of whether there are one or more subjects in the image frame, one position information is used to indicate the position of one or more subjects in the image frame, resulting in an inability to detect the position of each subject in the image frame when there are multiple subjects in the image frame. However, the electronic device in the embodiment of the present application performs subject detection on the image frame and can obtain multiple third position information corresponding to the multiple subjects. Compared with the conventional subject detection scheme, since the third position information of each of the multiple subjects in the image frame can be obtained during detection, the detection effect when performing subject detection on the image frame can be improved, and the third position information of each subject can be obtained more accurately. Therefore, when the electronic device obtains the first position information based on the multiple third position information, the accuracy of obtaining the first position information can be improved.

[0013] In combination with the first aspect, in some implementations of the first aspect, before obtaining the first location information based on the plurality of third location information, the method further includes:

[0014] Based on a plurality of subject information corresponding to the plurality of subjects one by one, scoring the plurality of subjects to obtain a plurality of scoring values ​​of the plurality of subjects, wherein one subject corresponds to one scoring value, and each subject information includes image feature information and image quality information of each subject; determining a subject corresponding to a scoring value that meets a preset condition among the plurality of scoring values ​​as a target subject, wherein the preset condition includes: the scoring value is greater than or equal to a preset scoring value;

[0015] And, based on the plurality of third position information, obtaining the first position information, the method comprises:

[0016] Based on the plurality of third position information, the third position information corresponding to each subject in the target subject is determined as the first position information.

[0017] Usually, when an electronic device determines a target subject among multiple subjects, it can randomly determine any subject among the multiple subjects as the target subject, which may result in a low image quality of the image frame finally displayed due to the low image quality at the position where the target subject is located in the image frame. In the embodiment of the present application, when the electronic device determines the target subject, it can score multiple subjects and determine the target subject according to the scoring results. Compared with the method of randomly determining the target subject among multiple subjects, the target subject can be accurately determined. In addition, the embodiment of the present application scores the subject based on the subject information (the subject information includes image feature information and image quality information), and determines the subject corresponding to the score value greater than or equal to the preset score value among multiple score values ​​as the target subject, which can improve the image quality at the position where the target subject is located in the image frame. Since the present application improves the image quality at the position where the target subject is located in the image frame, the image quality of the image frame finally displayed containing the target subject can be improved.

[0018] In combination with the first aspect, in some implementations of the first aspect, obtaining the first location information based on the plurality of third location information includes:

[0019] Any third position information among the plurality of third position information is determined as the first position information.

[0020] In combination with the first aspect, in some implementations of the first aspect, the first aspect ratio is a preset aspect ratio, and the preset aspect ratio is an aspect ratio of any image material among the N image materials.

[0021] In combination with the first aspect, in some implementations of the first aspect, in response to a user performing a cropping operation on N image materials, processing each image frame of each image material in at least part of the N image materials according to a first aspect ratio so that the aspect ratios of the N image materials are all the first aspect ratio includes:

[0022] In response to the user's operation on the smart cropping option of N image materials, a third interface is displayed, including multiple aspect ratios; in response to the user's operation of selecting a first aspect ratio, each frame of each image material in at least part of the N image materials is processed according to the first aspect ratio, so that the aspect ratio of the N image materials is the first aspect ratio.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: in a case where a first abnormality mark is obtained by performing subject detection on an image frame, fourth position information is obtained based on position information of any position of the image frame and the first aspect ratio, the fourth position information being used to indicate a position of a second cropping frame in the image frame, the first abnormality mark being used to indicate that a subject in the image frame is lost or missing, and the aspect ratio of the second cropping frame is the first aspect ratio; and the image frame is cropped according to the fourth position information to obtain a second image frame, and the aspect ratio of the second image frame is the first aspect ratio.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: performing subject detection on the image frame to obtain at least one candidate subject; matching the at least one candidate subject with a designated subject, the designated subject being a subject selected by the user when performing a cropping operation on N image materials; when the designated subject successfully matches at least part of the at least one candidate subject, determining the position of the designated subject in the image frame as first position information, wherein the designated subject is the target subject.

[0025] Usually, after an electronic device crops N image materials, it is unable to display the subject that the user wants to display in the cropped image materials, thereby reducing the user experience. However, in the embodiment of the present application, since the user can specify the subject (the specified subject is the subject that the user wants to display) when cropping N image materials, and then after subject detection is performed on the image frame to obtain at least one candidate subject, the at least one candidate subject is matched with the specified subject. When the specified subject successfully matches at least part of the at least one candidate subject, the position of the specified subject in the image frame is determined as the position information of the target subject in the image frame (i.e., the first position information). Subsequently, the image frame is processed according to the first position information, so that the subject that the user wants to display can be displayed in the cropped image material, thereby improving the user experience.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: performing subject detection on the image frame to obtain at least one candidate subject; matching the at least one candidate subject with a designated subject, the designated subject being a subject selected by the user when performing a cropping operation on N image materials; when the designated subject does not match any of the at least one candidate subject, obtaining a second exception mark; when performing subject detection on the image frame to obtain the second exception mark, obtaining fifth position information based on position information of any position of the image frame and the first aspect ratio, the fifth position information being used to indicate the position of a third cropping frame in the image frame, the aspect ratio of the third cropping frame being the first aspect ratio; cropping the image frame according to the fifth position information to obtain a third image frame, the aspect ratio of the third image frame being the first aspect ratio.

[0027] In combination with the first aspect, in some implementations of the first aspect, cropping the image frame according to the second position information to obtain the first image frame includes:

[0028] The image frame is cropped according to the second position information and the cropped image frame is scaled to obtain a first image frame.

[0029] In a second aspect, an electronic device is provided, and the electronic device is used to execute the method provided in the first aspect. Specifically, the electronic device may include a processing unit for executing any possible implementation manner of the first aspect.

[0030] In a third aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute a method in any possible implementation of the first aspect.

[0031] In a fourth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect.

[0032] In a fifth aspect, a chip is provided, comprising a memory for storing instructions; and a processor for calling and executing instructions from the memory, so that an electronic device equipped with the chip executes the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0034] Figure 2 It is a schematic diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0035] Figure 3 This is an example diagram of video processing provided in an embodiment of the present application.

[0036] Figure 4 This is an example diagram of a black-bordered screen provided in an embodiment of the present application.

[0037] Figure 5 It is a set of graphical user interfaces provided by the embodiments of the present application.

[0038] Figure 6 It is a set of graphical user interfaces provided by the embodiments of the present application.

[0039] Figure 7 It is a set of graphical user interfaces provided by the embodiments of the present application.

[0040] Figure 8 This is another set of graphical user interfaces provided by an embodiment of the present application.

[0041] Fig. 9 This is an example diagram of cropping image material provided in an embodiment of the present application.

[0042] Fig. 10A This is another example diagram of cropping image material provided in an embodiment of the present application.

[0043] Fig. 10B This is another example diagram of cropping image material provided in an embodiment of the present application.

[0044] Fig.11 This is another example diagram of cropping image material provided in an embodiment of the present application.

[0045] Fig. 12A This is another example diagram of cropping image material provided in an embodiment of the present application.

[0046] Fig. 12B This is another example diagram of cropping image material provided in an embodiment of the present application.

[0047] Fig.13 This is another example diagram of cropping image material provided in an embodiment of the present application.

[0048] Fig.14A This is another example diagram of cropping image material provided in an embodiment of the present application.

[0049] Fig. 14B This is another example diagram of cropping image material provided in an embodiment of the present application.

[0050] Fig.15 It is a schematic flow chart of an image processing method 200 provided in an embodiment of the present application.

[0051] Fig.16 It is a schematic flow chart of an image processing method 300 provided in an embodiment of the present application.

[0052] Fig.17 It is a schematic flowchart of an image processing method 400 provided in an embodiment of the present application.

[0053] Fig.18 It is a schematic flow chart of an image processing method 500 provided in an embodiment of the present application.

[0054] Fig.19It is a schematic flow chart of an image processing method 600 provided in an embodiment of the present application.

[0055] Fig. 20 It is an exemplary block diagram of an electronic device 700 provided in an embodiment of the present application.

[0056] Fig.21 It is a schematic structural diagram of an electronic device 800 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" or "multiple" refers to two or more than two.

[0058] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0059] The image processing method provided in the embodiment of the present application is applied to electronic devices. The electronic device may also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device may be an electronic device (mobile phone), a smart TV, a wearable device, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the electronic device.

[0060] Figure 1 1 is a schematic diagram of the structure of an electronic device 100 (taking a mobile phone as an example) provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, an antenna 1, an antenna 2, a mobile communication module 140, a wireless communication module 150, an audio module 160, a sensor module 170, a camera 181, a display screen 182, etc. The sensor module 170 may include a pressure sensor 170A, a gyroscope sensor 170B, etc.

[0061] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0063] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] Exemplarily, the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 182 and the camera 181. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 181 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 182 communicate via the DSI interface to implement the display function of the electronic device 100.

[0067] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 181, the display 182, the wireless communication module 150, the audio module 160, the sensor module 170, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0068] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0069] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0070] In the embodiment of the present application, when the electronic device 100 crops multiple image materials with different frame sizes, it can detect the subject of the image frame in the image material, generate a cropping frame according to the subject, the cropping frame includes the subject, and crop the image frame according to the cropping frame to obtain the cropped image material. The frame sizes of the multiple image materials can be made the same. Since the frame sizes of the multiple image materials are the same, the problem of black edges appearing in multiple image materials with different frame sizes can be solved. While solving the black edge problem, it can also ensure that the subject is included in the cropped image material, thereby avoiding the occurrence of situations such as loss of the subject and missing of the subject.

[0071] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 140, the wireless communication module 150, the modem processor and the baseband processor.

[0072] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0073] The mobile communication module 140 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 140 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 140 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 140 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 140 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 140 can be set in the same device as at least some of the modules of the processor 110.

[0074] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.), or displays an image or video through a display screen 182. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 140 or other functional modules.

[0075] The wireless communication module 150 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 150 can be one or more devices integrating at least one communication processing module. The wireless communication module 150 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 150 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0076] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 140, and the antenna 2 is coupled to the wireless communication module 150, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0077] The electronic device 100 implements the display function through a GPU, a display screen 182, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 182 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0078] The display screen 182 is used to display images, videos, etc. The display screen 182 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 182, where N is a positive integer greater than 1.

[0079] The electronic device 100 can realize the shooting function through ISP, camera 181, video codec, GPU, display screen 182 and application processor.

[0080] ISP is used to process the data fed back by camera 181. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 181.

[0081] The camera 181 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 181, where N is a positive integer greater than 1.

[0082] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0083] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0084] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0085] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0086] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0087] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 160 and the application processor.

[0088] The audio module 160 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 160 can also be used to encode and decode audio signals. In some embodiments, the audio module 160 can be arranged in the processor 110, or some functional modules of the audio module 160 can be arranged in the processor 110.

[0089] The pressure sensor 170A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 170A can be set on the display screen 182. There are many types of pressure sensors 170A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 170A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 182, the electronic device 100 detects the touch operation intensity according to the pressure sensor 170A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 170A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0090] The gyro sensor 170B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 170B. The gyro sensor 170B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 170B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 170B can also be used for navigation and somatosensory game scenes.

[0091] The hardware structure of the electronic device 100 is introduced here. It can be understood that: Figure 1 The components included in the hardware structure shown do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0092] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile terminal (such as a mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.

[0093] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0094] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0095] Figure 2 Schematic diagram of the software structure of the electronic device 100 of the embodiment of the present application. The software structure includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software structure is divided from top to bottom into an application layer, a media middle platform framework layer, a smart middle platform framework layer, a smart engine framework layer, and an application framework layer.

[0096] The application layer can include a series of application packages, such as camera, gallery, calendar, map, navigation, etc. When these application packages are running, they can access the various service modules provided by the media platform framework layer through the application programming interface (API) and execute corresponding services.

[0097] In some embodiments, the camera is used to take photos, videos, slow-motion images, panoramic images, etc. in response to user operations. After these images are taken by the camera, or after the user triggers the electronic device to take a screenshot, or after the user triggers the electronic device to record the screen, or after the electronic device downloads images from other devices, the electronic device can save these images in a gallery, so that the user can perform video editing operations on the images in the gallery, such as smart cropping operations.

[0098] The embodiment of the present application divides the gallery into: a business layer, an application function layer and a basic function layer from top to bottom.

[0099] The business layer provides a variety of services such as one-click filming, wonderful moments, and free creation. These services are presented in the form of controls in the user interface (UI) of the gallery. Users can trigger the gallery to perform corresponding video processing actions by operating a certain control. For example, after the user selects one or more image materials and clicks on the free creation control of the gallery, one or more image materials can be intelligently cropped, text processed, and special effects processed to obtain a created video. After the image materials are intelligently cropped, black edges can be avoided in the created video, and the subject can be avoided from being lost in the created video.

[0100] The application function layer includes a clipping framework. Each business in the business layer can call the clipping framework to provide clipping services for pictures and videos. Exemplarily, the clipping framework may include functional modules such as intelligent cropping, text, and special effects, wherein intelligent cropping includes configuration modules such as specifying the subject and the frame ratio, the specified subject is used to specify the subject in the cropped image material, and the frame ratio is used to specify the frame ratio of the cropped image material. The text module is used to add a text description to the subject in the image material. The special effects module is used to add a beautification effect to the subject in the image material, such as adjusting the brightness of the subject, beautifying the face of the subject (such as a person), etc. Exemplarily, the intelligent cropping module is used to detect the subject in the image material, match the detected subject with the specified subject, obtain the subject position, generate the position information of the cropping frame according to the subject position and the selected frame ratio, and crop the image material according to the position information of the cropping frame.

[0101] The basic function layer is used to perform basic function processing on the edited video clips after the editing framework edits multiple image materials. Exemplarily, the basic function layer may include basic function modules such as video scaling, synthesis preservation, and video effect rendering. Among them, video scaling is used to scale the image material obtained after cropping so that the cropped image material can completely fill the entire display screen. Synthesis preservation is used to store the image material obtained after cropping. Video effect rendering is used to add video effects to the image material obtained after cropping, such as adding style filters to the image material.

[0102] The media middle platform framework layer, the smart middle platform framework layer and the smart engine framework layer are software layers set between the application layer and the application framework layer. The media middle platform framework layer may include a parsing module, a decoding module, etc. The parsing module is used to parse the data of the streaming media protocol into standard corresponding encapsulation format data. When the video is transmitted on the network, various streaming media protocols are often used, such as Hyper Text Transfer Protocol (HTTP), Real Time Messaging Protocol (RTMR), etc. These protocols transmit some signaling data while transmitting video data. These signaling data include playback control (play, pause, stop), or description of network status, etc. During the parsing process, the signaling data will be removed and only the video data will be retained. For example, the data transmitted using the RTMP protocol, after the protocol decoding operation, outputs the data in the flash video format (FLash Video, FLV). The decoding module is used to decode the video compression encoding data into uncompressed video raw data. Through decoding, the compressed and encoded video data is output as uncompressed color data, such as color data (Luminance Chrominance, YUV), color data (RedGreenBlue, RGB), etc.

[0103] The smart middle platform framework layer includes a software development kit (SDK), which is a collection of related binary files, documents, examples and tools to assist in the development of application software. The SDK is used to call the algorithm modules in the smart engine framework layer to implement the specific functions of the algorithm modules.

[0104] The intelligent engine framework layer includes algorithm modules such as subject detection, subject tracking and subject segmentation. Subject detection is used to detect the subject in the image material, and subject tracking is used to track the specified subject. For example, the specified subject can be tracked by matching the subject detected by the subject detection module with the specified subject one by one. Subject segmentation is used to generate the position information of the cropping frame for cropping the video subject.

[0105] The application framework layer, referred to as the framework layer for short, can be used to support the operation of various modules in the media middle platform framework layer and the smart middle platform framework layer.

[0106] Understandably, Figure 2The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown, and each layer may include more or fewer components. For example, the media platform framework layer may also include a decapsulation module, an encoding module, etc. For another example, the software structure may also include a hardware abstraction layer, a system library, a kernel layer, etc., and the present application does not limit this.

[0107] The hardware abstraction layer is a package of the Linux kernel driver, providing an interface to the upper layer. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms. The kernel layer uses file descriptors to access files. File descriptors are non-negative integers. When opening an existing file or creating a new file, the kernel returns a file descriptor. Reading and writing files also requires the use of file descriptors to specify the files to be read and written.

[0108] It is understandable that in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0109] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0110] The following briefly summarizes the image processing method provided in the embodiment of the present application in combination with the above software architecture.

[0111] In an embodiment of the present application, the free creation service is located in the business layer of the application layer, which is a sub-service of video editing. N image materials to be cropped are selected at the business layer, and the N image materials are sent to the application function layer, where N is an integer greater than or equal to 2, and at least two of the N image materials have different aspect ratios.

[0112] In the application function layer, the free creation business is implemented based on the editing framework, which includes functional modules such as intelligent cropping, among which intelligent cropping includes configuration modules such as specifying the subject and the aspect ratio. In the application function layer, you can specify the subject in the cropped image material and the aspect ratio of the cropped image material, generate specified parameters such as the specified subject information and aspect ratio, and then send N image materials and the specified parameters to the media middle platform framework layer.

[0113] The media middle platform framework layer is mainly used to parse and decode N image materials, generate N decoded image materials, and then send the decoded N image materials and specified parameters to the smart middle platform framework layer.

[0114] The smart middle platform framework layer calls the algorithm module in the smart engine framework layer through the SDK, detects the subject in N image materials through the algorithm module, and matches the detected subject with the specified subject. After the match is successful, the position information of the cropping frame is determined according to the position information of the specified subject in the image material and the selected aspect ratio. Then the smart engine framework layer returns the calculated position information of the cropping frame to the smart middle platform framework layer, the smart middle platform framework layer returns it to the media middle platform framework layer, and finally returns it to the application layer. The application layer crops, processes basic functions and presents the N image materials according to the position information of the cropping frame.

[0115] The execution subject of the image processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the image processing method, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. The following takes an electronic device as an example, and combines the accompanying drawings to exemplarily illustrate the image processing method provided in the embodiment of the present application.

[0116] The technical problems to be solved by this application and the inventive concept of this application are introduced below.

[0117] Currently, in order to make the images or videos taken by electronic devices display better, users can import image materials into electronic devices or image processing software provided by third parties for processing, such as splicing multiple videos into one video through image processing software. However, when multiple image materials with different aspect ratios are imported into image processing software for splicing, black edges may appear in the processed image materials that are finally saved. For example, Figure 3 This is an example diagram of video processing provided in an embodiment of the present application. Figure 3The video shown in (a) is synthesized from two videos with different aspect ratios, the first video includes 3 image frames, and the second video includes 2 image frames. Due to the different aspect ratios of the first video and the second video, black edges appear on the second video in the processed image material that is finally saved.

[0118] In order to avoid black edges in the final saved processed image material, the common solution is to crop the imported image material at a fixed position so that the image materials with different aspect ratios imported into the image processing software have the same aspect ratio. For example, Figure 3 In the video shown in (b), in order to make the second video have the same aspect ratio as the first video, the second video is cropped from the middle position of the second video with a cropping frame having the same aspect ratio as the first video.

[0119] However, the position of the subject in the image material is uncertain. For example, if the subject in the video is an animal, then in a certain frame of the video, the animal is located in the lower left corner of the video screen, and in another frame of the video, the animal is located in the upper right corner of the video screen, and so on. After cropping at a fixed position of the imported image material, the fixed position may contain the subject, or may not contain the subject, or may only contain part of the subject. For example, if cropping is performed at the center position of the video, the center position of the video may contain the subject, or may not contain the subject, or may only contain part of the subject. As a result, the subject may be missing in the final processed image material, affecting the user experience. The final processed image material can be referenced. Figure 3 As shown in (c) in .

[0120] It should be noted that the black borders mentioned above may be videos or images with black borders on the top and bottom; or videos or images with black borders on the left and right; or videos or images with black borders on the top, bottom, left and right. For example, Figure 4 is an example diagram of a black-bordered screen provided in an embodiment of the present application. Figure 4 (a) refers to videos or images with black edges on the top and bottom. Figure 4 (b) refers to videos or images with black edges on the left and right. Figure 4 (c) in the figure refers to a video or image with black edges on the top, bottom, left, and right sides.

[0121] It should also be noted that the subject mentioned in the full text refers to the object that is emphasized in the video or image. For example, the subject can be a person, an animal, a landscape, etc., or a part of a person or an animal, such as a head, a hand, etc., which is not limited in the embodiments of the present application. The image material can contain only images, only videos, or both images and videos, which is not limited in the embodiments of the present application.

[0122] In view of the above problems, an embodiment of the present application provides an image processing method. In order to avoid the appearance of black edges in the final processed video, the present application can detect the position information of the subject of the imported image material and crop it according to the position information of the subject of the image material so that the cropped image material contains the subject. For example, the final processed video saved by the image processing method of the present application can refer to Figure 3 The video shown in (d).

[0123] Depend on Figure 3 As can be seen from the video shown in (d), the image processing method provided in the embodiment of the present application can not only avoid the appearance of black edges in the processed video that is finally saved, but also ensure that the subject will not be lost or missing, thereby improving the user experience.

[0124] The image processing method provided in the embodiments of the present application is introduced below in conjunction with specific application scenarios.

[0125] the following Figures 5 to 7 It is a set of graphical user interfaces (GUI) provided in the embodiments of the present application.

[0126] In one application scenario, for example, a user uses an electronic device to pre-shoot multiple image materials, and the electronic device stores the multiple image materials in a gallery. Figure 5 As shown in (a) of FIG. 1 , an icon of a gallery is displayed on the desktop of the electronic device. When the user wants the electronic device to generate a video clip without black edges based on the image material, the user can click the icon of the gallery on the desktop of the electronic device. In response to the user's click operation on the icon, the electronic device displays the following Figure 5 The gallery interface shown in (b) in FIG. 1 provides a "free creation" option, such as Figure 5 As shown in (c) of FIG. 1 , the user can click on the “free creation” option. In response to the user clicking on the “free creation” option, the electronic device displays the following Figure 5 The gallery interface shown in (d) in the figure may include a plurality of image materials taken recently, such as a plurality of image materials taken on October 1, 2020. Figure 5 As shown in (e) of FIG. 1 , the user can click on any one of the multiple image materials. In response to the user's click operation on the any one of the image materials, the electronic device displays the following Figure 5 The gallery interface shown in (f) includes all the materials in the gallery. Figure 5 Select the image material to be cropped in the gallery interface shown in (f) in the figure, for example Figure 5 As shown in (f) in FIG. 8 , it is assumed that the user selects 3 image materials. Figure 5 The gallery interface shown in (f) also includes confirmation options, such as Figure 5 As shown in (f) in the figure, check the option to confirm.

[0127] like Figure 6 As shown in (a) of FIG. 1 , after selecting the image material to be cropped, the user can click a confirmation option, such as clicking a check mark. In response to the user's click operation on the check mark option, the electronic device displays the following Figure 6 The gallery interface shown in (b) of FIG. 1 includes a "clip" option. When the user wants to perform intelligent cropping on the image material to be cropped, the user can click the "clip" option in the gallery interface. In response to the user's click operation on the "clip" option, the electronic device displays the following Figure 6 The gallery interface shown in (c) of FIG. 1 includes a "smart crop" option, and the user can click the "smart crop" option. In response to the user clicking the "smart crop" option, the electronic device displays the following Figure 6 The gallery interface shown in (d) in FIG. 4 includes multiple different aspect ratio options, such as a "9:16" option, a "16:9" option, a "1:1" option, a "4:3" option, a "3:4" option, etc. The user can Figure 6 In the gallery interface shown in (d) in the figure, select the aspect ratio. For example, if the user selects the "16:9" option, Figure 6 The gallery interface shown in (d) in FIG. 1 further includes a cancel option, for example, the cancel option is a cross, and the user clicks the cross option to cancel the selected aspect ratio, and the electronic device uses the preset aspect ratio when cropping. Figure 6 The gallery interface shown in (d) also includes a confirmation option, such as a check mark. Figure 6 As shown in (d) of FIG. 1 , the user can click the check mark option after selecting the aspect ratio. In response to the user clicking the check mark option, the electronic device displays the following Figure 6 The gallery interface shown in (e) in FIG. 1 includes an option to specify a subject, such as a check mark, and an option not to specify a subject, such as a cross. The user can Figure 6 In the gallery interface shown in (e) of FIG. 1 , a user selects a designated subject option, for example, the user clicks the “check mark” option. In response to the user clicking the “check mark” option, the electronic device displays the following Figure 6 In the gallery interface shown in (f) in FIG, the user can specify the subject by manually selecting the subject in the gallery interface, for example Figure 6 In the gallery interface shown in (f), the user can manually select the entire man as the designated subject in the preview box. Figure 6The gallery interface shown in (f) also includes a "reset" option. When the user is not satisfied with the currently selected subject or wants to reselect the subject, the user can click the "reset" option. The gallery interface also includes a confirmation option, such as a check mark. When the user has determined that the currently selected subject is the specified subject, the user can click the check mark.

[0128] In response to the user clicking the check mark, the electronic device starts to detect the subject in the image material, matches the detected subject with the specified subject, and determines whether the specified subject exists in the detected subject. If it is determined that the specified subject exists in the detected subject, then a cropping frame is determined according to the specified subject and the aspect ratio selected by the user, and the image material is cropped according to the cropping frame. Figure 6 In (f), after the electronic device has cropped the image material, the user can click on the "Export" option. In response to the user's click on the video export option, the electronic device exports the video clip and stores the video clip in the gallery. In this way, the user can view the video clip from the gallery.

[0129] In another example, a user Figure 6 After selecting the aspect ratio in the gallery interface shown in (d) and clicking the check mark option, the electronic device displays Figure 7 The gallery interface shown in (a) of FIG. 1 includes multiple different subject options, such as a "head" option, a "body" option, a "hand" option, a "flower and grass" option, etc. The user can select a subject from the following list: Figure 7 In the designated subject in the gallery interface shown in (a), for example, the user can click the "head" option to select the "head" as the designated subject. Figure 7 (a) in the example also includes a cancel option, such as a cross. When the user does not want to specify a subject, the user can click the cross. For example, Figure 7 As shown in (b) in the figure, the user can cancel the specified subject by clicking the "cross" option. Figure 7 (a) in the figure also includes a confirmation option, such as a check mark. When the user has determined that the currently selected "head" is the specified subject, the user can click the check mark, for example Figure 7 As shown in (c) in the figure, the user clicks the "check mark" option, and the electronic device responds to the user's click of the check mark operation, and starts to detect the subject in the image material, matches the detected subject with the specified subject, and determines whether the specified subject exists in the detected subject. If it is determined that the detected subject is the specified subject, then the cropping frame is determined according to the position information of the detected subject and the aspect ratio selected by the user, and the image material is cropped according to the cropping frame. Figure 7In (c), after the electronic device has cropped the image material, the user can click on the "Export" option. In response to the user's click on the video export option, the electronic device exports the video clip and stores the video clip in the gallery. In this way, the user can view the video clip from the gallery.

[0130] In one example, after the electronic device crops the image material according to the cropping frame, it scales the cropped image material so that the cropped image material can completely fill the entire display screen.

[0131] In one example, Figure 6 The gallery interface displayed in (b) or (c) also includes other functional options, which may include but are not limited to filter, music, text, special effects, picture-in-picture, etc. After the electronic device crops the image material and generates a video clip, the user can perform operations such as filtering, adding music, adding text, adding special effects, etc. on the generated video clip based on these functional options.

[0132] Figure 8 This is another set of GUIs provided by an embodiment of the present application.

[0133] In another application scenario, illustratively, a user uses an electronic device to pre-shoot multiple image materials, and the electronic device stores the multiple image materials in a gallery. In this way, the user can select an image material in the gallery, so that the electronic device generates a video clip without black edges based on the image material.

[0134] For example, Figure 8 As shown in (a) of FIG. 1 , after entering the gallery interface, the gallery interface includes multiple image materials taken recently. The user can click on any one of the multiple image materials, such as the first image material. In response to the user's click operation on the any one material, the electronic device displays the following Figure 8 The gallery interface shown in (b) of FIG. 1 includes all image materials in the gallery. In this way, the user can Figure 8 In the gallery interface shown in (b) in FIG. 1 , select the image material to be cropped, for example, Figure 8 As shown in (b) in FIG. 1 , it is assumed that the user has selected three image materials to be cropped. Figure 8 The gallery interface shown in (b) also includes a "Create" option. When the user wants to crop the image material to be cropped, the user can click the "Create" option. In response to the user clicking the "Create" option, the electronic device displays the following Figure 8 The gallery interface shown in (c) includes a "free creation" option, and the user can choose to click the "free creation" option to achieve the cropping of the image material to be cropped.

[0135] In one example, after the user clicks on the "free creation" option, the electronic device responds to the user clicking on the "free creation" option and displays a gallery interface that can refer to Figure 6 In this example, the user can click Figure 6 Click the "Clip" option shown in (b) Figure 6 The "Smart Crop" option shown in (c) of Figure 6 Select the aspect ratio in (d) and click to confirm the option. For example, click the "16:9" option and click the check mark. Figure 6 In (e) of the , select the specified subject option, such as the check mark option, and then Figure 6 Specify the subject in (f) and click Confirm. For example, Figure 6 In the preview box in the gallery interface shown in (f), the subject is specified by manually selecting the subject in the box and clicking the check mark option after specifying the subject. At this time, the electronic device begins to detect the subject in the image material, matches the detected subject with the specified subject, and determines whether the specified subject exists in the detected subject. If it is determined that the specified subject exists in the detected subject, then the cropping frame is determined according to the specified subject and the aspect ratio selected by the user, and the image material is cropped according to the cropping frame. Refer to Figure 6 In (f), after the electronic device has cropped the image material, the user can click on the "Export" option. In response to the user's click on the video export option, the electronic device exports the video clip and stores the video clip in the gallery. In this way, the user can view the video clip from the gallery.

[0136] In another example, after the user clicks on the "free creation" option, the electronic device responds to the user clicking on the "free creation" option and displays a gallery interface that can refer to Figure 6 (b), (c), (d) and Figure 7 In this example, the user can click Figure 6 Click the "Clip" option shown in (b) Figure 6 The "Smart Crop" option shown in (c) of Figure 6 Select the aspect ratio in (d) and click to confirm the option. For example, click the "16:9" option and click the check mark. Figure 7 In (a) of the example, the user can click the "Head" option to select "Head" as the specified subject, and then Figure 7Click the confirmation option in (c) in the image. For example, after the user clicks the "Head" option, click the check mark option. At this time, the electronic device begins to detect the subject in the image material, matches the detected subject with the specified subject, and determines whether the specified subject exists in the detected subject. If it is determined that the specified subject exists in the detected subject, then the cropping frame is determined according to the specified subject and the aspect ratio selected by the user, and the image material is cropped according to the cropping frame. Figure 7 In (c), after the electronic device has cropped the image material, the user can click on the "Export" option. In response to the user's click on the video export option, the electronic device exports the video clip and stores the video clip in the gallery. In this way, the user can view the video clip from the gallery.

[0137] It should be noted that in the above application scenarios, the user's operation of selecting the aspect ratio and the operation of specifying the subject are both optional operations.

[0138] The following four examples (Example 1, Example 2, Example 3 and Example 4) describe in detail how the electronic device crops N image materials when the user's operation of selecting the aspect ratio and the operation of specifying the subject are both optional operations.

[0139] Example 1: How does an electronic device crop N image materials when a user does not need to select an aspect ratio or specify a subject?

[0140] In this example, the user can click the "Smart Crop" option to crop the image material without having to select the aspect ratio or specify the subject. Figure 6 After selecting the "smart cropping" option shown in (c), the electronic device can crop N image materials in response to the operation.

[0141] In the implementation, in step A, in response to the user clicking the "smart crop" option, the electronic device performs subject detection on N image materials to obtain position information of the target subject in the image materials.

[0142] It should be understood that the purpose of performing intelligent cropping on N image materials is to make the aspect ratios of the N image materials the same, so as to solve the problem of black edges in the final saved video. Therefore, since the user of the embodiment of the present application does not need to perform the operation of selecting the aspect ratio, the embodiment of the present application can make the aspect ratio of the N image materials the same as the preset aspect ratio, wherein the preset aspect ratio can be the aspect ratio of any one of the N image materials. For example, the N image materials imported into the image processing software include a first video and a second video, and the preset aspect ratio can be the aspect ratio of the first video or the aspect ratio of the second video.

[0143] In some embodiments, in order to improve the performance of the electronic device and save the power consumption of the electronic device, the electronic device performs subject detection on N image materials, and the electronic device may perform subject detection on each image frame of each image material in some of the N image materials. For example, the N image materials include a first video segment and a second video segment, and the preset frame ratio is the first video segment. When the electronic device performs subject detection on the two videos, it may only perform subject detection on each image frame of the second video segment.

[0144] It should also be understood that when the electronic device performs subject detection on each image frame, the subject detection algorithm can be used for detection. For example, the subject detection algorithm can be a saliency detection algorithm. The saliency detection algorithm uses a network model with an encoding-decoding structure when detecting the subject. The implementation process of detecting the subject of the image frame by the saliency detection algorithm can be: inputting the image frame into the network model with an encoding-decoding structure to predict the subject in the image frame, outputting the predicted candidate subject mask, generating the target subject prediction frame according to the subject mask, and generating the position information of the target subject prediction frame, wherein the encoding structure can use a lightweight network to reduce the amount of calculation, the decoding network structure can fuse the features of different scales of the encoding structure, and the decoding structure can add a dilated convolution to expand the receptive field.

[0145] In one case, there is only one subject in the image material to be detected.

[0146] In the embodiment of the present application, the electronic device performs subject detection on N image materials to obtain the location information of the target subject in the image materials as follows:

[0147] The electronic device performs subject detection on each image frame of each image material in the partial image materials of N image materials, obtains position information of a subject in the image frame, and determines the position information of the subject in the image frame as the position information of the target subject in the image frame.

[0148] For example, Fig. 9is an example diagram of cropping image material provided by an embodiment of the present application. Assume Fig. 9 (a) in the figure is two videos with different aspect ratios imported into the image processing software. The aspect ratio of the first video is 9:16, and the aspect ratio of the second video is 16:9. The aspect ratio of the first video is a preset aspect ratio, and there is only one subject in the first video and the second video to be detected. Then, when the electronic device performs subject detection on the two videos, it can only perform subject detection on each image frame in the second video. For example, when detecting the subject in the image frame of the second video, the predicted candidate subject mask can be output by the saliency detection algorithm, such as Fig. 9 A1 shown in (a) is a candidate subject mask of the second video segment. A target subject prediction box is generated according to the candidate subject mask. The target subject prediction box can be a rectangle, a square, etc. The embodiment of the present application does not limit the shape of the target subject prediction box. For example, Fig. 9 B1 shown in (b) is the target subject prediction frame of the second video. The position information of the target subject prediction frame can represent the position information of the subject in the image frame. In this example, B1 can be determined as the position information of the target subject in the image frame.

[0149] In another case, there are multiple subjects in the image material to be detected.

[0150] In the embodiment of the present application, the electronic device performs subject detection on N image materials to obtain the location information of the target subject in the image materials as follows:

[0151] The electronic device performs subject detection on each image frame of each image material among some image materials of N image materials to obtain position information of multiple subjects in the image frame, and obtains position information of at least one target subject in the image frame based on the position information of the multiple subjects in the image frame.

[0152] In one implementation, the electronic device may use the position information of any subject among the multiple subjects in the image frame as the position information of the target subject in the image frame. In another implementation, the electronic device may use the position information of all subjects among the multiple subjects in the image frame as the position information of the target subject in the image frame. In another implementation, the electronic device may score the detected multiple subjects according to the image color features, image texture features, image quality, etc. of the detected multiple subjects. The scoring result may be used as a basis for evaluating whether a subject is the target subject. For example, when the scoring result of one or more subjects is greater than or equal to 60 points, the one or more subjects are the target subjects, and then the position information of the target subject may be obtained based on the target subject.

[0153] For example, Fig. 10A is another example diagram of cropping image material provided by the embodiment of the present application. Fig. 10A (a) in the figure is two videos with different aspect ratios imported into the image processing software. There is one subject in the first video and two subjects in the second video. Then, when the electronic device detects the subject in the image material in the second video, it can output the predicted candidate subject mask through the saliency detection algorithm, for example Fig. 10A A1 and A2 shown in (a) are candidate subject masks of the second video. Assume that A1 and A2 are processed and A2 is the target subject mask. Then, the target subject prediction box is generated according to the target subject mask, for example Fig. 10A B2 shown in (b) is the target subject prediction box of the second video.

[0154] For example, Fig. 10B This is another example diagram of cropping image material provided in an embodiment of the present application. Fig. 10B A1 and A2 shown in (a) are candidate subject masks of the second video. Assume that A1 and A2 obtained by processing A1 and A2 are both target subject masks. Then, the target subject prediction box is generated according to the target subject mask, for example Fig. 10B B1 and B2 shown in (b) are two target subject prediction boxes of the second video.

[0155] In some embodiments, the electronic device performs subject detection on N image materials, which may be that the electronic device performs subject detection on each image frame of each image material in all image materials of the N image materials to obtain position information of the target subject in the image material.

[0156] The implementation of this embodiment can refer to Fig. 9 , Fig. 10A and Fig. 10B The implementation method of performing subject detection on a certain image frame in the second video to obtain the position information of the target subject in the image material will not be repeated here.

[0157] In some embodiments, the subject cannot be detected because there is no subject in each frame of each image material in N image materials, or there is at least one subject but all subjects in at least one subject are missing, or the scoring results of all subjects in at least one subject are lower than a preset score, for example, the scoring results of all subjects are lower than 60 points, etc. At this time, the electronic device performs subject detection on the N image materials and generates a first abnormality identification.

[0158] In step B, the electronic device determines the position information of the first cropping frame in the image frame according to the position information of the target subject in the image frame and a preset frame ratio.

[0159] During implementation, the electronic device can change the width and height of the target subject prediction frame according to the position information of the target subject prediction frame and the preset aspect ratio, so that the aspect ratio of the changed target subject prediction frame conforms to the preset aspect ratio, and determine the position information of the changed target subject prediction frame as the position information of the first cropping frame in the image frame.

[0160] The changed target subject prediction frame meets two requirements: one is that the frame ratio of the changed target subject prediction frame is a preset frame ratio; the other is that the changed target subject prediction frame contains the subject.

[0161] For example: Fig. 9 The preset aspect ratio shown in (c) is 9:16, the target subject prediction box is B1, B1 is changed to C1, the aspect ratio of C1 is 9:16, and C1 contains the subject.

[0162] Another example: Fig. 10A The preset aspect ratio shown in (c) is 9:16, and the target subject prediction box is B2. B2 is changed to C2, and the aspect ratio of C2 is 9:16, and C2 contains the subject.

[0163] Another example: Fig. 10B The preset aspect ratio shown in (c) is 9:16, and the target subject prediction boxes are B1 and B2. B1 and B2 are changed to C2, and the aspect ratio of C2 is 9:16, and C2 contains the subject.

[0164] In some embodiments, when the first abnormality mark is obtained in step A, the electronic device obtains the position information of the second cropping frame in the image frame based on the position information of any position of the image frame and a preset frame ratio.

[0165] It should be understood that when the electronic device obtains the first abnormal identification, it proves that there is no subject in the image frame or the subject is missing or the scores of all subjects are lower than the preset score. At this time, the position information of the second cropping frame in the image frame can be obtained based on the position information of any position of the image frame, for example, the position information of the center position of the image frame and the preset screen ratio, for example, the preset screen ratio is 9:16, for example, the screen ratio of the second cropping frame is 9:16 and the second cropping frame is located in the middle position of the image frame.

[0166] Step C, cropping the image frame according to the position information of the first cropping frame in the image frame, scaling the cropped image frame to obtain scaled image material, exporting the scaled image material and displaying it in the gallery.

[0167] For example, Fig. 9 The C1 picture shown in (c) is enlarged to obtain the D1 picture, and the video finally exported and displayed in the gallery is Fig. 9 As shown in (d) in .

[0168] For example, Fig. 10A The C2 image shown in (c) is enlarged to obtain the D2 image, and the video finally exported and displayed in the gallery is Fig. 10A As shown in (d) in .

[0169] For example, Fig. 10B The C2 image shown in (c) is reduced to obtain the D2 image, and the video finally exported and displayed in the gallery is Fig. 10A As shown in (d) in .

[0170] In some embodiments, when the first abnormality mark is obtained in step A, the image frame can also be cropped according to the position information of the second cropping frame in the image frame, and the cropped image frame can be scaled to obtain the scaled image material, and the scaled image material can be exported and displayed in the gallery.

[0171] Example 2: How does an electronic device crop N image materials when a user selects an aspect ratio and does not need to specify a subject?

[0172] In this example, after clicking the "Smart Crop" option, the user can select the aspect ratio and crop the image material without specifying the subject. Figure 6 After selecting the "Smart Crop" option shown in (c), click Figure 6 The "16:9" option and the check mark option shown in (d) in the figure can enable the electronic device to crop N image materials in response to the user's operation.

[0173] In the implementation, step D, in response to the user clicking Figure 6 In the operation of the check option shown in (d), the electronic device performs subject detection on N image materials to obtain position information of the target subject in the image materials.

[0174] It should be understood that the purpose of intelligent cropping of N image materials is to make the aspect ratios of the N image materials the same, so as to solve the problem of black edges in the final saved video. Then, since the user in the embodiment of the present application has performed an operation of selecting the aspect ratio, the embodiment of the present application can make the aspect ratios of the N image materials the same as the selected aspect ratio, wherein the selected aspect ratio can be the same as the aspect ratio of a certain image material among the N image materials, and the selected aspect ratio can also be different from the aspect ratios of each image material among the N image materials. For example, the N image materials imported into the image processing software include a first video and a second video, the aspect ratio of the first video is 9:16, and the aspect ratio of the second video is 16:9, and the selected aspect ratio can be 16:9, 9:16, 4:3, etc.

[0175] It should also be understood that, when the selected aspect ratio is the same as the aspect ratio of a certain image material among N image materials, an implementation method of step D is the same as the implementation method in step A in which the electronic device performs subject detection on each frame of each image material among some of the N image materials to obtain the position information of the target subject in the image material, and will not be repeated here.

[0176] In this case, another implementation of step D is: the electronic device performs subject detection on each image frame of each image material in the N image materials to obtain position information of the target subject in the image material.

[0177] For example, Fig.11 is another example diagram of cropping image material provided by the embodiment of the present application. Fig.11 (a) in the figure is two videos with different aspect ratios imported into the image processing software. The aspect ratio of the first video is 9:16, and the aspect ratio of the second video is 16:9. Assuming that there is only one subject in the first video and the second video and the aspect ratio is 9:16, then when the electronic device detects the subject in the two videos, it can detect the subject for each image frame in the two videos. For example, when the electronic device detects the subject in the image frames of the two videos, it can output the predicted candidate subject mask through the saliency detection algorithm, such as Fig.11 A3 shown in (a) is the candidate subject mask of the first video segment, and A1 is the candidate subject mask of the second video segment. The target subject prediction box is generated according to the target subject mask, for example Fig.11 B1 shown in (b) is the target subject prediction frame of the second video segment, and B3 is the target subject prediction frame of the first video segment. The position information of the target subject prediction frame can represent the position information of the target subject.

[0178] It can also be understood that when the selected aspect ratio is different from the aspect ratio of each image material in the N image materials, the implementation method of step D is: the electronic device performs subject detection on each image frame of each image material in the N image materials to obtain the position information of the target subject in the image material.

[0179] For example: Fig. 12A is another example diagram of cropping image material provided by the embodiment of the present application. Fig. 12A (a) in the figure is two videos with different aspect ratios imported into the image processing software. The aspect ratio of the first video is 9:16, and the aspect ratio of the second video is 16:9. Assuming that there are two subjects in both the first video and the second video, when the electronic device detects the subjects in the two videos, it can output the predicted candidate subject masks through the saliency detection algorithm, for example Fig. 12A In (a), A1 and A2 are candidate subject masks of the second video segment, and A3 and A4 are candidate subject masks of the first video segment. Assuming that A1 and A2 are processed to obtain A2 as the target subject; A3 and A4 are processed to obtain A3 as the target subject, then the target subject prediction box is generated according to the target subject mask, for example Fig. 12A B2 shown in (b) is the target subject prediction box of the second video segment, and B3 is the target subject prediction box of the first video segment.

[0180] Another example: Fig. 12B This is another example of cropping an image material provided by an embodiment of the present application. When an electronic device detects subjects in two videos, it can output a predicted candidate subject mask through a saliency detection algorithm, such as Fig. 12B In (a), A1 and A2 are candidate subject masks of the second video, and A3 and A4 are candidate subject masks of the first video. Assuming that A1 and A2 are processed, A1 and A2 are both target subjects; A3 and A4 are processed, A3 and A4 are target subjects, then, the target subject prediction box is generated according to the target subject mask, for example Fig. 12B B2 and B1 shown in (b) are the target subject prediction boxes of the second video segment, and B3 and B4 are the target subject prediction boxes of the first video segment.

[0181] In some embodiments, the subject cannot be detected because there is no subject in each frame of each image material in N image materials, or there is at least one subject but all subjects in the at least one subject are missing, or the scoring results of all subjects in at least one subject are lower than a preset score, for example, the scoring results of all subjects are lower than 60 points, etc. At this time, the electronic device performs subject detection on the N image materials and generates a first abnormality identification.

[0182] In step E, the electronic device determines the position information of the first cropping frame in the image frame according to the position information of the target subject in the image frame and the selected frame ratio.

[0183] The implementation method of step E is the same as that of step B, which will not be repeated here.

[0184] When the selected aspect ratio is the same as the aspect ratio of a certain image material among the N image materials, the position information of the determined first cropping frame in the image frame can be referred to Fig. 9 C1 shown in (c) Fig. 10A C2 shown in (c) and Fig. 10B C2 shown in (c) can also be referred to Fig.11 C1 and C3 shown in (c).

[0185] Also refer to Fig. 12A , assuming Fig. 12A The aspect ratio of the first video is 9:16, the aspect ratio of the second video is 16:9, and the selected aspect ratio is 4:3, then, Fig. 12A The target subject prediction box of the first video shown in (c) is B3, which is changed to C3, the aspect ratio of C3 is 4:3, and C3 contains the subject. The target subject prediction box of the second video is B2, which is changed to C2, the aspect ratio of C2 is 4:3, and C2 contains the subject.

[0186] Also refer to Fig. 12B , Fig. 12B In (c), the target subject prediction boxes of the first video are B3 and B4, which are changed to C3, the aspect ratio of C3 is 4:3, and the subject is contained in C3. The target subject prediction boxes of the second video are B2 and B1, which are changed to C2, the aspect ratio of C2 is 4:3, and the subject is contained in C2.

[0187] When the selected aspect ratio is different from the aspect ratio of each of the N image materials, the position information of the determined first cropping frame in the image frame can be referred to Fig.11 C1 and C3 shown in (c) Fig. 12A C2 and C3 shown in (c), and Fig. 12B C2 and C3 shown in (c).

[0188] In some embodiments, when the first abnormality mark is obtained in step D, the electronic device obtains the position information of the second cropping frame in the image frame based on the position information of any position of the image frame and the selected frame ratio.

[0189] It should be understood that when the electronic device obtains the first abnormal identification, it proves that there is no subject in the image frame or the subject is lost or the scores of all subjects are lower than the preset score. At this time, the position information of the second cropping frame in the image frame can be obtained based on the position information of any position of the image frame, for example, the position information of the center position of the image frame and the selected picture ratio, for example, the selected picture ratio is 4:3, for example, the picture ratio of the second cropping frame is 4:3 and the second cropping frame is located in the middle position of the image frame.

[0190] Step F, cropping the image frame according to the position information of the first cropping frame in the image frame, scaling the cropped image frame to obtain scaled image material, exporting the scaled image material and displaying it in the gallery.

[0191] For example, Fig.11 The C1 and C3 images shown in (c) are enlarged to obtain the D1 and D3 images, and the video finally exported and displayed in the gallery is Fig.11 As shown in (d) in .

[0192] For example, Fig. 12A The C2 and C3 images shown in (c) are enlarged to obtain the D2 and D3 images, and the video finally exported and displayed in the gallery is Fig. 12A As shown in (d) in .

[0193] For example, Fig. 12B The C3 picture shown in (c) is reduced to obtain the D3 picture, and the C2 picture is enlarged to obtain the D2 picture. The video finally exported and displayed in the gallery is Fig. 12B As shown in (d) in .

[0194] In some embodiments, when the first abnormality mark is obtained in step E, the image frame can also be cropped according to the position information of the second cropping frame in the image frame, and the cropped image frame can be scaled to obtain the scaled image material, and the scaled image material can be exported and displayed in the gallery.

[0195] Example 3: How does the electronic device crop N image materials when the user does not need to select an aspect ratio but performs an operation to specify a subject?

[0196] In this example, after clicking the "Smart Crop" option, the user can specify the subject and crop the image without having to select the aspect ratio. Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6Click the cross option in the gallery interface of (d) in the pop-up window. Figure 6 Click the check mark option in the gallery interface of (e) in the pop-up Figure 6 In the gallery interface of (f), manually select the subject and click the check mark option. Another example: When the user clicks Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6 Click the cross option in the gallery interface of (d) in the pop-up window. Figure 7 In the gallery interface of (a) in the figure, specify the subject. For example, click the "Head" option. Figure 7 In the gallery interface of (c), click the check mark option. At this time, the electronic device can crop the image material in response to the above operation of the user.

[0197] In the implementation, step G, in response to the user clicking Figure 6 (f) or Figure 7 In the operation of the check option shown in (c), the electronic device performs subject detection on N image materials to obtain at least one candidate subject, matches the at least one candidate subject with the designated subject, and when the designated subject successfully matches at least part of the at least one candidate subject, determines the position information of the designated subject in the image frame as the position information of the target subject in the image frame.

[0198] It should be understood that since the user does not need to select the aspect ratio in the embodiment of the present application, the embodiment of the present application can make the aspect ratio of the N image materials the same as the preset aspect ratio. The definition of the preset aspect ratio can be referred to the above embodiment and will not be repeated here.

[0199] It should also be understood that since the user of the embodiment of the present application has performed an operation of specifying a subject, after subject detection is performed on N image materials, the detected subject needs to be matched with the specified subject. In layman's terms, the specified subject needs to be tracked among the detected subjects, and then the position information of the target subject in the image frame is determined based on the matching result. In the embodiment of the present application, when the user performs an operation of specifying a subject, one subject or multiple subjects may be specified, and the embodiment of the present application does not limit this.

[0200] In the embodiment of the present application, the process of implementing subject detection of N image materials by the electronic device has been described in the above embodiment and will not be repeated here. Fig.13 is another example diagram of cropping image material provided by the embodiment of the present application. Fig.13(a) in the figure is two videos with different aspect ratios imported into the image processing software. The aspect ratio of the first video is a preset aspect ratio and there is only one subject in each of the first video and the second video to be detected. Then, when the electronic device detects the subject in the image frames in the first video and the second video, it can output the predicted candidate subject mask through the saliency detection algorithm. For example, Fig.13 A2 shown in (a) is a candidate subject mask, and a target subject prediction box is generated according to the target subject mask, for example, Fig.13 B2 shown in (b) is a candidate subject prediction box, which can represent the candidate subject.

[0201] For example, Fig.14A This is another example diagram of cropping image material provided by an embodiment of the present application. There is only one subject in the first video to be detected, and there are two subjects in the second video. When the electronic device detects the subjects in the image frames in the first video and the second video, it can output the predicted candidate subject mask through the saliency detection algorithm, for example, Fig.14A The candidate subject mask of the first video shown in (a) is A2, and the candidate subject masks of the second video are A1 and A2. The target subject prediction box is generated according to the target subject mask, for example, Fig.14A The candidate subject prediction box of the first video shown in (b) is B2, and the candidate subject prediction boxes of the second video are B1 and B2. The candidate subject prediction boxes can represent the candidate subjects.

[0202] For example, Fig. 14B This is another example diagram of cropping image material provided by an embodiment of the present application. There are two subjects in the first video to be detected, and there are two subjects in the second video. When the electronic device detects the subjects in the image frames in the first video and the second video, it can output the predicted candidate subject mask through the saliency detection algorithm, for example, Fig.14A The candidate subject masks of the first video shown in (a) are A2 and A3, and the candidate subject masks of the second video are A1 and A2. The target subject prediction box is generated according to the target subject mask, for example, Fig. 14B The candidate subject prediction boxes of the first video shown in (b) are B2 and B3, and the candidate subject prediction boxes of the second video are B1 and B2. The candidate subject prediction boxes can represent the candidate subjects.

[0203] In an embodiment of the present application, matching processing of at least one candidate subject with a designated subject can be performed by a subject tracking algorithm. For example, the subject tracking algorithm can be a single subject tracking algorithm, wherein the single subject tracking algorithm is divided into two network branches, the input of one branch is the designated subject, and the input of the other branch is the candidate subject. The two network branches respectively extract image features (for example, image texture features, image color features, etc.), and match the candidate subject with the designated subject based on the extracted image features.

[0204] For example, the input of a branch is a specified subject (such as Fig.13 , Fig.14A and Fig. 14B The input of the other branch is the candidate subject (for example, the candidate subject can be Fig.13 The candidate subject B2 in the first video and the candidate subject B2 in the second video can also be Fig.14A The candidate subject B2 in the first video and the candidate subjects B1 and B2 in the second video can also be Fig. 14B The candidate subjects B2 and B3 in the first video and the candidate subjects B1 and B2 in the second video are selected, and the candidate subjects are matched with the designated subject according to the extracted image features of the candidate subjects and the designated subject. If the designated subject successfully matches a part of at least one of the candidate subjects, the part of the candidate subject and the designated subject are taken as a matching pair (for example, Fig.13 If the specified subject A2 can be successfully matched with a candidate subject B2 in the image frame of the first video and successfully matched with a candidate subject B2 in the image frame of the second video, then A2 and B2 are regarded as a matching pair. For another example, Fig.14A The specified subject A2 in can be successfully matched with a candidate subject B2 in the image frame of the first video, and successfully matched with B2 in the two candidate subjects B1 and B2 in the image frame of the second video, then A2 and B2 are regarded as a matching pair. For another example, Fig. 14B The specified subject A2 in the image frame can successfully match B2 of the two candidate subjects B2 and B3 in the image frame of the first video, and successfully match B2 of the two candidate subjects B1 and B2 in the image frame of the second video, then A2 and B2 are taken as a matching pair). If the specified subject does not match any of the subjects in at least one candidate subject, the matching event is identified as an abnormal event, and an abnormal flag is generated. The last two network branches jointly output the position information or abnormal flag of the target subject in the matching pair in the image frame. The target subject is the subject in the matching pair that matches the specified subject. The position information of the target subject in the image frame can be referred to Fig.13 or Fig.14A or Fig. 14B B2 shown in (c).

[0205] In other embodiments, the implementation method of matching at least one candidate subject with each of multiple designated subjects can refer to the implementation method of matching at least one candidate subject with one designated subject in the above embodiment, and the embodiments of the present application are not limited to this.

[0206] It should be understood that the two network branches can use lightweight networks to reduce the amount of calculation, and use a variety of methods including hole convolution to expand the receptive field of the network branches.

[0207] In some embodiments, because there is no subject in each frame of each image material in N image materials, or there is at least one subject but all subjects in at least one subject are missing, or the scoring results of all subjects in at least one subject are lower than a preset score, for example, the scoring results of all subjects are lower than 60 points, etc., the candidate subject cannot be detected. At this time, the electronic device performs subject detection on the N image materials and generates a first abnormality identification.

[0208] In other embodiments, when an electronic device performs subject detection on N image materials and obtains at least one candidate subject, the specified subject does not match any subject in at least one candidate subject due to reasons such as the absence of the specified subject in at least one candidate subject. At this time, a matching process is performed on at least one candidate subject and the specified subject to obtain a second exception identification.

[0209] In step H, the electronic device determines the position information of the first cropping frame in the image frame according to the position information of the target subject in the image frame and a preset frame ratio.

[0210] It should be understood that the implementation of step H is the same as that of step B, and will not be described in detail here. Fig.13 or Fig.14A or Fig. 14B B2 shown in (c) is the position information of the target subject in the image frame, C2 is the first cropping frame, the frame ratio of the first cropping frame is a preset frame ratio, and C2 contains the subject A2.

[0211] In some embodiments, when the first abnormality mark is obtained in step G, the electronic device obtains the position information of the second cropping frame in the image frame based on the position information of any position of the image frame and a preset frame ratio.

[0212] It should be understood that when the electronic device obtains the first abnormal identification, it proves that there is no subject in the image frame or the subject is lost or the scores of all subjects are lower than the preset score. At this time, the position information of the second cropping frame in the image frame can be obtained based on the position information of any position of the image frame, for example, the position information of the center position of the image frame and the preset screen ratio, for example, the preset screen ratio is 9:16, for example, the screen ratio of the second cropping frame is 9:16 and the second cropping frame is located in the middle position of the image frame.

[0213] In other embodiments, after performing subject detection on N image materials in step G to obtain at least one candidate subject, when a second abnormality mark is obtained, the electronic device obtains position information of the third cropping frame in the image frame based on the position information of any position of the image frame and a preset frame ratio.

[0214] It should be understood that the electronic device obtains the second abnormal identification, proving that there is no subject in the image frame that matches the specified subject. At this time, the position information of the third cropping frame in the image frame can be obtained based on the position information of any position of the image frame, for example, the position information of the center position of the image frame and the preset screen ratio, for example, the preset screen ratio is 9:16, for example, the screen ratio of the third cropping frame is 9:16 and the third cropping frame is located in the middle position of the image frame.

[0215] Step I: cropping the image frame according to the position information of the first cropping frame in the image frame, scaling the cropped image frame to obtain scaled image material, exporting the scaled image material and displaying it in the gallery.

[0216] It should be understood that the implementation of step I is the same as step C, and will not be repeated here. For example, the video that is exported after scaling and displayed in the gallery can refer to Fig.13 or Fig.14A or Fig. 14B The video shown in (d).

[0217] In some embodiments, when the first abnormal identification is obtained in step G, the image frame can also be cropped according to the position information of the second cropping frame in the image frame, and the cropped image frame can be scaled to obtain the scaled image material, and the scaled image material can be exported and displayed in the gallery.

[0218] In some embodiments, when the second abnormal identification is obtained in step G, the image frame can also be cropped according to the position information of the third cropping frame in the image frame, and the cropped image frame can be scaled to obtain the scaled image material, and the scaled image material can be exported and displayed in the gallery.

[0219] Example 4: How does an electronic device crop an image material when a user selects an aspect ratio and specifies a subject?

[0220] In this example, after clicking the "Smart Crop" option, the user can select the aspect ratio and specify the subject to crop the image material. Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6 In the gallery interface (d), select the aspect ratio. For example, click the "16:9" option and then click the check mark option. Figure 6 Click the check mark option in the gallery interface of (e) in the pop-up Figure 6 In the gallery interface of (f), manually select the subject and click the check mark option. Another example: When the user clicks Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6 Select the aspect ratio in the gallery interface (d) in the pop-up window. Figure 7 In the gallery interface of (a) in the figure, specify the subject. For example, click the "Head" option. Figure 7 In the gallery interface of (c), click the check mark option. At this time, the electronic device can crop the image material in response to the above operation of the user.

[0221] In the implementation, step J, in response to the user clicking Figure 6 (f) or Figure 7 In the operation of the check option shown in (c), the electronic device performs subject detection on N image materials to obtain at least one candidate subject, matches the at least one candidate subject with the designated subject, and when the designated subject successfully matches at least part of the at least one candidate subject, determines the position information of the designated subject in the image frame as the position information of the target subject in the image frame.

[0222] The implementation method of step J is the same as that of step G, which will not be repeated here.

[0223] In some embodiments, because there is no subject in each frame of each image material in N image materials, or there is at least one subject but all subjects in at least one subject are missing, or the scoring results of all subjects in at least one subject are lower than a preset score, for example, the scoring results of all subjects are lower than 60 points, etc., the candidate subject cannot be detected. At this time, the electronic device performs subject detection on the N image materials and generates a first abnormality identification.

[0224] In other embodiments, when an electronic device performs subject detection on N image materials and obtains at least one candidate subject, the specified subject does not match any subject in at least one candidate subject due to reasons such as the absence of the specified subject in at least one candidate subject. At this time, a matching process is performed on at least one candidate subject and the specified subject to obtain a second exception identification.

[0225] Step K: the electronic device determines the position information of the first cropping frame in the image frame according to the position information of the target subject in the image frame and the selected frame ratio.

[0226] The implementation method of step K is the same as that of step E, which will not be repeated here.

[0227] In some embodiments, when the first abnormality mark is obtained in step J, the electronic device obtains the position information of the second cropping frame in the image frame based on the position information of any position of the image frame and the selected frame ratio.

[0228] In other embodiments, after subject detection is performed on N image materials in step J to obtain at least one candidate subject, when a second abnormality mark is obtained, the electronic device obtains position information of the third cropping frame in the image frame based on the position information of any position of the image frame and the selected aspect ratio.

[0229] In step L, the electronic device crops the image frame according to the position information of the first cropping frame in the image frame, scales the cropped image frame to obtain scaled image material, exports the scaled image material and displays it in the gallery.

[0230] The implementation method of step L is the same as that of step I, and will not be repeated here.

[0231] In some embodiments, when the first abnormality mark is obtained in step J, the image frame can also be cropped according to the position information of the second cropping frame in the image frame, and the cropped image frame can be scaled to obtain the scaled image material, and the scaled image material can be exported and displayed in the gallery.

[0232] In some embodiments, when the second abnormality mark is obtained in step J, the image frame can also be cropped according to the position information of the third cropping frame in the image frame, and the cropped image frame can be scaled to obtain the scaled image material, and the scaled image material can be exported and displayed in the gallery.

[0233] The above embodiments describe in detail how the electronic device crops N image materials when the user's operation of selecting the aspect ratio and the operation of specifying the subject are both optional operations through Example 1, Example 2, Example 3 and Example 4. Figure 2 The software architecture in the example 1, example 2, example 3 and example 4 internal implementation process is described in detail.

[0234] The following is attached Fig.15 It is combined with Figure 2 The software architecture in the example describes the internal implementation process of Example 1 in detail.

[0235] Fig.15 2 is a schematic flow chart of an image processing method 200 provided in an embodiment of the present application. The method 200 may be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method 200 is described in detail by taking an electronic device as an example.

[0236] S201. The business layer receives the user's operation on the free creation option.

[0237] It should be understood that the business layer includes a free creation application. The free creation application is correspondingly presented as a free creation option on the interface. When the user clicks the free creation option, the business layer will enable the free creation application.

[0238] S202: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0239] Exemplarily, a free creation option is displayed in the UI interface of the gallery application of the electronic device. When the user clicks on the free creation option, multiple pictures and multiple videos are displayed in the gallery interface, and a confirmation control for the free creation (eg √) is displayed.

[0240] S203: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0241] In one example, after executing S202, the business layer can receive the image material to be cropped selected by the user and receive the user's confirmation to perform the free creation operation. Figure 5 In the gallery interface shown in (c) of the figure, click the free creation option. Figure 5 In the gallery interface shown in (f), select one or more videos and one or more pictures as the image materials to be cropped, and then click the √ control to confirm the execution of the free creation operation. The business layer can receive the image materials selected by the user and receive the user's confirmation to execute the free creation operation.

[0242] In another example, the business layer may directly receive the image material to be cropped selected by the user and receive the user's confirmation to perform the free creation operation without executing steps S201 and S202. Figure 8 In the gallery interface shown in (b) in FIG. 1 , select one or more videos and one or more pictures as the image materials to be cropped, then click the Create option, and Figure 8 After clicking the free creation option in the gallery interface shown in (c), the business layer can receive the image material selected by the user and receive the user's confirmation to perform the free creation operation.

[0243] It should be understood that after the user confirms to execute free creation, the modules at different layers interact to complete the algorithm initialization so as to use the initialized algorithm to crop the image material. The embodiment of the present application does not limit the method of algorithm initialization.

[0244] S204. In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the application function layer.

[0245] S205: The application function layer displays the image material.

[0246] It should be understood that after the user clicks the √ control, the graphical interface displays the interface of the image processing software, for example, Figure 6 (b) to (f) show the graphical interface. Figure 6 The image material is displayed in the preview box in the graphical interface shown in (b) to (f).

[0247] S206: The application function layer receives the user's operation on the smart cropping option.

[0248] For example, the user can Figure 6 In the graphical interface shown in (b) in the figure, click the "Clip" option. Figure 6 (c) in the figure clicks the “Smart Crop” option. At this time, the application function layer receives the user's operation on the Smart Crop option.

[0249] S207. In response to the user's operation on the smart cropping option, the application function layer sends the image material to the parsing module of the media platform framework layer.

[0250] S208: The parsing module parses the image material and generates a parsing file.

[0251] It should be understood that the parsing module can parse the data of the image material into standard corresponding encapsulation format data. The definition and function of the parsing module have been described in the above embodiment and will not be repeated here.

[0252] S209. The parsing module sends the parsing file to the decoding module of the media platform framework layer.

[0253] S210: The decoding module decodes the parsed file to generate a decoded file.

[0254] It should be understood that the decoding module is used to decode the video compression coded data into uncompressed video original data. The definition and function of the decoding module have been described in the above embodiment and will not be repeated here.

[0255] S211. The decoding module sends the decoding file to the software development kit of the smart middle platform framework layer.

[0256] It should be understood that after the decoding module sends the decoded file to the software development kit of the smart middle platform framework layer, the software development kit calls the algorithm module in the intelligent engine framework layer to process the decoded file to solve the problem of black edges in the final saved video.

[0257] S212. The subject detection module in the intelligent engine framework layer detects the subject in the decoded file and generates location information of the target subject.

[0258] It should be understood that the process of detecting the subject in the decoded file and generating the location information of the target subject can refer to Fig. 9 (b) or Fig. 10A (b) or Fig. 10B The description of (b) in FIG. 1 is not repeated here. The position information of the target subject refers to the position information of the target subject in the image frame.

[0259] In some embodiments, when the subject detection module in the intelligent engine framework layer detects the subject in the decoded file, the subject detection module may not be able to detect the subject due to the fact that the subject in the decoded file is missing, does not exist, or the scoring results of multiple subjects are lower than the preset score, for example, the scoring results of multiple subjects are all lower than 60 points. At this time, the subject detection module will consider that the detection event is an abnormal event and will generate a first abnormality flag.

[0260] S213. The subject detection module sends the location information of the target subject to the subject segmentation module.

[0261] In some embodiments, the subject detection module may also send a first abnormality indicator to the subject segmentation module.

[0262] S214: The subject segmentation module generates position information of a first cropping frame according to the position information of the target subject and a preset frame ratio.

[0263] It should be understood that the preset aspect ratio can be stored in a memory, and when the subject segmentation module receives the position information of the target subject, the preset aspect ratio can be called from the memory.

[0264] It should also be understood that the process of generating the position information of the first cropping frame according to the position information of the target subject and the preset frame ratio by the subject segmentation module can be referred to in Fig. 9 (c) or Fig. 10A (c) or Fig. 10B The relevant description of (c) in FIG. 1 is not repeated here. The position information of the first cropping frame refers to the position information of the first cropping frame in the image frame.

[0265] In some embodiments, when the subject segmentation module receives the first abnormality mark, it generates the position information of the second cropping frame according to the position information of any position in the image frame and a preset frame ratio.

[0266] S215. The main body segmentation module in the intelligent engine framework layer returns the position information of the first cropping frame to the application function layer in the application layer.

[0267] For example, the first cropping frame can refer to Fig. 9 The cropping frame C1 shown in (c) can also be referred to Fig. 10A or Fig. 10B The cropping frame C2 is shown in (c).

[0268] In some embodiments, the subject segmentation module returns the position information of the second cropping frame to the application function layer in the application layer.

[0269] S216: The application function layer crops the image material according to the position information of the first cropping frame to generate a cropped image material.

[0270] For example, the cropped image material can refer to Fig. 9 The image material cropped by the cropping frame C1 shown in (c) can also be referred to Fig. 10A or Fig. 10B (c) shows the image material after being cropped using the cropping frame C2.

[0271] In some embodiments, the application function layer may also crop the image material according to the position information of the second cropping frame to generate cropped image material.

[0272] S217: The application function layer sends the cropped image material to the basic function layer.

[0273] S218. The basic function layer performs basic function processing on the cropped image material to obtain the final cropped image material.

[0274] It should be understood that the basic function processing may include scaling processing, for example, Fig. 9 The image material cropped by the cropping frame C1 shown in (c) is enlarged to obtain Fig. 9 (d) shows the final cropped image material.

[0275] For example, the basic functional layer Fig. 10A The image material cropped by the cropping frame C2 shown in (c) is enlarged to obtain Fig. 10A (d) shows the final cropped image material.

[0276] For example, the basic functional layer Fig. 10B The image material cut by the cutting frame C2 shown in (c) is reduced to obtain Fig. 10B (d) shows the final cropped image material.

[0277] S219: The basic function layer instructs the business layer to display the final cropped image material.

[0278] It should be understood that the basic function layer can instruct the business layer to display the final cropped image material in the interface of the image processing software. Figure 6 The image material is displayed in the preview box in the graphical interface shown in (b) to (f).

[0279] In some embodiments, the basic function layer may further instruct the business layer to display image materials in the gallery.

[0280] S220: The business layer displays the final cropped image material.

[0281] For example, the final cropped image material can refer to Fig. 9 or Fig. 10A or Fig. 10B The image material shown in (d) .

[0282] The following is attached Fig.16 It is combined with Figure 2 The software architecture in the example 2 describes the internal implementation process in detail.

[0283] Fig.16 3 is a schematic flow chart of an image processing method 300 provided in an embodiment of the present application. The method 300 may be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method 300 is described in detail by taking an electronic device as an example.

[0284] S301. The business layer receives the user's operation on the free creation option.

[0285] S302: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0286] S303: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0287] S304: In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the application function layer.

[0288] S305: The application function layer displays the image material.

[0289] S306: The application function layer receives the user's operation on the smart cropping option and the operation on selecting the aspect ratio.

[0290] For example, the user can Figure 6 (c) Click the "Smart Crop" option. Figure 6 In the graphical interface shown in (d) of FIG. 1 , for example, the user clicks the 16:9 option, which means that the aspect ratio selected by the user is 16:9. The user clicks the 16:9 option and clicks Figure 6 After the √ control shown in (d) in the figure is clicked, the application function layer receives the user's operation on the smart cropping option and the operation of selecting the aspect ratio.

[0291] S307. In response to the user's operation of selecting an aspect ratio, the application function layer sends the image material and the selected aspect ratio to the parsing module of the media platform framework layer.

[0292] S308: The parsing module parses the image material and generates a parsing file.

[0293] It should be understood that the parsing module can parse the data of the image material into standard corresponding encapsulation format data. The definition and function of the parsing module have been described in the above embodiment and will not be repeated here.

[0294] S309: The parsing module sends the parsing file and the selected aspect ratio to the decoding module of the media platform framework layer.

[0295] S310: The decoding module decodes the parsed file to generate a decoded file.

[0296] S311. The decoding module sends the decoding file and the selected aspect ratio to the software development kit of the smart middle platform framework layer.

[0297] It should be understood that after the decoding module sends the decoded file and the selected aspect ratio to the software development kit of the smart middle platform framework layer, the software development kit calls the algorithm module in the intelligent engine framework layer to process the decoded file to solve the problem of black edges in the final saved video, and the software development kit can forward the selected aspect ratio to the algorithm module in the intelligent engine framework layer.

[0298] S312. The subject detection module in the intelligent engine framework layer detects the subject in the decoded file and generates the location information of the target subject.

[0299] It should be understood that the process of detecting the subject in the decoded file and generating the location information of the target subject can refer to Fig.11 The position information B1 and B3 of the target subject shown in (b) can also be referred to Fig. 12A The position information B2 and B3 of the target subject shown in (b) can also be referred to Fig. 12B (b) shows the position information B1, B2, B3 and B4 of the target subject.

[0300] In some embodiments, when the subject detection module in the intelligent engine framework layer detects the subject in the decoded file, the subject detection module may not be able to detect the subject due to the fact that the subject in the decoded file is missing, does not exist, or the scoring results of multiple subjects are lower than the preset score, for example, the scoring results of multiple subjects are all lower than 60 points. At this time, the subject detection module will consider that the detection event is an abnormal event and will generate a first abnormality flag.

[0301] S313: The subject detection module sends the location information of the target subject and the selected frame ratio to the subject segmentation module.

[0302] In some embodiments, the subject detection module may also send a first abnormality indicator to the subject segmentation module.

[0303] S314: The subject segmentation module generates position information of a first cropping frame according to the position information of the target subject and the selected frame ratio.

[0304] It should be understood that the position information of the first cropping frame generated by the subject segmentation module can refer to Fig.11 C1 and C3 shown in (c) can also be referred to Fig. 12A C2 and C3 shown in (c) can also be referred to Fig. 12B C2 and C3 shown in (c).

[0305] In some embodiments, when the subject segmentation module receives the first abnormality mark, it generates the position information of the second cropping frame according to the position information of any position in the image frame and the selected frame ratio.

[0306] S315. The main body segmentation module in the intelligent engine framework layer returns the position information of the first cropping frame to the application function layer in the application layer.

[0307] For example, the first cropping frame can refer to Fig.11 The cropping frames C1 and C3 shown in (c) can also be referred to Fig. 12A or Fig. 12B (c) shows the cropping boxes C2 and C3.

[0308] In some embodiments, the subject segmentation module may also return the position information of the second cropping frame to the application function layer in the application layer.

[0309] S316: The application function layer crops the image material according to the position information of the first cropping frame to generate a cropped image material.

[0310] For example, the cropped image material can refer to Fig.11 The image material cropped by the cropping frame C1 and the image material cropped by the cropping frame C3 shown in (c) in FIG. 1 can also be referred to as Fig. 12A or Fig. 12B (c) shows the image material cropped using the cropping frame C2 and the image material cropped using the cropping frame C3.

[0311] In some embodiments, the application function layer may also crop the image material according to the position information of the second cropping frame to generate cropped image material.

[0312] S317: The application function layer sends the cropped image material to the basic function layer.

[0313] S318. The basic function layer performs basic function processing on the cropped image material to obtain the final cropped image material.

[0314] It should be understood that the basic function processing may include scaling processing, for example, Fig.11 The image material cropped by the cropping frame C1 and the image material cropped by the cropping frame C3 shown in (c) are enlarged to obtain Fig.11 (d) shows the final cropped image material.

[0315] For example, the basic functional layer Fig. 12A The image material cropped by the cropping frame C2 and the image material cropped by the cropping frame C3 shown in (c) are enlarged to obtain Fig. 12A (d) shows the final cropped image material.

[0316] For example, the basic functional layer Fig. 12B The image material cropped by the cropping frame C2 and the image material cropped by the cropping frame C3 shown in (c) are reduced to obtain Fig. 12B (d) shows the final cropped image material.

[0317] S319: The basic function layer instructs the business layer to display the final cropped image material.

[0318] S320: The business layer displays the final cropped image material.

[0319] For example, the final cropped image material can refer to Fig.11 (d) or Fig. 12A (d) or Fig. 12B (d) The final cropped image material.

[0320] The following is attached Fig.17 It is combined with Figure 2 The software architecture in the example describes the internal implementation process of Example 3 in detail.

[0321] Fig.17 4 is a schematic flow chart of an image processing method 400 provided in an embodiment of the present application. The method 400 may be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method 400 is described in detail by taking an electronic device as an example.

[0322] S401. The business layer receives the user's operation on the free creation option.

[0323] S402: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0324] S403: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0325] S404: In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the application function layer.

[0326] S405: The application function layer displays the image material.

[0327] S406: The application function layer receives the user's operation on the smart cropping option and the operation on specifying the subject.

[0328] For example, the user can Figure 6 (c) Click the "Smart Crop" option in the Figure 6 Click the cross option in the gallery interface of (d) in the pop-up window. Figure 6 Click the check mark option in the gallery interface of (e) in the pop-up Figure 6 In the gallery interface of (f), manually select the subject and click the check mark option. Another example: When the user clicks Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6 Click the cross option in the gallery interface of (d) in the pop-up window. Figure 7In the gallery interface of (a) in the figure, specify the subject. For example, click the "Head" option. Figure 7 In the gallery interface of (c), the user clicks the check mark option. After the user clicks the check mark option, the application function layer receives the user's operation on the smart cropping option and the operation on the specified subject.

[0329] It should be understood that the user can specify one or more subjects in the image material as the designated subject, and the embodiment of the present application is not limited to this.

[0330] S407. In response to the user's operation of specifying a subject, the application function layer sends the image material and the specified subject information to the parsing module of the media platform framework layer.

[0331] S408: The parsing module parses the image material and generates a parsing file.

[0332] S409, the parsing module sends the parsing file and the specified subject information to the decoding module of the media platform framework layer.

[0333] S410: The decoding module decodes the parsed file to generate a decoded file.

[0334] S411. The decoding module sends the decoding file and specified subject information to the software development kit of the smart middle platform framework layer.

[0335] It should be understood that after the decoding module sends the decoded file and specified subject information to the software development kit of the smart middle platform framework layer, the software development kit calls the algorithm module in the intelligent engine framework layer to process the decoded file to solve the problem of black edges in the final saved video, and the software development kit can forward the specified subject information to the algorithm module in the intelligent engine framework layer.

[0336] S412. The subject detection module in the intelligent engine framework layer detects the subject in the decoded file to obtain at least one candidate subject.

[0337] It should be understood that the implementation process of S412 can refer to the implementation process of step G, which will not be repeated here.

[0338] In some embodiments, when the subject detection module in the intelligent engine framework layer detects the subject in the decoded file, the subject detection module may not be able to detect the subject due to the fact that the subject in the decoded file is missing, does not exist, or the scoring results of multiple subjects are lower than the preset score, for example, the scoring results of multiple subjects are all lower than 60 points. At this time, the subject detection module will consider that the detection event is an abnormal event and will generate a first abnormality flag.

[0339] S413: The subject detection module sends candidate subject and designated subject information to the subject tracking module.

[0340] In some embodiments, the subject detection module may also send a first abnormality indicator to the subject segmentation module.

[0341] S414. The subject tracking module performs matching processing on at least one candidate subject and the designated subject. When the designated subject successfully matches at least part of the at least one candidate subject, the position information of the designated subject in the image frame is determined as the position information of the target subject in the image frame.

[0342] The implementation process of S414 can refer to the implementation process of step G, which will not be repeated here.

[0343] In some embodiments, due to reasons such as the absence of a designated subject in at least one candidate subject, the designated subject does not match any subject in at least one candidate subject. At this time, matching processing is performed on at least one candidate subject and the designated subject to obtain a second exception identifier.

[0344] In some embodiments, when the subject tracking module receives the first abnormality identifier sent by the subject detection module, the step S414 is not executed, and the first abnormality identifier is directly forwarded to the subject segmentation module.

[0345] S415. The subject tracking module sends the location information of the target subject to the subject segmentation module.

[0346] In some embodiments, the subject tracking module may also send a first anomaly identifier to the subject segmentation module.

[0347] In other embodiments, the subject tracking module may further send a second abnormality indicator to the subject segmentation module.

[0348] S416: The subject segmentation module generates position information of a first cropping frame according to the position information of the target subject and a preset frame ratio.

[0349] It should be understood that the implementation process of S416 is the same as the implementation process of step H, and will not be repeated here.

[0350] In some embodiments, when the subject segmentation module receives the first abnormality mark, it generates the position information of the second cropping frame according to the position information of any position in the image frame and a preset frame ratio.

[0351] In other embodiments, when the subject segmentation module receives the second abnormality mark, it generates the position information of the third cropping frame according to the position information of any position in the image frame and a preset frame ratio.

[0352] S417. The main body segmentation module in the intelligent engine framework layer returns the position information of the first cropping frame to the application function layer in the application layer.

[0353] For example, the first cropping frame can refer to Fig.13 (c) or Fig.14A (c) or Fig. 14B (c) shows the position information C2 of the first cropping frame.

[0354] In some embodiments, the body segmentation module may return the position information of the second cropping frame to the application function layer in the application layer.

[0355] In some other embodiments, the subject segmentation module may also return the position information of the third cropping frame to the application function layer in the application layer.

[0356] S418: The application function layer crops the image material according to the position information of the first cropping frame to generate a cropped image material.

[0357] For example, the cropped image material can refer to Fig.13 or Fig.14A or Fig. 14B (c) shows the image material after being cropped using the cropping frame C2.

[0358] In some embodiments, the application function layer crops the image material according to the position information of the second cropping frame to generate the cropped image material.

[0359] In other embodiments, the application function layer crops the image material according to the position information of the third cropping frame to generate the cropped image material.

[0360] S419: The application function layer sends the cropped image material to the basic function layer.

[0361] S420: The basic function layer performs basic function processing on the cropped image material to obtain the final cropped image material.

[0362] It should be understood that the basic function processing may include scaling processing, for example, Fig.13 or Fig.14A or Fig. 14B The image material cropped by the cropping frame C2 shown in (c) is enlarged to obtain Fig.13 or Fig.14A or Fig. 14B (d) shows the final cropped image material.

[0363] S421. The basic function layer instructs the business layer to display the final cropped image material.

[0364] S422: The business layer displays the final cropped image material.

[0365] The following is attached Fig.18 It is combined with Figure 2 The software architecture in the example 4 describes the internal implementation process in detail.

[0366] Fig.18 1 is a schematic flow chart of an image processing method 500 provided in an embodiment of the present application. The method 500 may be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method 500 is described in detail by taking an electronic device as an example.

[0367] S501. The business layer receives the user's operation on the free creation option.

[0368] S502: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0369] S503: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0370] S504: In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the application function layer.

[0371] S505: The application function layer displays the image material.

[0372] S506: The application function layer receives the user's operation on the smart cropping option, the operation on selecting the aspect ratio, and the operation on specifying the subject.

[0373] For example, after clicking the "Smart Crop" option, the user can select the aspect ratio and specify the subject to crop the image material. Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6 In the gallery interface (d), select the aspect ratio. For example, click the "16:9" option and then click the check mark option. Figure 6 Click the check mark option in the gallery interface of (e) in the pop-up Figure 6 In the gallery interface of (f), manually select the subject and click the check mark option. Another example: When the user clicks Figure 6 After the "Smart Crop" option is shown in (c), the Figure 6 Select the aspect ratio in the gallery interface (d) in the pop-up window. Figure 7 In the gallery interface of (a) in the figure, specify the subject. For example, click the "Head" option. Figure 7 Click the check mark option in the gallery interface (c).

[0374] It should be understood that the user can specify one or more subjects in the image material as the designated subject, and the embodiment of the present application is not limited to this.

[0375] S507. In response to the user's operation on the intelligent cropping option, the operation of selecting the aspect ratio, and the operation of specifying the subject, the application function layer sends the image material, the selected aspect ratio, and the specified subject information to the parsing module of the media platform framework layer.

[0376] S508: The parsing module parses the image material and generates a parsing file.

[0377] S509. The parsing module sends the parsing file, the selected aspect ratio and the specified subject information to the decoding module of the media platform framework layer.

[0378] S510: The decoding module decodes the parsed file to generate a decoded file.

[0379] S511. The decoding module sends the decoding file, the selected aspect ratio and the specified subject information to the software development kit of the smart middle platform framework layer.

[0380] It should be understood that after the decoding module sends the decoded file, the selected aspect ratio and the specified subject information to the software development kit of the smart middle platform framework layer, the software development kit calls the algorithm module in the intelligent engine framework layer to process the decoded file to solve the problem of black edges in the final saved video, and the software development kit can forward the selected aspect ratio and specified subject information to the algorithm module in the intelligent engine framework layer.

[0381] S512. The subject detection module in the intelligent engine framework layer detects the subject in the decoded file to obtain at least one candidate subject.

[0382] It should be understood that the implementation process of S512 can refer to the implementation process of step J, which will not be repeated here.

[0383] In some embodiments, when the subject detection module in the intelligent engine framework layer detects the subject in the decoded file, the subject detection module may not be able to detect the subject due to the fact that the subject in the decoded file is missing, does not exist, or the scoring results of multiple subjects are lower than the preset score, for example, the scoring results of multiple subjects are all lower than 60 points. At this time, the subject detection module will consider that the detection event is an abnormal event and will generate a first abnormality flag.

[0384] S513: The subject detection module sends the candidate subject, the selected frame ratio and the designated subject information to the subject tracking module.

[0385] In some embodiments, the subject detection module may also send a first abnormality indicator to the subject segmentation module.

[0386] S514. The subject tracking module performs matching processing on at least one candidate subject and the designated subject. When the designated subject successfully matches at least part of the at least one candidate subject, the position information of the designated subject in the image frame is determined as the position information of the target subject in the image frame.

[0387] The implementation process of S514 can refer to the implementation process of step J, which will not be repeated here.

[0388] In some embodiments, due to reasons such as the absence of a designated subject in at least one candidate subject, the designated subject does not match any subject in at least one candidate subject. At this time, matching processing is performed on at least one candidate subject and the designated subject to obtain a second exception identifier.

[0389] In some embodiments, when the subject tracking module receives the first abnormality identifier sent by the subject detection module, the step S514 is not executed, and the first abnormality identifier is directly forwarded to the subject segmentation module.

[0390] S515. The subject tracking module sends the location information of the target subject and the selected frame ratio to the subject segmentation module.

[0391] In some embodiments, the subject tracking module may also send a first anomaly identifier to the subject segmentation module.

[0392] In other embodiments, the subject tracking module may further send a second abnormality indicator to the subject segmentation module.

[0393] S516: The subject segmentation module generates position information of a first cropping frame according to the position information of the target subject and the selected frame ratio.

[0394] It should be understood that the implementation process of S516 is the same as the implementation process of step K, and will not be repeated here.

[0395] In some embodiments, when the subject segmentation module receives the first abnormality mark, it generates the position information of the second cropping frame according to the position information of any position in the image frame and a preset frame ratio.

[0396] In other embodiments, when the subject segmentation module receives the second abnormality mark, it generates the position information of the third cropping frame according to the position information of any position in the image frame and a preset frame ratio.

[0397] S517. The main body segmentation module in the intelligent engine framework layer returns the position information of the first cropping frame to the application function layer in the application layer.

[0398] In some embodiments, the body segmentation module may return the position information of the second cropping frame to the application function layer in the application layer.

[0399] In some other embodiments, the subject segmentation module may also return the position information of the third cropping frame to the application function layer in the application layer.

[0400] S518: The application function layer crops the image material according to the position information of the first cropping frame to generate a cropped image material.

[0401] In some embodiments, the application function layer crops the image material according to the position information of the second cropping frame to generate the cropped image material.

[0402] In other embodiments, the application function layer crops the image material according to the position information of the third cropping frame to generate the cropped image material.

[0403] S519: The application function layer sends the cropped image material to the basic function layer.

[0404] S520: The basic function layer performs basic function processing on the cropped image material to obtain the final cropped image material.

[0405] S521. The basic function layer instructs the business layer to display the final cropped image material.

[0406] S522: The business layer displays the final cropped image material.

[0407] The above embodiments are combined with the attached Figure 2 The software architecture in the embodiment describes the internal implementation process of Example 1, Example 2, Example 3 and Example 4 in detail. Fig.18 The internal implementation process of another image processing method provided by the present application is introduced in detail.

[0408] Fig.19 6 is a schematic flow chart of an image processing method 600 provided in an embodiment of the present application. The method 600 may be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method 600 is described in detail by taking an electronic device as an example.

[0409] S601: The electronic device displays a first interface.

[0410] It should be understood that the first interface refers to an interface in a gallery application of an electronic device. For example, the first interface may be Figure 6 (b) shows the gallery interface.

[0411] It should also be understood that the first interface includes N image materials to be cropped, at least two of the N image materials have different aspect ratios, and N is an integer greater than or equal to 2. For example, Fig. 9 (a) in FIG. 2 shows that there are two image materials to be cropped, and the aspect ratios of the two image materials are different. For another example, there are three image materials to be cropped, namely, the first video segment, the second video segment, and the third video segment, wherein the aspect ratio of the first video segment is 16:9, the aspect ratio of the second video segment is 4:3, and the aspect ratio of the third video segment is 4:3.

[0412] S602: In response to a user's operation of cropping N image materials, each image frame of each image material in at least part of the N image materials is processed according to a first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio.

[0413] It should be understood that the user's operation of cropping N image materials may include the user clicking on the smart cropping option, the user selecting the aspect ratio, the user specifying the subject, etc. The above operations of the user can refer to Figures 5 to 9 The relevant operations in the GUI are shown.

[0414] It should also be understood that the first aspect ratio may be a preset aspect ratio, and the preset aspect ratio is the aspect ratio of any image material among N image materials. For example, the N image materials include 2 videos, the aspect ratio of the first video is 9:16, and the aspect ratio of the second video is 16:9. The preset aspect ratio may be the aspect ratio of the first video.

[0415] The first aspect ratio may also be an aspect ratio generated in response to a user's operation of cropping N image materials. Exemplary: in response to a user's operation of the smart cropping option of N image materials, a third interface including multiple aspect ratios is displayed; in response to a user's operation of selecting the first aspect ratio, each image frame of each image material in at least part of the N image materials is processed according to the first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio.

[0416] In this example, the user's operation on the smart cropping options of N image materials can be an operation in which the user clicks on the smart cropping option, for example Figure 6 (c) in FIG. 1 shows an operation in which a user clicks on the smart crop option. The third screen in the example can be referred to as Figure 6 The graphical interface shown in (d) includes multiple aspect ratios. The user selects any aspect ratio and clicks the check mark option to complete the operation of selecting the first aspect ratio.

[0417] In the embodiment of the present application, processing each image frame of each image material in at least part of the N image materials so that the aspect ratios of the N image materials are all the first aspect ratio includes:

[0418] Each image frame of each image material in a portion of the N image materials is processed so that the aspect ratios of the N image materials are all the first aspect ratio. Fig. 9 , Fig. 10A and Fig. 10B The illustrated process of processing the image material is to process each image frame in the second video of the two videos, so that the aspect ratios of the two videos are both the first aspect ratio. It should also be understood that the partial image material can be one image material or multiple image materials.

[0419] It should be understood that when at least part of the image material includes part of the N image materials, the aspect ratio of the image materials other than the part of the N image materials is the first aspect ratio. In layman's terms, when processing part of the N image materials, the aspect ratio of this part of the image material is different from the aspect ratio of the image material after the final processing (the first aspect ratio), and the aspect ratio of the image materials other than the part of the N image materials is the first aspect ratio and does not need to be processed. Fig. 9 , Fig. 10A and Fig. 10B , when only each image frame in the second video of the two videos is processed, the aspect ratio of the first video is the same as the first aspect ratio, and the aspect ratio of the second video is different from the first aspect ratio.

[0420] In the embodiment of the present application, when processing each image frame of each image material in some image materials among N image materials so that the aspect ratios of the N image materials are all the first aspect ratio, since there is no need to process all the image materials among the N image materials, the computing resources of the electronic device can be saved and the processing speed of the electronic device can be improved.

[0421] In the embodiment of the present application, processing each image frame of each image material in at least part of the N image materials so that the aspect ratios of the N image materials are all the first aspect ratio includes:

[0422] Each image frame of each image material in all the image materials in the N image materials is processed so that the aspect ratios of the N image materials are all the first aspect ratio. Fig.11 , Fig. 12A and Fig. 12BThe process of processing the image materials shown is to process each image frame of all the videos in the two videos, so that the aspect ratios of the two videos are both the first aspect ratio. It should be understood that in the embodiment of the present application, regardless of whether the N image materials include image materials with the aspect ratio of the first aspect ratio, each image frame of each image material in all the image materials in the N image materials can be processed.

[0423] In the embodiment of the present application, the process of the electronic device processing the image frame includes:

[0424] When the electronic device performs subject detection on an image frame to obtain first position information, the electronic device obtains second position information indicating the position of a first cropping frame in the image frame based on the first position information and the first aspect ratio, the first cropping frame includes the target subject, and the aspect ratio of the first cropping frame is the first aspect ratio; the image frame is cropped according to the second position information to obtain a first image frame, the first image frame includes the target subject, and the aspect ratio of the first image frame is the first aspect ratio.

[0425] It should be understood that when the electronic device performs subject detection on the image frame, if there is at least one subject in the image frame, and the at least one subject is not missing all subjects, and the at least one subject is not that all subjects have a scoring result lower than a preset score, then the electronic device performs subject detection on the image frame and can obtain first position information indicating the position of the target subject in the image frame. For the implementation method of how to obtain the first position information, reference can be made to the implementation method of how to obtain the position information of the target subject in the image frame in the above embodiment step A, step D, step G and step J, which will not be repeated here.

[0426] After the electronic device obtains the first position information, it can obtain the second position information indicating the position of the first cropping frame in the image frame based on the first position information and the first frame ratio. The implementation method of the electronic device obtaining the second position information can refer to the implementation method of step B, step E, step H and step K, which will not be repeated here.

[0427] In some embodiments, the electronic device performs cropping processing on the image frame according to the second position information to obtain the first image frame, including:

[0428] The electronic device crops the image frame according to the second position information and scales the cropped image frame to obtain a first image frame.

[0429] It should be understood that the implementation method of scaling the cropped image frame can refer to the implementation methods in step C, step F, step I and step L, which will not be repeated here.

[0430] S603: The electronic device displays a second interface including N image materials with a first aspect ratio.

[0431] It should be understood that the second interface is also an interface of the gallery application. After the electronic device executes S602, the cropped image material can be viewed in the second interface. For example, Figure 6 The cropped image material is viewed in the preview box of the interface of the gallery application shown in (f). In this example, Figure 6 The interface of the gallery application shown in (f) is the second interface.

[0432] In some embodiments, the second interface may also be Figure 5 In the interface of the gallery application shown in (d), after the electronic device executes S602, the cropped image material can be stored in the gallery, and the user can Figure 5 View the cropped image material in the interface shown in (d).

[0433] Usually, since at least two of the N image materials have different aspect ratios, black edges may appear in the spliced ​​N image materials. Therefore, in the embodiment of the present application, when cropping the N image materials to be cropped, each image frame of each image material in at least part of the N image materials may be processed so that the aspect ratios of the N image materials are the same, that is, the aspect ratios of the N image materials are all the first aspect ratio. Since the aspect ratios of the processed N image materials are all the same, black edges may be avoided in the processed image materials that are finally saved.

[0434] Furthermore, since the embodiment of the present application can perform subject detection on the image frame when cropping the N image materials to be cropped, when the position information of the target subject in the image frame can be detected, the position information of the first cropping frame is generated according to the position information of the target subject and the first frame ratio, and then the image frame is cropped according to the position information of the first cropping frame, that is, when the subject is detected on the image frame to obtain the first position information indicating the position of the target subject in the image frame, based on the first position information and the first frame ratio, the second position information indicating the position of the first cropping frame in the image frame is obtained, wherein the first cropping frame includes the target subject, and then the image frame is cropped according to the second position information to obtain the first image frame, wherein the first image frame includes the target subject. Therefore, compared with the cropping scheme that usually cropping the image material at a fixed position will result in missing or lost subject, since the first image frame includes the target subject, it is ensured that the processed image material finally saved includes the target subject, avoiding the occurrence of missing or lost subject, thereby improving the user experience.

[0435] Since there is at least one subject in the image frame, the following embodiments describe in detail the internal implementation process of determining the first position information when the image frame contains one subject or multiple subjects.

[0436] In some embodiments, the image frame includes a subject; and the method 600 further includes:

[0437] The electronic device performs subject detection on the image frame to obtain position information of the subject in the image frame, and determines the position information of the subject in the image frame as first position information.

[0438] It should be understood that the image frame in this embodiment only includes one subject, and the electronic device can detect the subject through a subject detection algorithm. The implementation method of including one subject in the image frame, detecting the subject by the electronic device, and determining the first position information can refer to the above embodiment, which will not be repeated here. Fig. 9 The first position information B1 shown in (b) is Fig.11 The first position information B1 and B3 shown in (b) are Fig.13 The first position information B2 shown in (b) etc.

[0439] In some embodiments, the image frame includes a plurality of subjects; and the method 600 further includes:

[0440] The electronic device performs subject detection on the image frame to obtain multiple third position information corresponding to the multiple subjects, each third position information is used to indicate the position of the corresponding subject in the image frame; based on the multiple third position information, the first position information is obtained, and the target subject includes at least part of the multiple subjects.

[0441] It should be understood that the third position information refers to the position of any subject in the image frame. In this embodiment, the image frame may contain multiple subjects, and the electronic device may detect the multiple subjects through a subject detection algorithm, and based on the multiple subjects, obtain multiple third position information corresponding to the multiple subjects. Please refer to the above embodiment, and will not be repeated here. Fig. 10A A1 and A2 shown in (a) are candidate subject masks of the second video segment, and these two candidate subject masks are two third position information corresponding to the two subjects one by one. Fig. 12A A1 and A2 shown in (a) are candidate subject masks of the second video, and these two candidate subject masks are two third position information corresponding one to one between the two subjects in the second video. A3 and A4 are candidate subject masks of the first video, and these two candidate subject masks are two third position information corresponding one to one between the two subjects in the first video.

[0442] Typically, when an electronic device performs subject detection on an image frame, regardless of whether there are one or more subjects in the image frame, one position information is used to indicate the position of one or more subjects in the image frame, resulting in an inability to detect the position of each subject in the image frame when there are multiple subjects in the image frame. However, the electronic device in the embodiment of the present application performs subject detection on the image frame and can obtain multiple third position information corresponding to the multiple subjects. Compared with the conventional subject detection scheme, since the third position information of each of the multiple subjects in the image frame can be obtained during detection, the detection effect when performing subject detection on the image frame can be improved, and the third position information of each subject can be obtained more accurately. Therefore, when the electronic device obtains the first position information based on the multiple third position information, the accuracy of obtaining the first position information can be improved.

[0443] After obtaining the multiple pieces of third location information, the electronic device may use one or more pieces of the multiple pieces of third location information as the first location information.

[0444] Exemplarily, the electronic device obtains the first location information based on the plurality of third location information, further comprising:

[0445] The electronic device scores the multiple subjects based on multiple subject information corresponding one-to-one to the multiple subjects, and obtains multiple score values ​​for the multiple subjects, where one subject corresponds to one score value, and each subject information includes image feature information and image quality information of each subject; the subject corresponding to the score value that meets preset conditions in the multiple score values ​​is determined as the target subject, and the preset conditions include: the score value is greater than or equal to the preset score value; based on multiple third position information, the third position information corresponding to each subject in the target subject is determined as the first position information.

[0446] It should be understood that the electronic device can score the detected multiple subjects based on the image color features, image texture features, image quality and other information of the detected multiple subjects. The scoring result can be used as a basis for evaluating whether a subject is a target subject. For example, when the scoring result of one or more subjects is greater than or equal to 60 points, the one or more subjects are target subjects. The electronic device can determine the location information of the one or more subjects as the first location information. For example: Fig. 10A The multiple subjects shown in (a) are A1 and A2, and A1 and A2 are scored respectively. Assuming that A2 is greater than 60 points, then A2 is the target subject, and the electronic device can determine the location information of A2 as the first location information. For example, Fig. 10A B2 shown in (b) is the first position information. For example: Fig. 10BThe multiple subjects of the second video shown in (a) are A1 and A2, and A1 and A2 are scored respectively. Assuming that A2 and A1 are both greater than 60 points, then A2 and A1 are both target subjects, and the electronic device can determine the position information of A1 and A2 as the first position information. For example, Fig. 10B B1 and B2 shown in (b) are both first position information.

[0447] Usually, when an electronic device determines a target subject among multiple subjects, it can randomly determine any subject among the multiple subjects as the target subject, which may result in a low image quality of the image frame finally displayed due to the low image quality at the position where the target subject is located in the image frame. In the embodiment of the present application, when the electronic device determines the target subject, it can score multiple subjects and determine the target subject according to the scoring results. Compared with the method of randomly determining the target subject among multiple subjects, the target subject can be accurately determined. In addition, the embodiment of the present application scores the subject based on the subject information (the subject information includes image feature information and image quality information), and determines the subject corresponding to the score value greater than or equal to the preset score value among multiple score values ​​as the target subject, which can improve the image quality at the position where the target subject is located in the image frame. Since the present application improves the image quality at the position where the target subject is located in the image frame, the image quality of the image frame finally displayed containing the target subject can be improved.

[0448] Exemplarily, the electronic device obtains the first location information based on the plurality of third location information, further comprising:

[0449] Any third position information among the plurality of third position information is determined as the first position information.

[0450] For example, Fig. 10A The multiple subjects shown in (a) are A1 and A2, respectively, A1 and A2 represent the third position information of the two subjects respectively, and A1 or A2 can be used as the position information for indicating the target subject in the image frame.

[0451] After obtaining the multiple pieces of third location information, the electronic device may use one piece of the multiple pieces of third location information as the first location information.

[0452] Exemplarily, the electronic device obtains the first location information based on the plurality of third location information, further comprising:

[0453] Based on multiple subject information corresponding one-to-one to the multiple subjects, the multiple subjects are scored to obtain multiple score values ​​of the multiple subjects, one subject corresponds to one score value, and each subject information includes image feature information and image quality information of each subject; the subject corresponding to the score value that meets the preset conditions in the multiple score values ​​is determined as the target subject, and the preset conditions include: the score value is greater than or equal to the preset score value; based on multiple third position information, the third position information corresponding to each subject in the target subject is determined as the first position information.

[0454] It should be understood that the electronic device can score the detected multiple subjects based on the image color features, image texture features, image quality and other information of the detected multiple subjects. The scoring result can be used as a basis for evaluating whether a subject is a target subject. For example, when the scoring result of one or more subjects is greater than or equal to 60 points, the one or more subjects are target subjects. The electronic device can determine the location information of the one or more subjects as the first location information. For example: Fig. 10A The multiple subjects shown in (a) are A1 and A2, and A1 and A2 are scored respectively. Assuming that A2 is greater than 60 points, then A2 is the target subject, and the electronic device can determine the location information of A2 as the first location information. For example, Fig. 10A B2 shown in (b) is the first position information. For example: Fig. 10B The multiple subjects of the second video shown in (a) are A1 and A2, and A1 and A2 are scored respectively. Assuming that A2 and A1 are both greater than 60 points, then A2 and A1 are both target subjects, and the electronic device can determine the position information of A1 and A2 as the first position information. For example, Fig. 10B B1 and B2 shown in (b) are both first position information.

[0455] Exemplarily, the electronic device obtains the first location information based on the plurality of third location information, further comprising:

[0456] Any third position information among the plurality of third position information is determined as the first position information.

[0457] For example, Fig. 10A The multiple subjects shown in (a) are A1 and A2, respectively, A1 and A2 represent the third position information of the two subjects respectively, and A1 or A2 can be used as the position information for indicating the target subject in the image frame.

[0458] Exemplarily, the electronic device obtains the first location information based on the plurality of third location information, further comprising:

[0459] All third position information among the plurality of third position information are determined as the first position information.

[0460] For example, Fig. 12B A3 and A4 shown in (a) are candidate subject masks of the first video, and the two candidate subject masks are two third position information corresponding to the two subjects in the first video. A1 and A2 are candidate subject masks of the first video, and the two candidate subject masks are two third position information corresponding to the two subjects in the first video. The electronic device can determine A3 and A4 of the first video and A1 and A2 of the second video as the first position information. For example, Fig. 12B B1, B2, B3 and B4 shown in (b) are all first position information.

[0461] In some embodiments, if there is no subject in the image frame, or there is at least one subject but the at least one subject is missing (for example, a part of the subject is missing), or the scoring results of all subjects are lower than the preset score, then the electronic device cannot detect the subject when performing subject detection on the image frame, and an abnormal flag will be generated. Therefore, method 600 also includes:

[0462] When the electronic device performs subject detection on an image frame and obtains a first abnormal identification, the electronic device obtains fourth position information based on the position information of any position of the image frame and the first aspect ratio, where the fourth position information is used to indicate the position of the second cropping frame in the image frame, the first abnormal identification is used to indicate that the subject in the image frame is lost or missing, and the aspect ratio of the second cropping frame is the first aspect ratio; the image frame is cropped according to the fourth position information to obtain a second image frame, and the aspect ratio of the second image frame is the first aspect ratio.

[0463] It should be understood that the first abnormal flag refers to the flag of the electronic device when it cannot detect the subject in the subject detection of the image frame. When the electronic device obtains the first abnormal flag, the electronic device can obtain the fourth position information based on the position information of any position of the image frame (for example, the position information of the center position of the image frame) and the first frame ratio. The implementation method of the electronic device obtaining the fourth position information has been described in the above embodiment and will not be repeated here.

[0464] The implementation manner in which the electronic device crops the image frame according to the fourth position information to obtain the second image frame has been described in the above embodiment and will not be repeated here.

[0465] In some embodiments, the method 600 further includes:

[0466] The electronic device performs subject detection on the image frame to obtain at least one candidate subject; performs matching processing on the at least one candidate subject and the designated subject; when the designated subject successfully matches at least part of the at least one candidate subject, determines the position of the designated subject in the image frame as the first position information, wherein the designated subject is the target subject.

[0467] It should be understood that the designated subject is the subject selected by the user when performing a cropping operation on N image materials. The cropping operation includes the operation of designating the subject. For the operation of designating the subject by the user, reference can be made to the description of the above examples 3 and 4, which will not be repeated here.

[0468] In the embodiment of the present application, after the user performs the operation of specifying the subject, the electronic device needs to match the subject in the image frame with the specified subject during the process of detecting the subject, and finally obtain the position information of the target subject in the image frame, that is, the first position information. The implementation method of the electronic device determining the first position information in this embodiment has been stated in step G and step J, and will not be repeated here.

[0469] Usually, after an electronic device crops N image materials, it is unable to display the subject that the user wants to display in the cropped image materials, thereby reducing the user experience. However, in the embodiment of the present application, since the user can specify the subject (the specified subject is the subject that the user wants to display) when cropping N image materials, and then after subject detection is performed on the image frame to obtain at least one candidate subject, the at least one candidate subject is matched with the specified subject. When the specified subject successfully matches at least part of the at least one candidate subject, the position of the specified subject in the image frame is determined as the position information of the target subject in the image frame (i.e., the first position information). Subsequently, the image frame is processed according to the first position information, so that the subject that the user wants to display can be displayed in the cropped image material, thereby improving the user experience.

[0470] It should also be understood that, in the process of matching the subject in the image frame with the designated subject, there may be a situation where no subject in the image frame matches the designated subject. Therefore, in some embodiments, the method 600 further includes:

[0471] The electronic device performs subject detection on an image frame to obtain at least one candidate subject; performs matching processing on the at least one candidate subject and a designated subject; obtains a second abnormality mark when the designated subject does not match any of the at least one candidate subject; obtains fifth position information based on position information of any position of the image frame and the first aspect ratio when the subject detection on the image frame obtains the second abnormality mark, the fifth position information is used to indicate the position of a third cropping frame in the image frame, the aspect ratio of the third cropping frame is the first aspect ratio; the image frame is cropped according to the fifth position information to obtain a third image frame, the aspect ratio of the third image frame is the first aspect ratio.

[0472] It should be understood that the second abnormal identification refers to an identification that no subject in the image frame matches the specified subject. In the case where the electronic device obtains the second abnormal identification, the implementation method of the electronic device obtaining the fifth position information based on the position information of any position of the image frame and the first frame ratio has been stated above and will not be repeated here. Subsequently, the electronic device crops the image frame according to the fifth position information to obtain the third image frame. The implementation method of the electronic device obtaining the third image frame has been stated in the above embodiment and will not be repeated here.

[0473] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0474] The electronic device provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0475] Fig. 20 is an exemplary block diagram of an electronic device 700 provided in an embodiment of the present application. The electronic device 700 includes a processing unit 71, and the processing unit 71 is used to perform the following operations:

[0476] Displaying a first interface, including N image materials to be cropped, at least two of the N image materials have different aspect ratios, and N is an integer greater than or equal to 2; in response to a user's operation of cropping the N image materials, processing each image frame of each image material in at least some of the N image materials according to the first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio, wherein when at least some of the image materials include some of the N image materials, the aspect ratios of the image materials other than the some of the N image materials are the first aspect ratio; displaying a second interface, including the N image materials having the first aspect ratio;

[0477] The process of processing the image frame includes:

[0478] When subject detection is performed on an image frame to obtain first position information, second position information for indicating a position of a first cropping frame in the image frame is obtained based on the first position information and the first frame ratio, the first position information being used to indicate a position of a target subject in the image frame, the first cropping frame including the target subject, and the frame ratio of the first cropping frame being the first frame ratio; the image frame is cropped according to the second position information to obtain a first image frame, the first image frame including the target subject, and the frame ratio of the first image frame being the first frame ratio.

[0479] It should be understood that the processing unit 71 can be used to execute each step performed by the electronic device in the method 600. For specific descriptions, please refer to the relevant descriptions above and will not be repeated here.

[0480] In addition, the processing unit 71 can also be used to execute each step performed by the electronic device in method 200, method 300, method 400, or method 500. For specific descriptions, please refer to the relevant descriptions above and will not be repeated here.

[0481] It should be understood that the electronic device 700 is embodied in the form of a functional unit. The term "unit" herein may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0482] In the embodiments of the present application, Figure 7 The electronic device in the embodiment may also be a chip or a chip system, for example, a system on chip (SoC).

[0483] Fig.21 8 is a schematic structural diagram of an electronic device 800 provided in an embodiment of the present application. The electronic device 800 is used to execute the corresponding steps and / or processes in the above method embodiment.

[0484] The electronic device 800 includes a processor 801, a transceiver 802, and a memory 803. The processor 801, the transceiver 802, and the memory 803 communicate with each other through an internal connection path, and the processor 801 can implement the functions of the processing unit 71 in various possible implementations of the electronic device 700. The memory 803 is used to store instructions, and the processor 801 is used to execute the instructions stored in the memory 803, or in other words, the processor 801 can call these stored instructions to implement the functions of the processing unit 71 in the electronic device 700.

[0485] Optionally, the memory 803 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. The processor 801 may be used to execute instructions stored in the memory, and when the processor 801 executes instructions stored in the memory, the processor 801 is used to execute the various steps and / or processes of the above-mentioned method embodiment corresponding to the electronic device.

[0486] The processor 801 is configured to perform the following steps:

[0487] Displaying a first interface, including N image materials to be cropped, at least two of the N image materials have different aspect ratios, and N is an integer greater than or equal to 2; in response to a user's operation of cropping the N image materials, processing each image frame of each image material in at least some of the N image materials according to the first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio, wherein when at least some of the image materials include some of the N image materials, the aspect ratios of the image materials other than the some of the N image materials are the first aspect ratio; displaying a second interface, including the N image materials having the first aspect ratio;

[0488] The process of processing the image frame includes:

[0489] When subject detection is performed on an image frame to obtain first position information, second position information for indicating a position of a first cropping frame in the image frame is obtained based on the first position information and the first frame ratio, the first position information being used to indicate a position of a target subject in the image frame, the first cropping frame including the target subject, and the frame ratio of the first cropping frame being the first frame ratio; the image frame is cropped according to the second position information to obtain a first image frame, the first image frame including the target subject, and the frame ratio of the first image frame being the first frame ratio.

[0490] It should be understood that the processor 801 can be used to execute each step performed by the electronic device in the method 600. For specific descriptions, please refer to the relevant descriptions above and will not be repeated here.

[0491] In addition, the processor 801 can also be used to execute each step performed by the electronic device in method 200 or method 300 or method 400 or method 500. For specific descriptions, please refer to the relevant descriptions above and will not be repeated here.

[0492] It should be understood that in the embodiments of the present application, the processor of the above-mentioned device may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0493] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software units in a processor for execution. The software unit can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0494] The embodiment of the present application provides a computer program product, when the computer program product is run on an electronic device, the electronic device executes the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above method-related embodiments, and will not be repeated here.

[0495] The embodiment of the present application provides a readable storage medium, the readable storage medium contains instructions, when the instructions are executed in an electronic device, the electronic device executes the technical solution of the above embodiment. The implementation principle and technical effect are similar, and will not be repeated here.

[0496] The embodiment of the present application provides a chip, the chip is used to execute instructions, when the chip is running, the technical solution in the above embodiment is executed. The implementation principle and technical effect are similar, and will not be repeated here.

[0497] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0498] It should be understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0499] It should also be understood that in the present application, "when", "if" and "if" all mean that the UE or base station will take corresponding actions under certain objective circumstances. It is not a time limit, and it does not require the UE or base station to make judgments when implementing it, nor does it mean that there are other limitations.

[0500] A person skilled in the art may understand that the various numerical numbers such as first and second involved in the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application, and also indicate the order of precedence.

[0501] In this application, elements expressed in the singular are intended to mean "one or more" rather than "one and only one", unless otherwise specified. In this application, "at least one" is intended to mean "one or more", and "more than one" is intended to mean "two or more", unless otherwise specified.

[0502] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A can be singular or plural, and B can be singular or plural.

[0503] In this document, the term "at least one of..." or "at least one of..." means all or any combination of the listed items. For example, "at least one of A, B, and C" may mean: A exists alone, B exists alone, C exists alone, A and B exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time, where A may be singular or plural, B may be singular or plural, and C may be singular or plural.

[0504] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0505] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0506] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0507] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0508] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0509] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0510] The same or similar parts between the various embodiments in this application can refer to each other. In the various embodiments in this application, and the various implementation methods / implementation methods / implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The above-described implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0511] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims. In short, the above is only a preferred embodiment of the technical solution of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Displaying a first interface, including N image materials to be cropped, wherein at least two of the N image materials have different aspect ratios, and N is an integer greater than or equal to 2; In response to a user's operation of cropping the N image materials, processing each image frame of each image material in at least part of the N image materials according to a first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio, wherein when the at least part of the image materials include part of the N image materials, the aspect ratios of the image materials in the N image materials other than the part of the image materials are the first aspect ratio; Displaying a second interface, including the N image materials having the first aspect ratio; The process of processing the image frame includes: In a case where first position information is obtained by subject detection on the image frame, second position information indicating a position of a first cropping frame in the image frame is obtained based on the first position information and the first frame ratio, wherein the first position information is used to indicate a position of a target subject in the image frame, the first cropping frame includes the target subject, and the frame ratio of the first cropping frame is the first frame ratio; The image frame is cropped according to the second position information to obtain a first image frame, where the first image frame includes the target subject, and the aspect ratio of the first image frame is the first aspect ratio.

2. The method according to claim 1, characterized in that The image frame includes a plurality of subjects; and the method further includes: Performing subject detection on the image frame to obtain a plurality of third position information corresponding to the plurality of subjects, each piece of third position information being used to indicate a position of a corresponding subject in the image frame; The first position information is obtained based on the plurality of third position information, and the target subject includes at least part of the plurality of subjects.

3. The method according to claim 2, characterized in that Before obtaining the first location information based on the plurality of third location information, the method further includes: Based on a plurality of subject information corresponding to the plurality of subjects one by one, scoring the plurality of subjects to obtain a plurality of scoring values ​​of the plurality of subjects, one subject corresponding to one scoring value, each subject information including image feature information and image quality information of each subject; Determine the subject corresponding to the score value that meets the preset condition among the multiple score values ​​as the target subject, wherein the preset condition includes: the score value is greater than or equal to the preset score value; and The obtaining the first location information based on the plurality of third location information includes: Based on the multiple pieces of third position information, the third position information corresponding to each subject in the target subject is determined as the first position information.

4. The method according to claim 2, characterized in that: The obtaining the first location information based on the plurality of third location information includes: Any third position information among the plurality of third position information is determined as the first position information.

5. The method according to any one of claims 1 to 4, characterized in that: The first aspect ratio is a preset aspect ratio, and the preset aspect ratio is the aspect ratio of any image material among the N image materials.

6. The method according to any one of claims 1 to 4, characterized in that: In response to a user's operation of cropping the N image materials, processing each image frame of each image material in at least part of the N image materials according to a first aspect ratio so that the aspect ratios of the N image materials are all the first aspect ratio, includes: In response to a user's operation on the smart cropping option of the N image materials, displaying a third interface including a plurality of aspect ratios; In response to a user selecting the first aspect ratio, each image frame of each image material in at least part of the N image materials is processed according to the first aspect ratio, so that the aspect ratios of the N image materials are all the first aspect ratio.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In a case where a first abnormality mark is obtained by performing subject detection on the image frame, fourth position information is obtained based on position information of any position of the image frame and the first frame ratio, the fourth position information is used to indicate a position of a second cropping frame in the image frame, the first abnormality mark is used to indicate that a subject in the image frame is lost or missing, and the frame ratio of the second cropping frame is the first frame ratio; The image frame is cropped according to the fourth position information to obtain a second image frame, wherein the aspect ratio of the second image frame is the first aspect ratio.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Performing subject detection on the image frame to obtain at least one candidate subject; Matching the at least one candidate subject with a designated subject, where the designated subject is a subject selected by a user when performing a cropping operation on the N image materials; When the designated subject successfully matches at least part of the at least one candidate subject, the position of the designated subject in the image frame is determined as the first position information, wherein the designated subject is the target subject.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Performing subject detection on the image frame to obtain at least one candidate subject; Matching the at least one candidate subject with a designated subject, where the designated subject is a subject selected by a user when performing a cropping operation on the N image materials; In the case where the designated subject does not match any of the candidate subjects in the at least one candidate subject, obtaining a second abnormality indicator; In a case where the second abnormality mark is obtained by performing subject detection on the image frame, fifth position information is obtained based on position information of any position of the image frame and the first frame ratio, where the fifth position information is used to indicate a position of a third cropping frame in the image frame, and the frame ratio of the third cropping frame is the first frame ratio; The image frame is cropped according to the fifth position information to obtain a third image frame, and the aspect ratio of the third image frame is the first aspect ratio.

10. The method according to any one of claims 1 to 6, characterized in that: The step of performing cropping processing on the image frame according to the second position information to obtain a first image frame includes: The image frame is cropped according to the second position information and the cropped image frame is scaled to obtain the first image frame.

11. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method as claimed in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 10.

13. A chip, characterized in that: The chip comprises: A memory for storing instructions; A processor, configured to call and execute the instructions from the memory, so that an electronic device equipped with the chip executes the method as claimed in any one of claims 1 to 10.

Images