Image capture method, device and storage medium

Abstract

The present application provides an image capture method, device, and storage medium. This method classifies the scene of a preview video stream obtained based on scene description text input by the user, strategically matches the scene labels obtained from the classification with a preset set of guidance parameters, determines the guidance parameters, matches the guidance parameters with the preview video stream, obtains an image-to-image matching score and an image-to-text matching score for each frame, thereby determining a set of candidate wonderful images, and calculates similarity between target features obtained by feature fusion based on the guidance parameters and each frame in the candidate wonderful image set. The candidate wonderful image with the highest similarity is output as a wonderful moment image. This achieves accurate capture of the wonderful moments of the target scene and improves the user experience.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image capture method, device, and storage medium. Background Art

[0002] When using an electronic device's camera, users often want to capture images of specific scenes (hereinafter referred to as "highlight moments"). Currently, electronic devices offer "continuous shooting" and "auto-capture" functions. However, due to issues such as human reaction delays and electronic device transmission delays, users often have difficulty capturing truly highlight moments, significantly reducing the user experience. Summary of the Invention

[0003] The embodiments of the present application provide an image capture method, device, and storage medium, which are intended to enable users to capture wonderful moments occurring in a target scene during the photo-taking process, thereby improving capture accuracy and enhancing the user experience.

[0004] In a first aspect, an embodiment of the present application provides an image capture method, which is applied to an electronic device, the method comprising: in response to an image capture instruction, obtaining a preview video stream and a scene description text; preprocessing the preview video stream to obtain an image frame set; performing scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set; determining a guide parameter from a preset guide parameter set based on the scene label, the guide parameter including a guide picture and a guide text; matching the guide picture and the guide text with each frame image in the image frame set respectively to obtain a picture-text matching score and an picture-picture matching score for each frame image; determining a candidate wonderful image set from the image frame set based on the picture-text matching score and the picture-picture matching score for each frame image; performing feature fusion on the guide picture and the guide text to obtain a target feature; calculating the similarity between each frame image in the candidate wonderful image set and the target feature; and screening out a wonderful moment image from the candidate wonderful image set based on the similarity, the wonderful moment image being the image frame with the highest similarity in the candidate wonderful image set.

[0005] The preview video stream can be Figure 10 ,After the mobile phone turns on the image capture function in response to the user’s ,image capture instruction, the mobile phone obtains the video stream of the ,target scene through the camera.

[0006] The scene description text can be Figure 7 , which is entered by the user through the "Select Scene" option. The scene description text can be a description of the target scene by the user, as well as a description of the captured image that is expected to be obtained.

[0007] In some possible implementations, the scene description text may be input in advance by the user, such as a guide text; after obtaining the preview video stream, the mobile phone obtains the scene label by matching it with the scene description text.

[0008] Among them, since there may be a large number of repeated video frames in the preview video stream, it is also necessary to perform a pre-processing operation of extracting frames after decoding the preview video stream, so as to improve the efficiency of subsequent data processing.

[0009] The guidance parameters in the preset guidance parameter set may be input by the user in advance, or may be set in when the system module corresponding to the method of the embodiment of the present application is initialized. The preset guidance parameter set includes partial scene images and description texts that the user expects to obtain.

[0010] The guide image and each frame image in the image frame set are input into the image-image matching model in the following embodiment, so as to obtain the image-image matching score of each frame image in the image frame set.

[0011] The guiding text and each frame image in the image frame set are input into the image-text matching model in the following embodiment to obtain the image-text matching score of each frame image in the image frame set.

[0012] The step of fusing features of the guide image and the guide text may include first extracting features of the guide image and then fusing the extracted image features with features of the guide text to obtain target features.

[0013] Therefore, after the user activates the image capture function, the preview video stream of the target scene and the scene description text entered by the user are obtained. The scene classification is then performed on the obtained preview video stream based on the scene description text. The scene labels obtained by classification are strategically matched with a set of preset guidance parameters to determine the guidance parameters. The guidance parameters are then matched with the preview video stream to obtain image-image matching scores and image-text matching scores for each frame of the image. This determines a set of candidate wonderful images, and the similarity is calculated between the target features obtained by feature fusion based on the guidance parameters and each frame of the candidate wonderful image set. The candidate wonderful image with the highest similarity is output as the wonderful moment image. This achieves accurate capture of the wonderful moments of the target scene and improves the user experience.

[0014] According to the first aspect, the screening of wonderful moment images from the candidate wonderful image set based on the similarity further includes: continuously acquiring multiple wonderful moment images, comparing the similarity of each wonderful moment image, and outputting the wonderful moment image with the highest similarity as the final wonderful moment image.

[0015] The process of acquiring a highlight image can be considered a computation cycle. To improve the accuracy of capturing highlight images of a target scene, it is possible to continuously acquire highlight images within multiple computation cycles, compare the similarity of each highlight image, and determine the final highlight image based on the similarity. This allows the image capture method to be optimized along the timeline.

[0016] According to the first aspect, or any implementation of the first aspect above, comparing the similarity of each wonderful moment image and outputting the wonderful moment image with the highest similarity as the final wonderful moment image also includes: scoring the multiple wonderful moment images based on a trained aesthetic evaluation model, and outputting the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

[0017] Among them, the trained aesthetic evaluation model can evaluate the image based on multiple indicators. For example, these indicators include but are not limited to composition, color, light, exposure, conception, etc.

[0018] According to the first aspect, or any implementation of the first aspect above, determining a candidate set of wonderful images from the set of image frames based on the image-text matching score and the image-image matching score of each frame of image includes: determining a plurality of image frames whose image-text matching scores and the image-image matching scores are both greater than a preset threshold as the candidate set of wonderful images.

[0019] The preset threshold value can be set according to actual needs.

[0020] According to the first aspect, or any implementation method of the first aspect above, determining the candidate wonderful image set from the image frame set based on the image-text matching score and the image-image matching score of each frame of image, also includes: constructing a picture-text matching score curve and an image-image matching score curve in the same coordinate system based on the picture-text matching score and the image-image matching score, and the coordinate system is an image frame-similarity coordinate system; obtaining the intersection of the picture-text matching score curve and the image-image matching score curve; intercepting the image-image score curve based on the intersection, and determining multiple image frames corresponding to the intercepted image-image score segments as the candidate wonderful image set.

[0021] According to the first aspect, or any implementation method of the first aspect above, the Tutu score curve is intercepted based on the intersection, and multiple image frames corresponding to the intercepted Tutu score line segment are determined as a set of candidate wonderful images, including: filtering the Tutu score line segment according to a preset peak prominence threshold, and determining multiple image frames corresponding to the filtered Tutu score line segment as a set of candidate wonderful images.

[0022] According to the first aspect, or any implementation of the first aspect above, filtering the graph score line segment according to a preset peak protrusion threshold, further comprising: filtering the graph score line segment according to a preset peak protrusion threshold and a peak width threshold.

[0023] According to the first aspect, or any implementation of the first aspect above, the scene classification of the image frame set based on the scene description text to obtain scene labels corresponding to the image frame set includes: inputting the scene description text and the image frame set into a pre-trained multi-label classification model for label classification to obtain one or more scene labels corresponding to the image frame set.

[0024] According to the first aspect, or any implementation of the first aspect above, before the image capture instruction triggered by the user is used to obtain the preview video stream and scene description text, the method also includes: displaying a first interface, the first interface including a camera application icon; detecting a first click operation on the camera application icon; in response to the first click operation, displaying a second interface, the second interface including a capture start icon; detecting a second click operation on the capture start icon; in response to the second click operation, displaying a scene selection bar in the second interface; detecting the completion of a third click operation on the scene selection bar; in response to the third click operation, sending an image capture instruction and displaying a preview image in the second interface.

[0025] For the specific implementation details of this part, please refer to the embodiment for Figure 6 and Figure 7 The description part will not be repeated here.

[0026] According to the first aspect, or any implementation of the first aspect above, in response to the second click operation, a scene selection bar is displayed in the second interface, and it also includes: in response to the second click operation, finding the option for displaying the scene definition in the second interface; detecting a fourth click operation on the scene definition option; in response to the fourth click operation, displaying a third interface, the third interface including a first input control and a second input control, the first input control is used to add the guide text, and the second input control is used to add the guide picture.

[0027] For the specific implementation details of this part, please refer to the embodiment for Figure 7 and Figure 8 The description part will not be repeated here.

[0028] In a second aspect, embodiments of the present application provide an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory storing program instructions, which, when executed by the processor, cause the electronic device to execute instructions of the method of the first aspect or any possible implementation of the first aspect.

[0029] In a third aspect, an embodiment of the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer program comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, embodiments of the present application provide a chip comprising a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the method of the first aspect or any possible implementation of the first aspect to control the receive pin to receive a signal and to control the transmit pin to send a signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0033] Figure 2 This is a schematic diagram illustrating an exemplary user photo-taking operation;

[0034] Figure 3 is a schematic diagram of an exemplary wonderful moment;

[0035] Figure 4 is a schematic diagram of an exemplary user interface;

[0036] Figure 5 Schematic diagram of an automatic snapshot interface shown as an example;

[0037] Figure 6 This is an exemplary diagram showing the operation of a user entering a snapshot scene;

[0038] Figure 7 The following is an exemplary diagram of a user selecting a scene operation;

[0039] Figure 8 This is a schematic diagram of an exemplary user-defined scenario operation;

[0040] Figure 9 is another exemplary diagram of a user-defined scenario operation;

[0041] Figure 10 Schematic diagram of a scene snapshot shown as an example;

[0042] Figure 11 is a schematic diagram of an exemplary video capture operation;

[0043] Figure 12 Schematic diagram of an exemplary image capture process flow;

[0044] Figure 13 Schematic diagram of the multi-label scene classification process shown as an example;

[0045] Figure 14 is a schematic diagram of a similarity score curve shown as an example;

[0046] Figure 15 is a schematic diagram of image feature processing shown as an example;

[0047] Figure 16 The figure is a schematic diagram showing an exemplary method of screening images of optimal wonderful moments;

[0048] Figure 17 is another schematic diagram of a similarity score curve shown as an example;

[0049] Figure 18 Schematic diagram of an image capture system framework shown as an example;

[0050] Figure 19 A schematic diagram of an illustrative aesthetic model training;

[0051] Figure 20 is a schematic diagram of an illustrative aesthetic evaluation model;

[0052] Figure 21 FIG. 1 is a schematic diagram of the software structure of an electronic device shown as an example. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0055] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0056] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0058] In order to better understand the technical solutions provided by the embodiments of the present application, before describing the technical solutions of the embodiments of the present application, the hardware structures of the electronic devices (such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.) to which the embodiments of the present application are applicable are first described in conjunction with the accompanying drawings. For the convenience of explanation, Figure 1 Let’s take a mobile phone as an example.

[0059] See also Figure 1The mobile phone 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0060] Among them, the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor (Modem), a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc., which are not listed one by one here and this application does not limit this.

[0061] The controller as the processing unit mentioned above can be the nerve center and command center of the mobile phone 100. In actual applications, the controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0062] Regarding the aforementioned neural network processor, specifically in the technical solution provided in the embodiments of the present application, it can be used to train the neural network model for capturing wonderful moments described in the embodiments of the present application. It is understandable that in order to reduce the resource usage of the mobile phone 100, the neural network model can also be trained by a cloud server or other server and sent to the mobile phone 100.

[0063] The ISP mentioned above is used to output digital image signals to the DSP for processing.

[0064] The DSP mentioned above is used to convert digital image signals into image signals in standard RGB, YUV and other formats.

[0065] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0066] In addition, the processor 110 may further include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc., which are not listed here one by one and are not limited in this application.

[0067] In an embodiment of the present application, the processor 110 may be configured to obtain a preview video stream or a video stream being captured by a user from a camera; the processor 110 may also obtain a video selected by the user and guidance parameters. The guidance parameters include, but are not limited to, one or more of guidance text and guidance images. The guidance parameters may be used to obtain an image that is most similar to the guidance parameters from the preview video stream or the video stream.

[0068] Continue to see Figure 1 The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored in the external memory card.

[0069] Continue to see Figure 1 The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area.

[0070] Continue to see Figure 1 The charging management module 140 is used to receive charging input from the charger.

[0071] Continue to see Figure 1The wireless communication function of the mobile phone 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0072] It should be noted that in some implementations, the neural network model used to capture wonderful moments can be trained by a cloud server or other server. For this implementation scenario, the mobile phone 100 can communicate with the cloud server or other server that provides the neural network through the mobile communication module 150 or the wireless communication module 160. For example, the mobile phone 100 can send a request to the cloud server to obtain or update the neural network model through the mobile communication module 150. Accordingly, the cloud server can send the trained neural network model to the mobile phone 100 based on the request of the mobile phone 100.

[0073] In addition, it should be noted that in the scenario where the neural network model is trained by a cloud server (or other server), the cloud server can customize the neural network model suitable for different mobile phones 100 according to the customization requirements corresponding to mobile phones 100 with different configurations, and update the iterative training of the neural network model based on the capture results feedback from different mobile phones 100.

[0074] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0075] Continue to see Figure 1 The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., which are not listed here one by one and are not limited in this application.

[0076] Continue to see Figure 1 The camera 193 is used to capture still images or videos. The mobile phone 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194 and the application processor.

[0077] Continue to see Figure 1 , the display screen 194 is used to display images, videos, etc.

[0078] This concludes the introduction to the hardware structure of the mobile phone 100. It should be understood that Figure 1 The illustrated mobile phone 100 is merely an example. In a specific implementation, the mobile phone 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different component configuration. Figure 1The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0079] A "highlight moment" is a special, important, memorable, or exciting moment. These moments can occur in personal life, such as birthdays, graduations, and weddings, or in public events, such as a spectacular game at a sports meet or the climax of a concert. Highlight moments are typically characterized by uniqueness, importance, memorability, and excitement. They often evoke strong emotions, bringing about intense feelings such as joy, emotion, and awe. Highlight moments manifest in different ways across different fields. In sports, a highlight moment might be a crucial goal, a brilliant save, or a thrilling moment. In a concert, a highlight moment might be a moving melody, a stirring chord, or an enchanting performance. In short, highlight moments are an indispensable part of people's lives. They allow us to experience the beauty and richness of life and become important elements for people to recall and share.

[0080] For example, when photographing scenic spots and historical sites, a highlight moment is when the camera captures a complete, clear, and noise-free image of the building; when photographing a hurdler, a highlight moment is when the athlete jumps and crosses the hurdle, and the image is clear, without jitter, and noise; when photographing a close-up portrait, a highlight moment is when the person opens their eyes and smiles, and the image is clear, without jitter, and noise. The processed images of these highlight moments are then output as photos.

[0081] However, during the shooting process, due to problems such as human reaction delay and transmission delay of electronic devices, users often find it difficult to take wonderful photos of truly wonderful moments.

[0082] In order to better understand the existing process of capturing wonderful moments, the following Figure 2 To the attached Figure 5 Provide explanation.

[0083] See also Figure 2 In (1), a mobile phone interface 10a is shown as an example. For example, the interface 10a displays multiple application icons, such as the icon 10a-1 of the camera application, and icons of applications such as the address book, phone, message, clock, calendar, gallery, memo, file management, email, music, calculator, video, recorder, weather, browser, and settings.

[0084] It should be noted that in some possible implementations, Figure 2The interface 10a shown in (1) can be called the main interface. When the user clicks the icon 10a-1 in the interface 10a, the camera application can be used to implement shooting functions, such as taking photos, shooting (recording) videos, etc.

[0085] Continue to refer to Figure 2 In (1), for example, when the user clicks the icon 10a-1 of the camera application, the mobile phone responds to the user operation, recognizes that the icon corresponding to the user click operation is the icon of the camera application, and then calls the corresponding interface in the application framework layer to start the camera application, and starts the camera driver by calling the kernel layer, and collects the image stream (preview stream in this case) through the camera. At this time, the mobile phone displays the interface of the camera application, for example Figure 2 The interface 10b shown in (2) will display the preview stream corresponding screen, such as Figure 2 (2) shows a screen of a user playing basketball.

[0086] As the camera function of mobile phones improves, the camera application supports more and more shooting modes. For example, the shooting mode may include aperture mode, night scene mode, portrait mode, photo mode, video mode, smile mode, professional mode, etc. Figure 2 Mode options are displayed in the shooting mode list 10b-2 in (2).

[0087] For example, when the user clicks on the icon option corresponding to a certain shooting mode, the phone displays the camera application interface in the corresponding shooting mode. For example, if the user clicks on the icon option of the photo mode, the phone displays the interface of the camera application in the photo mode. Figure 2 Interface 10b shown in (2).

[0088] It should be noted that in some possible implementations, after the mobile phone starts the camera application, the camera application selects the photo mode by default, that is, the photo mode is the default shooting mode of the camera application. Among them, in the photo mode, the photo shutter control 10b-1 will also be displayed in the interface 10b. Exemplarily, when the user clicks the photo shutter control 10b-1, for example, clicking the photo shutter control 10b-1 N times in succession, the mobile phone will detect the user operation acting on the photo shutter control 10b-1, and then take a photo in response to the operation (take N times, and get picture P1, picture P2... picture Pn respectively). At this time, the thumbnail corresponding to the control 10b-3 displayed in the interface 10b for opening the system's built-in gallery will be updated to the thumbnail of the user playing basketball that has just been taken (the thumbnail of the picture taken by clicking the photo shutter control 10b-1 for the last time), as shown in FIG. Figure 2 The control 10b-3' in the interface 10b' is shown in (3).

[0089] Among them, in the above shooting scene, picture P5 corresponds to the wonderful moment in the basketball scene, that is, picture P5 is the wonderful photo that the user expects to get. Picture P1 and picture Pn are photos of the user dribbling the ball, that is, photos that the user does not expect to get. The above photos, time and user expectation values ​​are used to establish a coordinate system (see Figure 3 ), the user needs to press the shutter button at time t2 to capture the wonderful moment. However, due to human reaction delay (including the time required to complete the shooting operation), the time when the mobile phone receives the user's shooting operation will lag, resulting in the failure to capture the wonderful moment image P5.

[0090] To this end, the manufacturer also provides a "smart photo" function to help users capture wonderful moments.

[0091] See also Figure 4 In (1), a mobile phone camera function setting interface 20a is exemplarily shown. For example, the interface 20a includes multiple function icons, such as smart code scanning, AI photography, flash, filter and setting 20a-1 icons.

[0092] Continue to see Figure 4 In (1), when the user clicks the icon 20a-1 in the interface 20a, the setting interface of the camera application can be entered.

[0093] Continue to see Figure 4 In (1), for example, when the user clicks icon 20a-1, the mobile phone responds to the user's operation and enters the camera application setting interface. At this time, the mobile phone displays the setting interface, for example Figure 4 Interface 20b shown in (2).

[0094] See also Figure 4 In (2), the camera setting interface 20b is exemplarily shown. Exemplarily, the interface 20b includes setting parameters for taking photos and videos. The photo parameters include photo ratio, smart photo 20b-1, and filter recommendation; the video parameters include video resolution, video frame rate, high-efficiency video format, high dynamic range rendering (HDR10+), AI movie tone, and multiple recordings.

[0095] Continue to see Figure 4 In (2), when the user clicks on the smart photo taking 20b-1 in the interface 20b, the smart photo taking setting interface is entered.

[0096] Continue to see Figure 4 In (2), for example, when the user clicks on the smart photo 20b-1, the mobile phone responds to the user operation and enters the smart photo setting interface. At this time, the mobile phone displays the smart photo setting interface, for example Figure 4 Interface 20c shown in (3).

[0097] See also Figure 4 In (3), an intelligent photo setting interface 20c is exemplarily shown. For example, the interface 20c includes multiple intelligent photo functions and their corresponding setting buttons, such as voice-activated photo, gesture photo, smile capture, and automatic capture 20c-1.

[0098] It should be noted that the automatic capture in the above interface 20c is performed by the mobile phone when it recognizes a preset specific scene. The above specific scenes include people's smiling faces, jumping, running, and automatic photos of cats, dogs and other wonderful moments.

[0099] Continue to see Figure 4 In (3), for example, when the user clicks on the automatic snapshot 20c-1 to start the automatic snapshot, the camera application starts the automatic snapshot function and returns to the user photo taking interface, for example Figure 5 Interface 30a shown in (1).

[0100] See also Figure 5 In (1), an exemplary photo taking interface 30a with the automatic snapshot function turned on is shown. After the mobile phone enters the interface 30a, the user points the mobile phone camera at the target scene, and the camera application on the mobile phone automatically recognizes the wonderful moment and takes a snapshot.

[0101] Continue to see Figure 5 In (1), for example, when the camera application recognizes a "jumping" scene, it automatically takes a snapshot. In this way, the mobile phone can trigger a snapshot through a specific scene without the user having to operate during the photo-taking process, thus capturing the wonderful moment.

[0102] However, this method can only capture images based on limited scenes and preset actions. The resulting image scenes and types are very limited, lack versatility, and cannot meet the user's personalized and diverse needs.

[0103] In view of this, an embodiment of the present application provides an image capture method, which captures the target scene according to the user's personalized and diverse needs, thereby obtaining wonderful moments and improving the user experience.

[0104] In a possible implementation, the image taken with the camera application is still used as the Figure 2 Taking the scene of the user jumping and dunking as shown in (2) as an example, based on the image capture method provided in the embodiment of the present application, after determining the requirements input by the user, the mobile phone converts the above requirements into guidance parameters, and based on the guidance parameters, determines the wonderful moments from the preview video stream obtained from the camera application.

[0105] In order to explain this beneficial effect in detail, Figures 6 to 10 Provide explanation.

[0106] See also Figure 6 In (1), the main interface 40a of the mobile phone is shown as an example. For example, the main interface 40a includes a camera application 40a-1. When the user clicks the camera application 40a-1, the mobile phone responds to the user operation and enters the camera's photo interface. At this time, the mobile phone displays the camera's photo interface, for example Figure 6 Interface 40b shown in (2).

[0107] It should be noted that the image capture method provided in the embodiment of the present application can be combined with a camera application to provide services to users, or it can be combined with other applications to provide services to users. When combined with other applications, it is necessary to call the image capture method provided by the image capture method in other applications.

[0108] For example, the above-mentioned other applications may include image photography applications or image editing applications, and the embodiments of the present application do not limit this.

[0109] See also Figure 6 In (2), an example of an image capture start interface 40b is shown. For example, the start interface 40b includes a capture start control 40b-1. The user can use the image capture function by clicking the control 40b-1 to set the scene.

[0110] It should be noted that, compared with the above interface 40b Figure 5 As can be seen from the interface 30a provided in (1), a snapshot start control 40b-1 is added to the lower left corner of the interface 40b. The position of the control 40b-1 can be set according to the user's operating habits. It can also be set in the function setting bar at the top of the camera application, or in other locations that are more convenient for user operation. This embodiment of the application does not limit this.

[0111] Continue to see Figure 6 In (2), for example, when the user clicks on the control 40b-1, the mobile phone responds to the user's operation and enters the scene setting interface of the image capture interface. At this time, the mobile phone displays the scene setting interface, for example Figure 7 Interface 50a shown in (1).

[0112] See also Figure 7 In (1), a scene setting interface 50a is shown as an example. The interface 50a includes a scene selection control 50a-1.

[0113] For example, after the user clicks on the control 40b-1, the mobile phone responds to the operation and displays the control 50a-1. The control 50a-1 displays the "Select Scenario" option and the "Define Scenario" option.

[0114] It should be noted that the above-mentioned "Select Scene" option may correspond to a preset scene, which includes but is not limited to half-length portrait, landscape, dynamic moment, sports, smiling face, and street scene.

[0115] In some possible implementations, the above-mentioned preset scenes may be pre-installed in the mobile phone, or may be obtained in real time by the mobile phone accessing the cloud server through the network, and this embodiment does not impose any restrictions on this.

[0116] In some other possible implementations, the above-mentioned preset scenes may also be generated by collecting historical user-defined scenes.

[0117] Continue to see Figure 7 In (1), for example, after the user clicks the "Select Scene" option, the mobile phone responds to the operation and displays a preset scene selection interface, such as Figure 7 Interface 50b shown in (2).

[0118] See also Figure 7 In (2), a scene selection interface 50b is shown as an example. For example, the interface 50b includes a scene selection control 50b-1, which is a sliding selection control. The control lists all preset scenes, and the user can select the desired preset scene by sliding left or right in the control 50b-1.

[0119] Exemplarily, the information included in the above-mentioned preset scene includes a scene picture and a scene text description.

[0120] It should be noted that after the user selects a preset scene, the mobile phone can obtain the corresponding scene picture and scene text description, and convert them into guidance parameters, and then use the image capture method provided in the embodiment of the present application to capture the target scene based on the guidance parameters.

[0121] Continue to see Figure 7 In (2), for example, after the user selects the "sports" scene in the control 50b-1, the image capture interface can be entered.

[0122] In other possible implementations, in order to further improve the accuracy of the snapshot, the scene in the control 50b-1 can also be refined. That is, after the user clicks the "sports" scene in the control 50b-1, it enters the subdivided scene selection interface, for example Figure 7 Interface 50c shown in (3).

[0123] See also Figure 7In (3), a subdivision scene selection interface 50c is shown as an example. The interface 50c includes a subdivision scene selection control 50c-1. The control 50c-1 is a sliding selection control that lists all subdivision scenes corresponding to the preset scene. The user can slide left or right in the control 50c-1 to select the required subdivision scene.

[0124] Continue to see Figure 7 In (3), for example, after the user clicks on the "Sports" scene in the control 50b-1 of the interface 50b, the mobile phone displays the interface 50c, and the sub-scenes of the "Sports" scene, such as jump slam, flying slam, vertical jump, and fancy jump, are listed in the control 50c-1 of the interface 50c. After the user completes the selection of the preset scene and the sub-scene, the mobile phone displays the image capture interface and obtains the video stream of the current scene through the camera, and captures the wonderful moments from the video stream, thereby realizing the image capture function provided by the embodiment of the application.

[0125] In some other possible implementations, the above preset scenarios cannot meet the personalized needs of users. Therefore, the embodiment of the present application also provides the option of customizing scenarios.

[0126] Continue to see Figure 7 In (1), for example, after the user clicks the "Define Scene" option, the mobile phone responds to the operation and displays the custom scene interface, for example Figure 8 Interface 60a shown in (1).

[0127] See also Figure 8 In (1), a custom scene interface 60a is shown as an example. The interface 60a includes a text prompt bar, an operation bar 60a-1 for guiding text, and an operation bar 60b-2 for guiding images. The operation bar 60a-1 includes title information, an add button, and a modify button; the operation bar 60b-2 includes title information, an add button, and a modify button.

[0128] For example, the content in the above text prompt bar is "Please select the guidance parameters, and the mobile phone will capture the wonderful moment image according to the selected guidance parameters."

[0129] For example, the title information of the operation bar 60a-1 is "Guidance Text", and the title information of the operation bar 60b-2 is "Guidance Image".

[0130] It is understandable that the content in the above-mentioned text prompt bar, the title information of the operation bar 60a-1 and the title information of the operation bar 60b-2 can also be other text content and / or image content that is easier for users to understand, and the embodiment of the present application does not limit this.

[0131] Continue to see Figure 8In (1), for example, after the user clicks the add button in the operation bar 60a-1, the mobile phone responds to the operation and displays the guide text adding interface, for example Figure 8 Interface 60b of (2).

[0132] See also Figure 8 In (2), a guide text adding interface 60b is exemplarily shown. The interface 60b includes one or more options. The one or more options can be used to trigger the mobile phone to display guide texts from different sources. The one or more options include, but are not limited to, a "select template text" option, a "custom input text" option, a "local input text" option, and a "network search text" option.

[0133] Among them, the "Select Template Text" option is used to trigger the mobile phone to display an interface containing template text. The interface containing template text may include one or more sentences of template text. The one or more sentences of template text can serve as guiding text. The mobile phone can receive input for the template text and use the template text as guiding text. The "Custom Input Text" option is used to trigger the mobile phone to display a text box. The mobile phone can receive text entered into the text box and use the text as guiding text. The "Local Input Text" option is used to trigger the mobile phone to display the interface of an application that stores local text, such as a notepad application interface. The interface of the notepad application may include one or more sentences of text. The one or more sentences of text can serve as guiding text. The mobile phone can receive input of user-selected text and use the selected text as guiding text. The "Web Search Text" option is used to trigger the mobile phone to display a web search interface. The web search interface may include a search bar. The mobile phone can receive keywords entered into the search bar, search for text corresponding to the keywords based on the keywords, receive input of user-selected text, and use the selected text as guiding text.

[0134] Based on the guide text added by the user, the mobile phone can obtain the video stream through the camera and capture the wonderful moments from the video stream, thereby realizing the image capture function provided in the embodiment of the present application.

[0135] It should be noted that Figure 7 The "Define Scene" option in (1) can also be independently set as a user-defined scene entry, so that users can enter customized scenes in advance. Its entry can be set in the system setting interface or as an independent system application, which is not limited in this embodiment of the application.

[0136] In addition, it should be noted that in some possible implementations, Figure 7 If the "Define Scene" option in (1) is set to a separate entry, Figure 7 The "Define Scene" option in (1) can also be cancelled, leaving only the "Select Scene" option.

[0137] In some other possible implementations, the user can also Figure 8 In the interface 60a of (1), click the add button in the operation bar 60a-2, and the mobile phone responds to the operation by displaying the guide image adding interface, for example Figure 9 Interface 70a of (1).

[0138] See also Figure 9 In (1), a guide image adding interface 70a is exemplarily shown. The interface 70a includes one or more options. The one or more options can be used to trigger the mobile phone to display guide images from different sources. The one or more options include but are not limited to the "Select Template Image" option, the "Custom Input Image" option, the "Local Input Image" option 70a-1, and the "Network Search Image" option.

[0139] Among them, the "Select Template Image" option is used to trigger the mobile phone to display an interface containing a template image. The interface containing a template image can contain one or more template images. The one or more template images can serve as a guide image. The mobile phone can receive input for the template image and use the template image as the guide image. The "Custom Input Image" option is used to receive an image input by the user and use the input image as the guide image. For example, the "Custom Input Image" option can receive a user-drawn image input by the user through a touchpad or input device (for example, a stylus, a digital tablet, etc.) and use the input image as the guide image. The "Local Input Image" option 70a-1 is used to trigger the mobile phone to display the interface of an application that stores local images, such as a gallery application interface or a file management application interface. The "Web Search Image" option is used to trigger the mobile phone to display a web search interface. The web search interface can include a search bar. The mobile phone can receive keywords input into the search bar, search for images corresponding to the keywords based on the keywords, receive input of an image selected by the user, and use the selected image as the guide image.

[0140] Continue to see Figure 9 In (1), for example, after the user clicks the "local input picture" option 70a-1, the mobile phone responds to the operation and displays the gallery interface. After the user selects the guide picture through the gallery interface, the mobile phone displays the guide picture confirmation interface, for example Figure 9 Interface 70b shown in (2).

[0141] See also Figure 9 In (2), the guide image confirmation interface 70b is shown as an example. For example, after the user completes uploading the image, an information confirmation bar 70b-1 pops up in the interface 70b. The information confirmation bar 70b-1 includes a prompt message, an OK button, and a Cancel button.

[0142] Exemplarily, after the user clicks the OK button, the mobile phone uses the currently displayed image as the guide image; after the user clicks the Cancel button, the mobile phone returns to interface 70a to allow the user to re-upload the guide image.

[0143] After the guide image is added, the mobile phone can obtain the video stream through the camera based on the guide image added by the user, and capture the wonderful moments from the video stream, thereby realizing the image capture function provided in the embodiment of the present application.

[0144] In some other possible implementations, users can also upload guide images and guide texts at the same time, so that the mobile phone can capture the wonderful moments in the video stream based on the guide images and guide texts, thereby improving the accuracy of the capture. Figure 10 , Figure 10 A set of schematic diagrams for capturing wonderful moments based on guide pictures and guide texts are shown.

[0145] See also Figure 10 In (1), an example is shown of the interface 80a after the user turns on the image capture function. The interface 80a includes a capture state control 80a-1. When the user turns on the capture, Figure 6 The snapshot start control 40b-1 in (2) will be selected and a background color will be added, becoming the status control 80a-1 in the interface 80a.

[0146] Exemplarily, after the user points the mobile phone camera at the target scene, the real-time image of the scene is displayed in the image preview frame of the camera, and the preview video stream is sent to the mobile phone.

[0147] See also Figure 10 In (2), a preview video stream acquisition interface 80b is exemplarily shown. The interface includes an image preview area and a thumbnail control 80b-1.

[0148] Exemplarily, after the user points the camera of the mobile phone at the target scene, the mobile phone continuously obtains the preview video stream of the target scene through the camera, wherein the preview video stream includes the P1 image frame, the P5 image frame and the Pn image frame.

[0149] Exemplarily, after the mobile phone obtains the preview video stream, it captures the wonderful moments in the scene through the above-mentioned guide picture, guide text and the picture capture method provided in the embodiment of the present application.

[0150] Exemplarily, the mobile phone stores the captured wonderful moment in the gallery and updates the thumbnail on the thumbnail control 80b-1 in the interface 80b.

[0151] See also Figure 10In (3), an example of the interface 80c after the thumbnail is updated is shown. For example, in the interface 80c, the mobile phone uses the picture capture method provided by the embodiment of the present application to determine that the picture frame P5 is a wonderful moment based on the guide picture and the guide text, stores the picture frame P5 in the gallery, and updates the thumbnail of the thumbnail control 80b-1 in the interface 80b, displaying it as the thumbnail control 80b-1'.

[0152] At this point, the embodiment of the present application has completed the automatic capture of wonderful moments of the target scene based on the guide picture, guide text and the image capture method provided by the embodiment of the present application, greatly improving the accuracy and versatility of image capture.

[0153] In another possible implementation, the user can also open the video recording function of the camera application on the mobile phone. Figure 2 The scene of the user jumping and dunking shown in (2) is recorded, and the recorded video is used to capture the wonderful moments from the video using the image capture method provided in the embodiment of the present application.

[0154] In order to explain this beneficial effect in detail, Figure 11 Provide explanation.

[0155] See also Figure 11 , exemplarily showing a set of scenes for capturing images of videos in a gallery.

[0156] See also Figure 11 In (1), a mobile phone main interface 90a is shown as an example. The main interface 90a includes multiple application icons. Among them, the multiple application icons include a clock, a calendar, a gallery 90a-1, a notepad, a file manager, etc.

[0157] For example, the user clicks on the gallery 90a-1, and the mobile phone responds to the operation and enters the gallery interface, for example Figure 11 Interface 90b shown in (2).

[0158] See also Figure 11 In (2), a gallery interface 90b is shown as an example. In the interface 90b, pictures and videos taken / downloaded by the user are displayed in columns according to the shooting time / download time.

[0159] For example, the interface 90b includes a video 90b-1 shot by the user yesterday and a downloaded guide picture 90b-2.

[0160] In some possible implementations, the guide image 90b-2 is used to guide the image capture method provided in the embodiment of the present application to capture the wonderful moments in the video 90b-1.

[0161] It should be noted that the gallery interface 90b displays a large number of videos and pictures that the user has already taken. These videos and pictures may be taken by the user himself or downloaded from the Internet. The user hopes to capture the wonderful moments of the video 90b-1 using the downloaded picture 90b-2.

[0162] Continue to see Figure 11 In (2), for example, after the user clicks on video 90b-1, the mobile phone responds to the operation and enters the video operation interface, for example Figure 11 Interface 90c shown in (3).

[0163] See also Figure 11 In (3), a video operation interface 90c is shown as an example. The interface 90c includes multiple video operation controls, such as a video play control, a share control, a delete control, a filter control 90c-1 and more function controls.

[0164] Exemplarily, after the user clicks the filter control 90c-1, the mobile phone captures the wonderful moments of the video 90b-1 by executing the image capture method provided in the embodiment of the present application, and updates the captured picture as the video cover to the video 90b-1.

[0165] At this point, the embodiment of the present application has completed the automatic capture of wonderful moments of videos in the gallery based on the guide pictures, guide texts and the image capture method provided by the embodiment of the present application, greatly improving the accuracy and versatility of image capture.

[0166] It should be noted that after the user clicks the filter control 90c-1, the gallery application can also display Figure 8 The guide parameter setting interface in (1) can be used by the user to add guide text and guide image through the guide parameter setting interface, thereby capturing the wonderful moments of the video 90b-1.

[0167] Furthermore, it should be noted that in some possible implementations, the gallery application is not associated with the navigation parameter setting interface, and the user does not enter the navigation parameter setting interface after clicking filter control 90c-1. After the user clicks filter control 90c-1, the image capture method provided in the embodiment of the present application captures the highlights of the video according to the preset scene, thereby obtaining an image of the highlights.

[0168] In order to better understand the process of capturing wonderful moments by the image capturing method provided by this application, the following embodiments still take the mobile phone 100 as an example, and combine Figures 12 to 20 The image processing flow during image capture is explained.

[0169] For example, Figure 12 As shown, the image capture method includes the following steps:

[0170] S101. Based on the image capture instruction triggered by the user, obtain a preview video stream and scene description text.

[0171] refer to Figures 6 to 10 In the user interface shown, the mobile phone detects that the user has entered the camera photo page and has selected a scene by clicking on either control 40b-1 or control 50a-1. The mobile phone then determines that the user has triggered an image capture command. At this point, the mobile phone obtains the scene description text corresponding to the scene selected by the user and obtains a preview video stream through the camera.

[0172] The scene description text can be the user's Figure 7 The scene description corresponding to the subdivision scene selected by the control 50c-1 in (3); or the scene description corresponding to the subdivision scene selected by the user by clicking Figure 8 The guiding text input after the control 60a-1 in (1) can also be the image scene description text preset in the mobile phone, which is not limited in the embodiment of the present application.

[0173] For example, the scene description text may be “surfing scene, stretching posture, the moment of rolling up waves”, “jumping / body tilting to wipe off snowflakes”, “shooting scene” and so on.

[0174] S102. Preprocess the preview video stream to obtain an image frame set.

[0175] The phone's camera generates a RAW image in real time based on the light within the viewing area and sends it to the upper-layer application that calls the camera, such as the camera app. For example, at a frame rate of 30fps, the camera reports a RAW image frame to the upper-layer application every 33.3 milliseconds (ms), meaning the upper-layer application receives a RAW image frame every 33.3ms.

[0176] Therefore, in order to reduce computing costs and improve processing efficiency, the preview video stream can also be pre-processed to reduce the number of image frames involved in subsequent calculations. Pre-processing methods include decoding, frame extraction, and other operations.

[0177] S103. Perform scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set.

[0178] Among them, the mobile phone can use a multi-label classification model to determine the scene labels corresponding to the images in the image frame set.

[0179] For example, the scene tag may include but is not limited to a combination of one or more parameters such as the subject in the image, the subject's action, the subject's emotion, the subject's expression, and the shooting scene.

[0180] For example, scene tags include but are not limited to looking back, fireworks, lightning, parent-child, running, jumping, skateboarding, skiing, badminton, basketball, splashing water, flying hair, flapping wings, graduation celebration, romantic warmth, affectionate gaze, passion on the field, cat jumping, dog jumping, etc.

[0181] In some possible implementations, see Figure 13 The scene description text and image frame set are input into the text encoder (text_encoder) and image encoder (image_encoder) respectively, obtaining aligned text vectors (text_embedding) and frame vectors (frame_embedding). The similarity scores (scene_similartity_scores) between each image frame and the text in the text vectors and frame vectors are then calculated to determine the scene label corresponding to the image frame set.

[0182] The multi-label classification model can be a pre-trained model, and its training process can be found in Figure 14 . Its training data can be image-text data pairs captured from the Internet, manually annotated, and image-text data pairs generated by large language models. Exemplarily, the image-text data pairs captured from the Internet can be obtained by searching various image and film sharing websites on the Internet, selecting suitable image-text data sources for capture, and capturing the images and paired texts, and then performing data preprocessing such as data cleaning, deduplication, and filtering. For single-modal data such as collected images, manual annotation can be performed, such as manually writing descriptive text for the images or using other text generation tools to assist in generating descriptive text. For existing text data, matching can be performed through keyword search, and then manually selecting images for claims. The collected data is input into the model for training to obtain a multi-label classification model.

[0183] Specifically, the training process of the multi-label classification model can be trained in the cloud and then sent to the mobile phone, or it can be trained on the mobile phone. The embodiments of the present application do not limit this.

[0184] For example, the cosine similarity algorithm is used to calculate the similarity between each frame image and the scene description text in the text vector and frame vector. The similarities are then sorted by the top selector (top_K_selector) to obtain the top-ranked scene labels. The closer the cosine similarity value is to 1, the higher the similarity between the image and the scene description text; the closer the cosine similarity value is to 0, the lower the similarity between the image and the scene description text.

[0185] In some possible implementations, since the confidence of the top-ranked scene labels cannot be determined at this time, it is necessary to input the obtained scene labels and image frames into the image-text matching model (ITModel validation) for binary classification to further determine the matching degree between the scene labels and the image frames, thereby obtaining the final scene labels of the image frame set.

[0186] For example, when determining the degree of match between a scene label and an image frame, it can be determined based on the maximum confidence value, thereby outputting a unique label; it can also be determined based on a confidence range according to business needs, thereby outputting multiple labels. This embodiment of the present application does not limit this.

[0187] In other possible implementations, the electronic device uses a multi-label classification model to average the features of all images in the image frame set to obtain average feature information for the image frame set. The average feature information includes the features of all images in the image frame set. The mobile phone can use an image scene classification algorithm to determine the scene label corresponding to the average feature information and use this scene label as the scene label for the image frame set.

[0188] Furthermore, it is not limited to identifying the scene label of the image frame set based on the average feature information. For example, the mobile phone can also identify the scene label of the image frame set based on the first image in the image frame set; or, the mobile phone can identify and determine the scene label of an image randomly selected from the image frame set, and use the scene label of the random image as the scene label of the image frame set. The embodiments of the present application do not impose any restrictions on this.

[0189] S104. Determine guidance parameters based on the scene label, where the guidance parameters include a guidance picture and a guidance text.

[0190] The guidance parameters include but are not limited to one or more of the guidance pictures and guidance texts. The guidance parameters can be used to obtain the image most similar to the guidance parameters from the image frame set, that is, to capture wonderful moments from the image frame set.

[0191] Furthermore, the guidance parameters may be preset in the storage or may be input by the user, and this embodiment of the present application does not impose any limitation on this.

[0192] For example, see Figure 8In the user interface of the present invention, the guiding text may be one or more of template text, text input by the user, text input from the local mobile phone, and text obtained by network search. The template text may include one or more text examples provided by the mobile phone. The text input by the user may be text typed or voice input by the user. The text input from the local mobile phone may be text stored in the mobile phone (for example, text in the memo application, text in the file application, etc.). The text obtained by network search may be text retrieved by the mobile phone based on the search term provided by the user, and so on.

[0193] For example, the guidance text can be descriptive sentences in the style of Tang poetry, Song lyrics, Yuan opera, modern poetry, and prose. Based on the guidance text, the mobile phone can filter images from the image collection that match the description of the guidance text. For example, if the guidance text includes "turn head and smile", the mobile phone can filter images from the image collection that include the subject turning their head and smiling.

[0194] For example, see Figure 9 The user interface of the guide image may be one or more of a template image, an image inputted by a user, an image inputted locally from the mobile phone, and an image obtained by a network search. The template image may include one or more image examples provided by the mobile phone. The image inputted locally from the mobile phone may be an image stored by the mobile phone (for example, an image from a gallery application, a video from a video application, etc.). The image inputted by the user may be an image drawn by the user and received by the mobile phone through a touchpad or input device (for example, a stylus, a digital tablet, etc.). The image obtained by the network search may include an image retrieved by the mobile phone based on a search term provided by the user. Based on the guide image, the mobile phone may filter out images similar to the guide image from the image collection. For example, when the guide image includes "the subject who turns his head and smiles", the mobile phone may filter out images including "the subject who turns his head and smiles" from the image collection.

[0195] Specifically, the most similar guidance pictures and guidance texts are screened out from the guidance parameters according to the above-mentioned scene labels. The screening method can be found in step S103.

[0196] In some possible implementations, scene tags may be set for the guidance text and guidance image in the guidance parameters first, so as to match the scene tags determined by the image frame with the scene tags in the guidance parameters to determine the guidance parameters.

[0197] In other possible implementations, multiple guiding parameters may be selected. After the guiding parameters are passed through the encoder to obtain a guiding vector, the mobile phone can also perform pooling operations on the guiding vector, such as max pooling, average pooling, k-max pooling, chunk-max pooling, etc., to sample multiple guiding vectors and obtain a reduced-dimensionality guiding vector. The mobile phone can then determine a set of candidate images based on the reduced-dimensionality guiding vector. In this way, the mobile phone can reduce the number of calculation parameters and the amount of computation while retaining the main features of the image, thus preventing overfitting.

[0198] S105. Match the guide image and the guide text with each image frame in the image frame set respectively to obtain an image-text matching score and an image-image matching score for each image frame.

[0199] Specifically, the guide picture and the image frame set are input into the pre-trained image-image matching model (IIMoel) to calculate the image-image matching score of each frame image; the guide text and the image frame set are input into the pre-trained image-text matching model (ITMoel) to calculate the image-text matching score of each frame image.

[0200] The image-image matching score represents the similarity between the guide image and the images in the image frame set. The higher the image-image matching score, the higher the similarity between the guide image and the images in the image frame set. The image-text matching score represents the similarity between the guide text and the images in the image frame set. The higher the image-text matching score, the higher the similarity between the guide text and the images in the image frame set.

[0201] S106 . Determine a candidate set of wonderful images from the image frame set based on the image-text matching score and the image-image matching score of each frame of image.

[0202] In some possible implementations, the image-text matching score and the image-image matching score can be screened using a threshold selection method. When both the image-text matching score and the image-image matching score are greater than a set threshold, the corresponding image is determined to be a wonderful image and added to the wonderful image collection. The above-mentioned threshold can be set according to actual needs.

[0203] In other possible implementations, see Figure 14Based on the image-text matching score and image-image matching score for each frame in the image frame set, we construct an image-text matching score curve and an image-image matching score curve in the image frame-similarity coordinate system, and obtain the intersection of the matching score curve and the image-image matching score curve. Compared to the guide text, the guide image better reflects the user's desired capture result. Therefore, we use the above intersection to extract the image-image matching score, and use the multiple image frames corresponding to the intercepted interval as the candidate wonderful image set.

[0204] S107. Perform feature fusion on the guide image and the guide text to obtain target features.

[0205] For example, see Figure 15 , the guide image is subjected to feature extraction, and then the extracted features are fused with the text features of the guide text to finally obtain the target features.

[0206] S108. Calculate the similarity between each frame of image in the candidate wonderful image set and the target feature.

[0207] Among them, the target features and each frame image in the candidate wonderful image set are input into IIMoel, and the similarity of each frame image is calculated.

[0208] S109. Screening out a wonderful moment image from the candidate wonderful image set according to the similarity, wherein the wonderful moment image is the image frame with the highest similarity in the candidate wonderful image set.

[0209] Specifically, the similarity of each frame of image is sorted, and the image with the highest similarity is output as the wonderful moment image.

[0210] In this way, the mobile phone can process the scene description text entered by the user and the preview video stream of the target scene according to the preset or input guidance parameters, and capture the wonderful moment images from the preview video stream, thereby obtaining the user-customized scene and the required wonderful images, thereby improving the user experience.

[0211] In other possible implementations, since the wonderful moment image in the current preview video stream may not be the best wonderful moment image, multiple wonderful moment image calculations can be performed in the embodiment of the present application, and the wonderful moment image in the current calculation cycle is compared with the wonderful moment image obtained by subsequent calculations to screen out the better quality wonderful moment image as the final wonderful moment.

[0212] The image capture method provided in the embodiment of the present application also includes:

[0213] Step S110 : Continuously acquire multiple wonderful moment images, compare the similarity of each wonderful moment image, and output the wonderful moment image with the highest similarity as the final wonderful moment image.

[0214] For example, see Figure 16 The mobile phone continuously captures a video stream of the target scene through its camera and performs three consecutive calculations on the video frames in the video stream, with the calculation cycles being denoted as calculation cycle t1, calculation cycle t2, and calculation cycle t3. The number of image frames involved in each calculation cycle is preset; the image frames in each calculation cycle may or may not overlap with those in adjacent calculation cycles.

[0215] The highlight moment image obtained during calculation period t1 is ks1, the highlight moment image obtained during calculation period t2 is ks2, and the highlight moment image obtained during calculation period t3 is ks3. The similarities among ks1, ks2, and ks3 are compared in step S108, and the highlight moment image ks1 with the highest similarity is output as the final highlight moment.

[0216] It should be noted that when the current calculation cycle is t1, the search space for optimization can be expanded along the timeline by comparing the similarity with the highlight images obtained at t2 and t3. To further improve the accuracy of the capture, if the similarity of ks1 is lower than ks2, t2 can be set as the current calculation cycle and the optimization search can be continued along the timeline. This will add the highlight images within all calculation cycles to the candidate results and select the highlight image with the highest similarity score. The number of subsequent calculation cycles can be set based on the performance of the mobile phone.

[0217] In some other possible implementations, aesthetic evaluation or fine-grained image evaluation may be performed on the wonderful moment images in the candidate results, thereby obtaining aesthetic scores or attribute scores of various granularities for these wonderful moment images.

[0218] The image capture method provided in the embodiment of the present application also includes:

[0219] Step S111 . Score the multiple wonderful moment images based on the trained aesthetic evaluation model, and output the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

[0220] After obtaining the aesthetic score of each wonderful moment image, the aesthetic evaluation threshold (aesthetic evaluation threshold T A =0.5), directly filter out candidate results with values ​​less than T A The remaining wonderful moment images in the candidate results are sorted by aesthetic scores, and the one with the highest score is set as the final wonderful moment image.

[0221] In some other possible implementations, in order to reduce the result output delay, the candidate highlight set in step S106 may be filtered, thereby reducing the image frames involved in subsequent calculations.

[0222] The image capture method provided in the embodiment of the present application also includes:

[0223] Step S106 ′: Filter the candidate wonderful image set according to a preset peak prominence threshold.

[0224] For example, see Figure 17 , in the case of obtaining a set of candidate wonderful images, by setting the prominence threshold of the peak, for example, the prominence threshold T prominence = 0.3, directly filter the image frames with lower matching degree, and sort the video frame images with a score greater than the prominence threshold according to the image matching score, and finally output the image frame with the highest score as the highlight moment image.

[0225] In other possible implementations, the image capture method provided in the embodiment of the present application further includes:

[0226] Step S106 : Filter the candidate wonderful image set according to a preset peak prominence threshold and a preset peak width threshold.

[0227] Furthermore, peak prominence measures how prominent the peak is relative to the surrounding baseline of the curve, and is also defined as the vertical distance between the peak and its lowest contour. Therefore, a peak width threshold can be added, for example, Width ≥ 4, to further filter image frames that meet the prominence threshold, reducing the number of frames involved in subsequent calculations.

[0228] For example, Figure 18 As shown, the mobile phone includes multiple modules, and the mobile phone implements the image capture method provided in the embodiment of the present application through the above-mentioned multiple modules. Among them, the above-mentioned multiple modules include but are not limited to one or more guidance modules, one or more encoders, a scene detection module, one or more scene matching modules, one or more matching models, a key segment selection module, and a key image selection module.

[0229] Among them, one or more guidance modules can be used to determine the guidance parameters and send the guidance parameters to the encoder. For example, one or more guidance modules may include but are not limited to a text guidance (guided_texts) module, an image guidance (guided_images) module, etc. Among them, the text guidance module can be used to determine the guidance text. The image guidance module can be used to determine the guidance image. Among them, the mobile phone can obtain the guidance text and guidance image through a network search (web_search) and input them into the corresponding guidance module. For example, Figure 18 The arrow from the text guidance module to the image guidance module may indicate that the text guidance module receives keywords input by the user, searches for images based on the keywords, and sends the searched images to the image guidance module.

[0230] Optionally, the text guidance module can receive text input by the user (for example, typed through an electronic device or voice input) and transmit the input text to the image guidance module. The image guidance module uses the input text as a keyword to search the Internet, searches for images that match the text description, and uses the image selected by the user as the guidance image.

[0231] Among them, one or more encoders can be used to convert the guidance parameters into guidance vectors. For example, the one or more encoders may include but are not limited to a text encoder (text_encoder) and an image encoder (image_encoder). The text encoder can be used to process the guidance text to obtain a guided text vector (guided_text_embedding) corresponding to the guidance text, and the guided text vector can be used to represent the characteristics of the guidance text. The image encoder can obtain a vector corresponding to the image set based on an image set. For example, a mobile phone can obtain an image set by decoding, extracting frames, and other operations on a video file or a video stream obtained by a camera, and the image set includes one or more image subsets. The image encoder can obtain a frame vector (frame_embeddings) corresponding to the image subset based on the image subset, and the frame vector can be used to represent the characteristics of the image of the image subset. The image encoder can also obtain a guided image vector (guided_image_embedding) based on the guidance image, and the guided image vector can be used to represent the characteristics of the guidance image.

[0232] The scene detection module calculates the similarity between the frame vectors of the input video and various scenes, and determines the scene labels corresponding to the frame vectors. Specifically, it determines the scene labels to which a subset of images belongs, such as splashing water, flying hair, or flying wings. The scene detection module sends the scene labels of the image frame set to one or more matching modules.

[0233] Among them, one or more matching modules can be used to screen guidance parameters that are similar to scene labels. One or more matching modules may include but are not limited to a text strategy (text_strategy) matching module and an image strategy (image_strategy) matching module. Among them, the text matching module is used to determine one or more guidance text vectors that are similar to the scene label; the image strategy matching module is used to determine one or more guidance image vectors that are similar to the scene label. It can be understood that after the mobile phone matches the scene label corresponding to the image frame set with the guidance parameter through the matching module, the mobile phone can calculate the similarity between the matched guidance parameter and the image frame set through one or more matching models.

[0234] The one or more matching models may include but are not limited to an image-text matching model (ITMoel) and an image-image matching model (IIMoel). When the text matching strategy module sends the guide text vector to the image-text matching model, the scene detection module may send a frame vector corresponding to the image frame vector to the image-text matching model. When the image matching strategy module sends the guide image vector to the image-image matching model, the scene detection module may send a frame vector corresponding to the image frame set to the image-image matching model.

[0235] Among them, ITMoel can be used to process the frame vector provided by the scene detection module with the guide text vector to obtain a picture-text matching score. The picture-text matching score is used to represent the similarity between the frame vector and the guide text vector. The higher the picture-text matching score, the higher the similarity between the frame vector and the guide text vector. IIModel is used to process the frame vector provided by the scene detection module with the image vector to obtain a picture-picture matching score. The picture-picture matching score is used to represent the similarity between the frame vector and the image vector. The higher the picture-picture matching score, the higher the similarity between the frame vector and the image vector.

[0236] The key clip selection (key_clip_selector) module can calculate a comprehensive score for the image frame set based on the image-text matching score and the image-image matching score. The key clip selection module can also determine a candidate collection of excellent images based on the comprehensive score of the image frame set. The candidate collection of excellent images is the subset of target images in the image frame set with the highest comprehensive score. For example, the key clip selection module can assign different weighting coefficients to the image-text matching score and the image-image matching score to calculate the comprehensive score. For example, the key clip selection module can assign a weighting coefficient of 0.75 to the image-text matching score and a weighting coefficient of 0.25 to the image-image matching score. In this way, since users tend to acquire information visually, assigning a higher weighting to the image-image matching score can provide users with result images that better meet their requirements. In some examples, the key clip selection module can also set a preset similarity. The key clip selection module can filter out image frames with comprehensive scores below the preset similarity. Alternatively, the key clip selection module can add image frames with comprehensive scores above the preset similarity to the candidate collection of excellent images.

[0237] The key image selection (key_frame_selector) module can obtain a wonderful moment image based on the candidate wonderful image set provided by the key segment selection module. Among them, the key image selection module can perform feature fusion of the guide picture and the guide text, and calculate the similarity between the fused target features and the image frames in the candidate wonderful image set, and output the image with the highest similarity as the wonderful moment image. The key image selection module can also obtain the output wonderful moment images multiple times in a row, and compare the similarity scores of multiple wonderful moment images to obtain the optimal wonderful moment image, thereby realizing expansion optimization on the timeline. The key image selection module can also perform aesthetic scoring on the wonderful moment images obtained multiple times, so as to output the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

[0238] In this way, the electronic device can process the input video stream according to the guidance text and guidance image input by the user through the processing of various modules, and filter out the result images that meet the user's personalized and diverse needs from the video stream, further diversifying the number and types of captured scenes.

[0239] In some possible implementations, the aesthetic evaluation model can be trained on a cloud server and then delivered to the mobile phone, or it can be trained locally on the mobile phone. Here, we will use the training of the aesthetic evaluation model on a cloud server as an example to illustrate its use.

[0240] For example, see Figure 19The cloud server can classify the collected images according to the scene. For example, the cloud server can classify the images into scene types such as looking back, lightning, parent-child, and running. In this way, the cloud server can obtain images of various scene types and ensure that the number of images of each scene in the acquired images is relatively small, allowing the aesthetic evaluation model to make reasonable evaluations of images of various scenes.

[0241] The cloud server can perform subject detection on the image, determining the image's subject. After determining the subject, it controls the number of images of different subjects to minimize the difference in the number of images of different subjects. The cloud server then receives scores from annotators based on aesthetic evaluation metrics such as composition, color, lighting, exposure, and conception. Based on the scores of each metric, the cloud server can generate an aesthetic score for the image. Depending on the subject of the image, the annotators' scoring criteria for composition metrics may vary. For example, taking the score of each measurement indicator as 10 points, when the annotator scores the composition of the image, the image can be divided into nine grids evenly. When the main subject of the image is a person, if the area where the person is located is located in the upper six grids of the nine grids, the annotator can mark the image with a score of 6 points or above on the composition factor. If the area where the person is located is located in the lower six grids of the nine grids, the annotator can mark the image with a score of 5 points or below on the composition factor. When the main subject of the image is an animal, if the area where the animal is located is located in the lower six grids of the nine grids, the annotator can mark the image with a score of 6 points or above on the composition factor. It should be noted that the measurement indicators and the corresponding score values ​​are only examples, and other methods can also be used to determine the score of the image.

[0242] The cloud server can collect the rating results of multiple maintainers for the same image and use the average of the multiple rating results as the aesthetic score of the image.

[0243] The cloud server can then classify the images. Specifically, it can classify the images by score and determine the number of images in each score range. The cloud server can maintain a consistent number of images in each score range. This ensures that the number of images in each score range is roughly equal, preventing data imbalance. Based on the similar number of images in each score range, the cloud server develops an aesthetic evaluation model that reasonably evaluates images of each score.

[0244] The cloud server can train an aesthetic evaluation model based on the correspondence between images and their aesthetic scores. For example, the aesthetic evaluation model constructed by the cloud server is as follows: Figure 20As shown, the aesthetic evaluation model includes an input image module, an encoder, and a predicted distribution module. The input image module is used to receive a set of images selected by the user. The encoder includes a backbone network, a fully-connected layer, and a normalized exponential function layer (softmax layer). The encoder is used to process the input image set to obtain an image vector for each image in the image set. The image vector is used to represent the score of each measurement indicator of the image and can be used to calculate the aesthetic score of the image. The predicted distribution module is used to process the image vector output by the encoder to obtain the aesthetic score of the image. Exemplarily, the predicted distribution module can determine the probability of the image being in each score based on the image vector. The predicted distribution module can add the product of the score and the probability, that is, calculate the mathematical expectation to obtain the aesthetic score of the image.

[0245] The mobile phone can download the aesthetic evaluation model from the cloud server. After obtaining the aesthetic evaluation model, the mobile phone can use the aesthetic evaluation model to score all images in the candidate results, thereby obtaining aesthetic scores for all images in the candidate results. Based on the aesthetic scores, the mobile phone can select a result image from the candidate results, where the result image is one or more images in the candidate results with the highest aesthetic scores. Optionally, the final highlight moment image is one or more images in the candidate results with an aesthetic score greater than a preset aesthetic score. In this way, the aesthetic evaluation model can score each image in the candidate image set and select the one or more images with the highest aesthetic scores as the final highlight moment image.

[0246] To better understand Figure 1 The software structure of the electronic device 100 is shown in FIG. The software structure of the electronic device 100 is described below. Before describing the software structure of the electronic device 100, the architecture that can be adopted by the software system of the electronic device 100 is first described.

[0247] Specifically, in actual applications, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

[0248] In addition, it is understandable that the software systems used by current mainstream electronic devices include but are not limited to Windows systems, Android systems, and iOS systems. For ease of explanation, the embodiment of the present application takes the layered architecture Android system as an example to exemplify the software structure of the electronic device 100.

[0249] In addition, the image capture solution provided in the subsequent embodiments of this application is also applicable to other systems in specific implementations.

[0250] See also Figure 21 , which is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0251] like Figure 21 As shown, the layered architecture of electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some implementations, the Android system is divided into five layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0252] The application layer can include a series of application packages. Figure 21 As shown, the application package may include camera, settings, music, Bluetooth, gallery and other applications, which are not listed here one by one and are not limited in this application.

[0253] Specifically in the embodiment of the present application, the opening entrance of the image capture solution can be set in the camera application, for example Figure 6 Control 40b-1 in (2).

[0254] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. In some implementations, these programming interfaces and programming frameworks can be described as functions. For example, the application framework layer may include a camera access interface. The camera access interface may include camera management and camera devices. The camera access interface is used to provide an application programming interface and a programming framework for camera applications.

[0255] The system library and runtime layer include the system library and the Android runtime. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0256] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0257] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0258] The system library can include multiple functional modules, such as image capture processing engine, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), shape manager, etc.

[0259] Specifically, in the embodiments of this application, the image capture processing engine is pre-installed with pre-trained models from the embodiments of this application, including an image-text matching model, an image-image matching model, an aesthetic evaluation model, and a multi-label classification model. The camera application feeds the preview video stream into the image capture processing engine, which then calls the corresponding processing model to process the video stream, thereby outputting images of the moment. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.

[0260] It can be understood that the 2D graphics engine mentioned above is a drawing engine for 2D drawing.

[0261] The HAL layer is the interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: camera hardware abstraction layer, display driver hardware abstraction layer, and power hardware abstraction layer.

[0262] The kernel layer is the layer between hardware and software. It includes at least a display driver, a camera driver, a power management driver, and a sensor driver. The display driver can include drivers for the secondary and primary screens, among others. The sensor driver can include drivers for gyroscopes, accelerometers, Hall sensors, and other sensors, used to detect angle changes between the primary and secondary screens of an outward-facing folding screen phone. The camera driver drives the corresponding camera. The capacitor management driver drives the power management module to power the display.

[0263] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0264] In addition, it is understandable that the division of the above functional modules is only an example for better understanding the technical solution of this embodiment and is not the only limitation to this embodiment. In actual application, the above functions can also be integrated into one functional module, and this embodiment does not limit this.

[0265] In addition, in actual applications, the above functional modules can also be expressed as services or frameworks. For example, the speech recognition module can be expressed as a speech recognition service or a speech recognition framework, etc. This embodiment does not impose any restrictions on this.

[0266] This concludes the introduction to the software structure of the electronic device 100. It is understood that: Figure 21 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0267] In addition, it is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0268] In addition, it should be noted that the image capture methods provided in the above embodiments implemented by electronic devices in actual application scenarios can also be performed by a chip system included in the electronic device, wherein the chip system may include a processor. The chip system can be coupled to a memory so that when the chip system is running, it calls a computer program stored in the memory to implement the steps performed by the above electronic device. The processor in the chip system can be an application processor or a processor other than an application processor.

[0269] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image capture method in the above-mentioned embodiment.

[0270] In addition, an embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the image capture method in the above-mentioned embodiment.

[0271] In addition, an embodiment of the present application also provides a chip (which can also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the image capture method in the above-mentioned embodiment to control the receiving pin to receive the signal, so as to control the transmitting pin to send the signal.

[0272] In addition, it can be seen from the above description that the electronic device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0273] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image capture method, characterized in that: Applied to electronic equipment, the method includes: In response to the image capture instruction, obtaining a preview video stream and a scene description text; Preprocessing the preview video stream to obtain an image frame set; Performing scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set; Determining guidance parameters from a preset guidance parameter set based on the scene label, the guidance parameters including a guidance picture and a guidance text; Matching the guide image and the guide text with each frame of the image frame set, respectively, to obtain an image-text matching score and an image-image matching score for each frame of the image; Determining a candidate set of wonderful images from the image frame set according to the image-text matching score and the image-image matching score of each frame of image; Performing feature fusion on the guide image and the guide text to obtain target features; Calculating the similarity between each frame image in the candidate wonderful image set and the target feature; A wonderful moment image is screened out from the candidate wonderful image set according to the similarity, where the wonderful moment image is an image frame with the highest similarity in the candidate wonderful image set.

2. The method according to claim 1, characterized in that The selecting of wonderful moment images from the candidate wonderful image set according to the similarity also includes: Continuously acquiring a plurality of the wonderful moment images, comparing the similarity of each wonderful moment image, and outputting the wonderful moment image with the highest similarity as the final wonderful moment image.

3. The method according to claim 2, characterized in that The comparing the similarity of each wonderful moment image and outputting the wonderful moment image with the highest similarity as the final wonderful moment image further includes: Scoring the multiple wonderful moment images based on the trained aesthetic evaluation model, and outputting the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

4. The method according to claim 1, wherein Determining a candidate set of wonderful images from the image frame set according to the image-text matching score and the image-image matching score of each frame of image includes: A plurality of image frames whose image-text matching scores and image-image matching scores are both greater than a preset threshold are determined as the candidate wonderful image set.

5. The method according to claim 1, wherein The step of determining a candidate set of wonderful images from the image frame set based on the image-text matching score and the image-image matching score of each frame of image further includes: Based on the image-text matching score and the image-image matching score, constructing an image-text matching score curve and an image-image matching score curve in the same coordinate system, wherein the coordinate system is an image frame-similarity coordinate system; Obtaining an intersection point of the image-text matching score curve and the image-image matching score curve; The image-to-image matching score curve is intercepted based on the intersection, and a plurality of image frames corresponding to the intercepted image-to-image score line segments are determined as a candidate wonderful image set.

6. The method according to claim 5, characterized in that After intercepting the image score curve based on the intersection, and determining multiple image frames corresponding to the intercepted image score line segments as a candidate wonderful image set, the method includes: The image score line segments are filtered according to a preset peak prominence threshold, and a plurality of image frames corresponding to the filtered image score line segments are determined as the candidate wonderful image set.

7. The method according to claim 6, characterized in that Filtering the image score line segments according to a preset peak prominence threshold also includes: The image score line segments are filtered according to a preset peak prominence threshold and a peak width threshold.

8. The method according to claim 1, characterized in that The performing scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set includes: The scene description text and the image frame set are input into a pre-trained multi-label classification model for label classification to obtain one or more scene labels corresponding to the image frame set.

9. The method according to claim 1, characterized in that Before acquiring the preview video stream and the scene description text based on the image capture instruction triggered by the user, the method further includes: Displaying a first interface, wherein the first interface includes a camera application icon; Detecting a first click operation on the camera application icon; In response to the first click operation, displaying a second interface, wherein the second interface includes a snapshot start icon; Detecting a second click operation on the snapshot start icon; In response to the second click operation, displaying a scene selection bar in the second interface; detecting completion of a third click operation on the scene selection bar; In response to the third click operation, an image capture instruction is sent, and a preview image is displayed in the second interface.

10. The method according to claim 9, characterized in that In response to the second click operation, displaying a scene selection bar in the second interface further includes: In response to the second click operation, displaying a scene definition option on the second interface; detecting a fourth click operation on the scene definition option; In response to the fourth click operation, a third interface is displayed, where the third interface includes a first input control and a second input control, where the first input control is used to add the guide text, and the second input control is used to add the guide picture.

11. An electronic device, characterized in that: The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image capture method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the image capture method according to any one of claims 1 to 10.

Images