Image snapshot method and device and storage medium

Abstract

The invention provides an image snapshot method and device, and a storage medium. The method comprises the following steps: performing scene classification on an obtained preview video stream based on a scene description text input by a user, performing strategy matching on a scene label obtained by classification and a preset guide parameter set, determining a guide parameter, matching the guide parameter with the preview video stream to obtain an image matching score and an image-text matching score of each frame of image, and carrying out similarity calculation on the target features obtained by carrying out feature fusion based on the guide parameters and each frame of image in the candidate wonderful image set, and outputting the candidate wonderful image with the highest similarity as a wonderful moment image. Therefore, precise snapshot of the wonderful moments of the target scene is realized, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image capture method, device and storage medium. Background Art

[0002] When using the camera of an electronic device, users usually want to capture images of specific scenes (hereinafter referred to as "wonderful moments"). Currently, electronic devices provide "continuous shooting" and "automatic snapshot" functions, but due to problems such as human reaction delay and electronic device transmission delay, users often find it difficult to capture truly wonderful moments. This greatly reduces the user experience. Summary of the invention

[0003] The embodiments of the present application provide an image capture method, device and storage medium, which are intended to enable users to capture wonderful moments occurring in a target scene during the photo-taking process, thereby improving capture accuracy and enhancing the user experience.

[0004] In a first aspect, an embodiment of the present application provides an image capture method, which is applied to an electronic device, the method comprising: in response to an image capture instruction, obtaining a preview video stream and a scene description text; preprocessing the preview video stream to obtain an image frame set; classifying the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set; determining a guide parameter from a preset guide parameter set based on the scene label, the guide parameter including a guide picture and a guide text; matching the guide picture and the guide text with each frame image in the image frame set respectively to obtain a picture-text matching score and an picture-picture matching score of each frame image; determining a candidate wonderful image set from the image frame set according to the picture-text matching score and the picture-picture matching score of each frame image; performing feature fusion on the guide picture and the guide text to obtain a target feature; calculating the similarity between each frame image in the candidate wonderful image set and the target feature; screening out a wonderful moment image from the candidate wonderful image set according to the similarity, the wonderful moment image being the image frame with the highest similarity in the candidate wonderful image set.

[0005] The preview video stream can be Fig.10 After the mobile phone turns on the image capture function in response to the user's image capture instruction, the mobile phone obtains the video stream of the target scene through the camera.

[0006] The scene description text can be Figure 7 , which is input by the user through the "Select Scene" option. The scene description text can be a description of the target scene by the user, as well as a description of the captured image that is expected to be obtained.

[0007] In some possible implementations, the scene description text may be input in advance by the user, such as a guide text; after acquiring the preview video stream, the mobile phone obtains the scene label by matching it with the scene description text.

[0008] Among them, since there may be a large number of repeated video frames in the preview video stream, it is also necessary to perform a preprocessing operation of extracting frames after decoding the preview video stream, so as to improve the efficiency of subsequent data processing.

[0009] The guidance parameters in the preset guidance parameter set may be input by the user in advance, or may be placed when the system module corresponding to the method of the embodiment of the present application is initialized. The preset guidance parameter set includes partial scene images and description texts that the user expects to obtain.

[0010] The guide image and each frame image in the image frame set are input into the image-image matching model in the following embodiment, so as to obtain the image-image matching score of each frame image in the image frame set.

[0011] The guiding text and each frame image in the image frame set are input into the image-text matching model in the following embodiment, and the image-text matching score of each frame image in the image frame set can be obtained.

[0012] The step of fusing the features of the guide image and the guide text may be to first extract the features of the guide image, and then fuse the extracted image features with the features of the guide text to obtain the target features.

[0013] Therefore, after the user turns on the image capture function, by obtaining the preview video stream of the target scene and the scene description text input by the user, the obtained preview video stream is classified based on the scene description text input by the user, the classified scene label is strategically matched with the preset guidance parameter set, the guidance parameters are determined, the guidance parameters are matched with the preview video stream, and the image-image matching score and image-text matching score of each frame of the image are obtained, so as to determine the candidate wonderful image set, and the target features obtained by feature fusion based on the guidance parameters are calculated with each frame of the candidate wonderful image set for similarity, and the candidate wonderful image with the highest similarity is output as the wonderful moment image. In this way, the wonderful moments of the target scene can be accurately captured, and the user experience is improved.

[0014] According to the first aspect, the method of selecting a wonderful moment image from a set of candidate wonderful images according to the similarity further includes: continuously acquiring a plurality of the wonderful moment images, comparing the similarity of each wonderful moment image, and outputting the wonderful moment image with the highest similarity as the final wonderful moment image.

[0015] Among them, the process of obtaining a wonderful moment image can be regarded as a calculation cycle. When capturing the wonderful moment image in the target scene, in order to improve the accuracy of capturing, the wonderful moment images in multiple calculation cycles can be continuously obtained, and the similarity of each wonderful moment image can be compared, and the final wonderful moment image can be determined by the similarity. In this way, the image capture method can be optimized on the time axis.

[0016] According to the first aspect, or any implementation of the first aspect above, comparing the similarity of each wonderful moment image and outputting the wonderful moment image with the highest similarity as the final wonderful moment image, further includes: scoring the multiple wonderful moment images based on a trained aesthetic evaluation model, and outputting the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

[0017] Among them, the trained aesthetic evaluation model can evaluate the image based on multiple indicators. For example, these indicators include but are not limited to composition, color, light, exposure, conception, etc.

[0018] According to the first aspect, or any implementation of the first aspect above, determining a candidate set of wonderful images from the image frame set based on the image-text matching score and the image-image matching score of each frame of image includes: determining a plurality of image frames whose image-text matching scores and the image-image matching scores are both greater than a preset threshold as the candidate set of wonderful images.

[0019] The preset threshold value may be set according to actual needs.

[0020] According to the first aspect, or any implementation method of the first aspect above, determining a set of candidate wonderful images from the set of image frames based on the image-text matching score and the image-image matching score of each frame of image, also includes: constructing a picture-text matching score curve and a picture-image matching score curve in the same coordinate system based on the picture-text matching score and the picture-image matching score, and the coordinate system is an image frame-similarity coordinate system; obtaining the intersection of the picture-text matching score curve and the picture-image matching score curve; intercepting the picture-image score curve based on the intersection, and determining multiple image frames corresponding to the intercepted picture-image score segments as the set of candidate wonderful images.

[0021] According to the first aspect, or any implementation method of the first aspect above, after intercepting the TuTu score curve based on the intersection, and determining the multiple image frames corresponding to the intercepted TuTu score line segment as a set of candidate wonderful images, it includes: filtering the TuTu score line segment according to a preset peak prominence threshold, and determining the multiple image frames corresponding to the filtered TuTu score line segment as a set of candidate wonderful images.

[0022] According to the first aspect, or any implementation of the first aspect above, filtering the image score line segment according to a preset peak protrusion threshold, also includes: filtering the image score line segment according to a preset peak protrusion threshold and a peak width threshold.

[0023] According to the first aspect, or any implementation of the first aspect above, the scene classification of the image frame set based on the scene description text to obtain scene labels corresponding to the image frame set includes: inputting the scene description text and the image frame set into a pre-trained multi-label classification model for label classification to obtain one or more scene labels corresponding to the image frame set.

[0024] According to the first aspect, or any implementation of the first aspect above, before the image capture instruction based on user triggering obtains the preview video stream and scene description text, the method also includes: displaying a first interface, the first interface including a camera application icon; detecting a first click operation on the camera application icon; in response to the first click operation, displaying a second interface, the second interface including a capture start icon; detecting a second click operation on the capture start icon; in response to the second click operation, displaying a scene selection bar in the second interface; detecting completion of a third click operation on the scene selection bar; in response to the third click operation, sending an image capture instruction and displaying a preview image in the second interface.

[0025] For specific implementation details of this part, please refer to the embodiment for Figure 6 and Figure 7 The description part will not be repeated here.

[0026] According to the first aspect, or any implementation of the first aspect above, in response to the second click operation, a scene selection bar is displayed in the second interface, and it also includes: in response to the second click operation, finding the option for displaying scene definition in the second interface; detecting a fourth click operation on the scene definition option; in response to the fourth click operation, displaying a third interface, the third interface including a first input control and a second input control, the first input control is used to add the guide text, and the second input control is used to add the guide picture.

[0027] For specific implementation details of this part, please refer to the embodiment for Figure 7 and Figure 8 The description part will not be repeated here.

[0028] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes instructions of the method in the first aspect or any possible implementation of the first aspect.

[0029] In a third aspect, an embodiment of the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer program, which includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0033] Figure 2 This is a schematic diagram of an exemplary user's photo-taking operation;

[0034] Figure 3 is a schematic diagram of an exemplary wonderful moment;

[0035] Figure 4 A schematic diagram of an exemplary user interface;

[0036] Figure 5 is a schematic diagram of an automatic snapshot interface shown as an example;

[0037] Figure 6 This is an exemplary schematic diagram of a user entering a snapshot scene operation;

[0038] Figure 7 The following is a schematic diagram of an exemplary user selecting a scene operation;

[0039] Figure 8 The following is a schematic diagram of an operation of a user-defined scenario shown as an example;

[0040] Fig. 9 is another schematic diagram of a user-defined scenario operation shown as an example;

[0041] Fig.10 is a schematic diagram of a scene capture shown as an example;

[0042] Fig.11 is a schematic diagram of an exemplary video capture operation;

[0043] Fig.12 is a schematic diagram of an exemplary image capture processing flow;

[0044] Fig.13 The following is a schematic diagram of a multi-label scene classification process shown as an example;

[0045] Fig.14 is a schematic diagram of a similarity score curve shown as an example;

[0046] Fig.15 is a schematic diagram of image feature processing shown as an example;

[0047] Fig.16 The figure is a schematic diagram of exemplarily showing the best wonderful moment image screening;

[0048] Fig.17 is another schematic diagram of a similarity score curve shown as an example;

[0049] Fig.18 is a schematic diagram of an image capture system framework shown as an example;

[0050] Fig.19 A schematic diagram of an aesthetic model training is shown as an example;

[0051] Fig. 20 is a schematic diagram of an aesthetic evaluation model shown as an example;

[0052] Fig.21 The software structure diagram of an electronic device is shown as an example. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0055] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0058] In order to better understand the technical solution provided by the embodiment of the present application, before describing the technical solution of the embodiment of the present application, the hardware structure of the electronic devices (such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, and cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.) to which the embodiment of the present application is applicable is first described in conjunction with the accompanying drawings. For the convenience of description, Figure 1 Let's take a mobile phone as an example.

[0059] See also Figure 1The mobile phone 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0060] Among them, the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor (Modem), a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc., which are not listed one by one here and the present application does not limit this.

[0061] The controller as the processing unit mentioned above may be the nerve center and command center of the mobile phone 100. In practical applications, the controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0062] Regarding the neural network processor mentioned above, specifically in the technical solution provided in the embodiment of the present application, it can be used to train the neural network model for capturing wonderful moments mentioned in the embodiment of the present application. Understandably, in order to reduce the resource occupation of the mobile phone 100, the neural network model can also be trained by a cloud server or other server and sent to the mobile phone 100.

[0063] The ISP mentioned above is used to output digital image signals to the DSP for processing.

[0064] The DSP mentioned above is used to convert digital image signals into image signals in standard RGB, YUV and other formats.

[0065] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation to this embodiment.

[0066] In addition, the processor 110 may further include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc., which are not listed here one by one, and the present application does not limit this.

[0067] In the embodiment of the present application, the processor 110 can be used to obtain the preview video stream or video stream being shot by the user from the camera; the processor 110 can also obtain the video selected by the user and the guidance parameters. The guidance parameters include but are not limited to one or more of the guidance text and the guidance picture, and the guidance parameters can be used to obtain the image most similar to the guidance parameters from the preview video stream or video stream.

[0068] Continue to see Figure 1 The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0069] Continue to see Figure 1 The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area.

[0070] Continue to see Figure 1 The charging management module 140 is used to receive charging input from the charger.

[0071] Continue to see Figure 1The wireless communication function of the mobile phone 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0072] It should be noted that, in some implementations, the neural network model used to capture wonderful moments can be obtained by training a cloud server or other server. For this implementation scenario, the mobile phone 100 can communicate with the cloud server or other server that provides the neural network through the mobile communication module 150 or the wireless communication module 160. For example, the mobile phone 100 can send a request to obtain or update the neural network model to the cloud server through the mobile communication module 150. Accordingly, the cloud server can send the trained neural network model to the mobile phone 100 according to the request of the mobile phone 100.

[0073] In addition, it should be noted that in the scenario where the neural network model is trained by a cloud server (or other server), the cloud server can customize the neural network model suitable for different mobile phones 100 according to the customization requirements corresponding to mobile phones 100 with different configurations, and update and iterate the training of the neural network model based on the capture results fed back by different mobile phones 100.

[0074] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation to this embodiment.

[0075] Continue to see Figure 1 The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., which are not listed here one by one and are not limited in this application.

[0076] Continue to see Figure 1 , the camera 193 is used to capture static images or videos. The mobile phone 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0077] Continue to see Figure 1 , display screen 194 is used to display images, videos, etc.

[0078] This concludes the introduction to the hardware structure of the mobile phone 100. It should be understood that Figure 1 The illustrated mobile phone 100 is merely an example, and in a specific implementation, the mobile phone 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different component configuration. Figure 1The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0079] "A wonderful moment" refers to a special, important, memorable or exciting moment. These moments may be in personal life, such as birthdays, graduations, weddings, etc., or in public events, such as exciting games at sports events, climaxes at concerts, etc. Wonderful moments usually have characteristics such as uniqueness, importance, memorability and excitement. These moments can often inspire people's emotions and bring strong emotional experiences, such as joy, touching, shocking, etc. In different fields, wonderful moments also have different forms of expression. In sports competitions, a wonderful moment may be a key goal, a wonderful save or an exciting moment of the game. In a concert, a wonderful moment may be a moving melody, a thrilling chord or an intoxicating performance. In short, wonderful moments are an indispensable part of people's lives. They make people feel the beauty and richness of life, and they have also become an important content for people to recall and share.

[0080] For example, when photographing scenic spots and historical sites, the moment when the camera images the building completely, clearly, without shaking and noise is a wonderful moment; when photographing a hurdler, the moment when the athlete jumps across the hurdle and the image is clear, without shaking and noise is a wonderful moment; when photographing a close-up of a person, the moment when the person opens his eyes and smiles and the image is clear, without shaking and noise is a wonderful moment. The photos output after the images of the wonderful moments are processed are called wonderful photos.

[0081] However, during the shooting process, due to problems such as human reaction delay and transmission delay of electronic devices, users often find it difficult to take wonderful photos of truly wonderful moments.

[0082] In order to better understand the existing process of capturing wonderful moments, the following is combined with the attached Figure 2 To Attachment Figure 5 Provide explanation.

[0083] See also Figure 2 In (1), a mobile phone interface 10a is exemplarily shown. Exemplarily, the interface 10a displays multiple application icons, such as the icon 10a-1 of the camera application, and icons of applications such as address book, phone, information, clock, calendar, gallery, memo, file management, email, music, calculator, video, recorder, weather, browser, and settings.

[0084] It should be noted that in some possible implementations, Figure 2The interface 10a shown in (1) can be referred to as the main interface. When the user clicks the icon 10a-1 in the interface 10a, the camera application can be used to implement shooting functions, such as taking photos, shooting (recording) videos, etc.

[0085] Continue to refer to Figure 2 In (1), for example, when the user clicks the icon 10a-1 of the camera application, the mobile phone responds to the user operation, identifies the icon corresponding to the user's click operation as the icon of the camera application, and then calls the corresponding interface in the application framework layer to start the camera application, and starts the camera driver by calling the kernel layer, and collects the image stream (preview stream in this case) through the camera. At this time, the mobile phone displays the interface of the camera application, for example Figure 2 The interface 10b shown in (2) will display a screen corresponding to the preview stream in the interface 10b, such as Figure 2 (2) shows a screen of a user playing basketball.

[0086] As the shooting function of mobile phones improves, the camera application supports more and more shooting modes. For example, the shooting mode may include aperture mode, night scene mode, portrait mode, photo mode, video mode, smile mode, professional mode, etc. Figure 2 Mode options are displayed in the shooting mode list 10b-2 in (2).

[0087] For example, when the user clicks on an icon option corresponding to a shooting mode, the mobile phone displays the camera application interface in the corresponding shooting mode. For example, if the user clicks on the icon option of the photo mode, the mobile phone displays the interface of the camera application in the photo mode. Figure 2 Interface 10b shown in (2).

[0088] It should be noted that in some possible implementations, after the mobile phone starts the camera application, the camera application selects the photo mode by default, that is, the photo mode is the default shooting mode of the camera application. Among them, in the photo mode, the photo shutter control 10b-1 will also be displayed in the interface 10b. Exemplarily, when the user clicks the photo shutter control 10b-1, for example, clicking the photo shutter control 10b-1 N times in succession, the mobile phone will detect the user operation acting on the photo shutter control 10b-1, and then take a photo in response to the operation (shoot N times, and you will get pictures P1, P2...Pn respectively). At this time, the thumbnail corresponding to the control 10b-3 displayed in the interface 10b for opening the system's built-in gallery will be updated to the thumbnail of the user playing basketball that has just been taken (the thumbnail of the picture taken by clicking the photo shutter control 10b-1 for the last time), such as Figure 2 Control 10b-3' in interface 10b' shown in (3).

[0089] Among them, in the above shooting scene, picture P5 corresponds to the wonderful moment in the basketball scene, that is, picture P5 is the wonderful photo that the user expects to get. Pictures P1 and Pn are photos of the user dribbling the ball, that is, photos that the user does not expect to get. The above photos, time and user expectations are used to establish a coordinate system (see Figure 3 ), the user needs to press the shutter button at time t2 to capture the wonderful moment. However, due to the human reaction delay (including the time required to complete the shooting operation), the time when the mobile phone receives the user's shooting operation will lag, resulting in the failure to capture the image P5 of the wonderful moment.

[0090] To this end, the manufacturer also provides a "smart photo" function to assist users in capturing wonderful moments.

[0091] See also Figure 4 In (1), a mobile phone camera function setting interface 20a is exemplarily shown. Exemplarily, the interface 20a includes multiple function icons, such as smart code scanning, AI photography, flash, filter and setting 20a-1 icons.

[0092] Continue to see Figure 4 In (1), when the user clicks the icon 20a-1 in the interface 20a, the setting interface of the camera application can be entered.

[0093] Continue to see Figure 4 In (1), for example, when the user clicks icon 20a-1, the mobile phone responds to the user operation and enters the camera application setting interface. At this time, the mobile phone displays the setting interface, such as Figure 4 Interface 20b shown in (2).

[0094] See also Figure 4 In (2), the camera setting interface 20b is exemplarily shown. Exemplarily, the interface 20b includes setting parameters for taking photos and videos. The photo parameters include photo ratio, smart photo 20b-1 and filter recommendation; the video parameters include video resolution, video frame rate, high-efficiency video format, high dynamic range lighting rendering (HDR10+), AI movie tone and one-shot multi-gain.

[0095] Continue to see Figure 4 In (2), when the user clicks on the smart photo taking 20b-1 in the interface 20b, the smart photo taking setting interface can be entered.

[0096] Continue to see Figure 4 In (2), for example, when the user clicks on the smart camera 20b-1, the mobile phone responds to the user operation and enters the smart camera setting interface. At this time, the mobile phone displays the smart camera setting interface, for example Figure 4 Interface 20c shown in (3).

[0097] See also Figure 4 In (3), an intelligent photo setting interface 20c is exemplarily shown. Exemplarily, the interface 20c includes multiple intelligent photo functions and their corresponding setting buttons, such as voice-controlled photo, gesture photo, smile capture and automatic capture 20c-1.

[0098] It should be noted that the automatic capture in the above interface 20c is performed by the mobile phone when it recognizes a preset specific scene. The above specific scenes include people's smiling faces, jumping, running, and automatic photos of cats, dogs and other wonderful moments.

[0099] Continue to see Figure 4 In (3), for example, when the user clicks on the automatic capture 20c-1 to turn on the automatic capture, the camera application turns on the automatic capture function and returns to the user photo taking interface, for example Figure 5 Interface 30a shown in (1).

[0100] See also Figure 5 In (1), a photo taking interface 30a with the automatic snapshot function turned on is exemplarily shown. After the mobile phone enters the interface 30a, the user points the mobile phone camera at the target scene, and the camera application on the mobile phone automatically recognizes the wonderful moment and takes a snapshot.

[0101] Continue to see Figure 5 In (1), for example, when the camera application recognizes a "jumping" scene, it automatically takes a snapshot. In this way, the mobile phone can trigger the snapshot through a specific scene to capture the wonderful moment without the user's operation during the photo-taking process.

[0102] However, this method can only capture images based on limited scenes and preset actions. The resulting image scenes and types are very limited, lack versatility, and cannot meet the personalized and diverse needs of users.

[0103] In view of this, an embodiment of the present application provides an image capture method, which captures the target scene according to the user's personalized and diverse needs, thereby obtaining wonderful moments and improving the user experience.

[0104] In a possible implementation, the image taken using the camera application is still used as Figure 2 Taking the scene of the user jumping and dunking shown in (2) as an example, based on the image capture method provided in the embodiment of the present application, after determining the requirements input by the user, the mobile phone converts the above requirements into guidance parameters, and based on the guidance parameters, determines the wonderful moments from the preview video stream obtained by the camera application.

[0105] In order to explain this beneficial effect in detail, the following Figures 6 to 10 Provide explanation.

[0106] See also Figure 6 In (1), the main interface 40a of the mobile phone is shown as an example. For example, the main interface 40a includes a camera application 40a-1. When the user clicks the camera application 40a-1, the mobile phone responds to the user operation and enters the camera's photo taking interface. At this time, the mobile phone displays the camera's photo taking interface, for example Figure 6 Interface 40b shown in (2).

[0107] It should be noted that the image capture method provided in the embodiment of the present application can be combined with a camera application to provide services to users, or can be combined with other applications to provide services to users. When combined with other applications, the image capture method provided by the image capture method needs to be called in other applications.

[0108] Exemplarily, the above-mentioned other applications may include image photography applications, or may include image editing applications, which is not limited in this embodiment of the present application.

[0109] See also Figure 6 In (2), an image capture start interface 40b is exemplarily shown. Exemplarily, the start interface 40b includes a capture start control 40b-1. The user can use the image capture function by clicking the control 40b-1 to set the scene.

[0110] It should be noted that, compared with the above interface 40b and Figure 5 As can be seen from the interface 30a provided in (1), a snapshot start control 40b-1 is added to the lower left corner of the interface 40b. The position of the control 40b-1 can be set according to the user's operating habits, and it can also be set in the function setting bar at the top of the camera application, or in other positions that are more convenient for the user to operate. This embodiment of the application does not limit this.

[0111] Continue to see Figure 6 In (2), for example, when the user clicks on the control 40b-1, the mobile phone responds to the user's operation and enters the scene setting interface of the image capture interface. At this time, the mobile phone displays the scene setting interface, for example Figure 7 Interface 50a shown in (1).

[0112] See also Figure 7 In (1), a scene setting interface 50a is exemplarily shown. The interface 50a includes a scene selection control 50a-1.

[0113] Exemplarily, after the user clicks on the control 40b-1, the mobile phone responds to the operation and displays the control 50a-1. The control 50a-1 displays the "Select Scenario" option and the "Define Scenario" option.

[0114] It should be noted that the above-mentioned "Select Scene" option may correspond to a preset scene, and the above-mentioned preset scenes include but are not limited to half-length portrait, landscape, dynamic moment, sports, smiling face, and street scene.

[0115] In some possible implementations, the above-mentioned preset scenes may be pre-installed in the mobile phone, or may be obtained in real time by the mobile phone accessing the cloud server through the network, and this embodiment does not impose any limitation on this.

[0116] In some other possible implementations, the above-mentioned preset scenes may also be generated by collecting historical customized scenes of users.

[0117] Continue to see Figure 7 In (1), for example, after the user clicks the "Select Scene" option, the mobile phone responds to the operation and displays a preset scene selection interface, such as Figure 7 Interface 50b shown in (2).

[0118] See also Figure 7 In (2), a scene selection interface 50b is exemplarily shown. Exemplarily, the interface 50b includes a scene selection control 50b-1, which is a sliding selection control. The control lists all preset scenes, and the user can slide left and right in the control 50b-1 to select the desired preset scene.

[0119] Exemplarily, the information included in the above-mentioned preset scene includes a scene picture and a scene text description.

[0120] It should be noted that after the user selects a preset scene, the mobile phone can obtain the corresponding scene picture and scene text description, and convert them into guidance parameters, and then use the image capture method provided in the embodiment of the present application to capture the target scene based on the guidance parameters.

[0121] Continue to see Figure 7 In (2), illustratively, after the user selects the "sports" scene in the control 50b-1, the image capture interface can be entered.

[0122] In some other possible implementations, in order to further improve the accuracy of the snapshot, the scene in the control 50b-1 can also be refined. That is, after the user clicks the "sports" scene in the control 50b-1, the scene selection interface is entered, for example Figure 7 Interface 50c shown in (3).

[0123] See also Figure 7In (3), a subdivided scene selection interface 50c is exemplarily shown. The interface 50c includes a subdivided scene selection control 50c-1, which is a sliding selection control. The control lists all subdivided scenes corresponding to the preset scene, and the user can slide left and right in the control 50c-1 to select the required subdivided scene.

[0124] Continue to see Figure 7 In (3), for example, after the user clicks on the "sports" scene in the control 50b-1 of the interface 50b, the mobile phone displays the interface 50c, and lists the subdivided scenes of the "sports" scene, such as jump slam, flying slam, vertical jump and fancy jump, in the control 50c-1 of the interface 50c. After the user completes the selection of the preset scene and the subdivided scene, the mobile phone displays the image capture interface, and obtains the video stream of the current scene through the camera, and captures the wonderful moment from the video stream, thereby realizing the image capture function provided by the embodiment of the present application.

[0125] In some other possible implementations, the above preset scenes cannot meet the personalized needs of users. Therefore, the embodiments of the present application also provide the option of customizing scenes.

[0126] Continue to see Figure 7 In (1), for example, after the user clicks the "Define Scene" option, the mobile phone responds to the operation and displays a custom scene interface, for example Figure 8 Interface 60a shown in (1).

[0127] See also Figure 8 In (1), a custom scene interface 60a is exemplarily shown. The interface 60a includes a text prompt bar, an operation bar 60a-1 for guiding text, and an operation bar 60b-2 for guiding images. The operation bar 60a-1 includes title information, an add button, and a modify button; the operation bar 60b-2 includes title information, an add button, and a modify button.

[0128] Exemplarily, the content in the above text prompt bar is "Please select the guidance parameters, the mobile phone will capture the wonderful moment image according to the selected guidance parameters."

[0129] Exemplarily, the title information of the operation bar 60a-1 is "Guide text", and the title information of the operation bar 60b-2 is "Guide image".

[0130] It is understandable that the content in the above-mentioned text prompt bar, the title information of the operation bar 60a-1 and the title information of the operation bar 60b-2 may also be other text content and / or image content that is easier for users to understand, and the embodiment of the present application does not limit this.

[0131] Continue to see Figure 8In (1), for example, after the user clicks the add button in the operation bar 60a-1, the mobile phone responds to the operation and displays a guide text adding interface, for example Figure 8 Interface 60b of (2).

[0132] See also Figure 8 In (2), a guide text adding interface 60b is exemplarily shown. The interface 60b includes one or more options. The one or more options can be used to trigger the mobile phone to display guide texts from different sources. The one or more options include but are not limited to the "select template text" option, the "custom input text" option, the "local input text" option, and the "network search text" option.

[0133] Among them, the "select template text" option is used to trigger the mobile phone to display an interface containing template text, and the interface containing template text may include one or more sentences of template text, and the one or more sentences of template text may be used as guide text. The mobile phone may receive input for the template text and use the template text as guide text. The "custom input text" option is used to trigger the mobile phone to display a text box, and the mobile phone may receive text input for the text box, and use the text as guide text. The "local input text" option is used to trigger the mobile phone to display an interface of an application that stores local text, such as a notepad application interface, and the interface of the notepad application may include one or more sentences of text, and the one or more sentences of text may be used as guide text. The mobile phone may receive input of text selected by the user, and use the selected text as guide text. The "network search text" option is used to trigger the mobile phone to display a network search interface, and the network search interface may include a search bar. The mobile phone may receive keywords input for the search bar, search for text corresponding to the keywords based on the keywords, receive input of text selected by the user, and use the selected text as guide text.

[0134] Based on the guide text added by the user, the mobile phone can obtain the video stream through the camera and capture the wonderful moments from the video stream, thereby realizing the image capture function provided in the embodiment of the present application.

[0135] It should be noted that Figure 7 The "Define Scene" option in (1) can also be independently set as a user-defined scene entry, so that users can input customized scenes in advance. Its entry can be set in the system setting interface or can be independently used as a system application, which is not limited in the present embodiment.

[0136] In addition, it should be noted that in some possible implementations, Figure 7 If the "Define Scene" option in (1) is set to a separate entry, Figure 7 The "Define Scene" option in (1) can also be cancelled, leaving only the "Select Scene" option.

[0137] In some other possible implementations, the user can also Figure 8 In the interface 60a of (1), the add button of the operation bar 60a-2 is clicked, and the mobile phone responds to the operation by displaying the guide image adding interface, for example Fig. 9 Interface 70a of (1).

[0138] See also Fig. 9 In (1), a guide image adding interface 70a is exemplarily shown. The interface 70a includes one or more options. The one or more options can be used to trigger the mobile phone to display guide images from different sources. The one or more options include but are not limited to the "select template image" option, the "custom input image" option, the "local input image" option 70a-1 and the "network search image" option.

[0139] Among them, the "select template image" option is used to trigger the mobile phone to display an interface containing a template image. The interface containing a template image may include one or more template images. The one or more template images may be used as a guide image. The mobile phone may receive input for the template image and use the template image as a guide image. The "custom input image" option is used to receive an image input by a user and use the input image as a guide image. For example, the "custom input image" option may receive an image drawn by a user input through a touch pad or input device (e.g., a stylus, a digital tablet, etc.), and use the input image as a guide image. The "local input image" option 70a-1 is used to trigger the mobile phone to display an interface of an application that stores local images, such as a gallery application interface and a file management application interface. The "network search image" option is used to trigger the mobile phone to display a network search interface. The network search interface may include a search bar. The mobile phone may receive keywords input for the search bar, search for images corresponding to the keywords based on the keywords, receive input of an image selected by the user, and use the selected image as a guide image.

[0140] Continue to see Fig. 9 In (1), for example, after the user clicks the "local input picture" option 70a-1, the mobile phone responds to the operation and displays the gallery interface. After the user selects the guide picture through the gallery interface, the mobile phone displays the guide picture confirmation interface, for example Fig. 9 Interface 70b shown in (2).

[0141] See also Fig. 9 In (2), a confirmation interface 70b of the guide image is exemplarily shown. Exemplarily, after the user completes uploading the image, an information confirmation bar 70b-1 pops up in the interface 70b. The information confirmation bar 70b-1 includes prompt information, a confirmation button, and a cancel button.

[0142] Exemplarily, after the user clicks the OK button, the mobile phone uses the currently displayed image as the guide image; after the user clicks the Cancel button, the mobile phone returns to the interface 70a so that the user can re-upload the guide image.

[0143] After the guide image is added, the mobile phone can obtain the video stream through the camera based on the guide image added by the user, capture the wonderful moments from the video stream, thereby realizing the image capture function provided in the embodiment of the present application.

[0144] In some other possible implementations, the user can also upload the guide image and the guide text at the same time, so that the mobile phone can capture the wonderful moments in the video stream based on the guide image and the guide text, thereby improving the accuracy of the capture. Fig.10 , Fig.10 A set of schematic diagrams for capturing wonderful moments based on guide pictures and guide texts are shown.

[0145] See also Fig.10 In (1), an example is shown of an interface 80a after a user turns on the image capture function. The interface 80a includes a capture status control 80a-1. When the user turns on the capture, Figure 6 The snapshot start control 40b-1 in (2) will be selected and have a background color added, becoming the status control 80a-1 in the interface 80a.

[0146] Exemplarily, after the user points the mobile phone camera at the target scene, the real-time image of the scene is displayed in the image preview frame of the camera, and the preview video stream is sent to the mobile phone.

[0147] See also Fig.10 In (2), a preview video stream acquisition interface 80b is exemplarily shown. The interface includes an image preview area and a thumbnail control 80b-1.

[0148] Exemplarily, after the user points the camera of the mobile phone at the target scene, the mobile phone continuously obtains the preview video stream of the target scene through the camera, wherein the preview video stream includes a P1 image frame, a P5 image frame, and a Pn image frame.

[0149] Exemplarily, after the mobile phone obtains the preview video stream, the wonderful moments in the scene are captured using the above-mentioned guide image, guide text and the image capture method provided in the embodiment of the present application.

[0150] Exemplarily, the mobile phone stores the captured wonderful moment in a gallery and updates the thumbnail on the thumbnail control 80b-1 in the interface 80b.

[0151] See also Fig.10In (3), the interface 80c after the thumbnail is updated is exemplarily shown. Exemplarily, in the interface 80c, the mobile phone determines that the picture frame P5 is a wonderful moment based on the guide picture and the guide text through the picture capture method provided by the embodiment of the present application, stores the picture frame P5 in the gallery, and updates the thumbnail of the thumbnail control 80b-1 in the interface 80b, which is displayed as the thumbnail control 80b-1'.

[0152] At this point, the embodiment of the present application has completed the automatic capture of the wonderful moments of the target scene based on the guide picture, guide text and the image capture method provided by the embodiment of the present application, which greatly improves the accuracy and versatility of image capture.

[0153] In another possible implementation, the user can also open the video recording function of the camera application on the mobile phone. Figure 2 The scene of the user jumping and dunking as shown in (2) is recorded, and the recorded video is used with the image capture method provided in the embodiment of the present application to obtain the wonderful moments from the video.

[0154] In order to explain this beneficial effect in detail, the following Fig.11 Provide explanation.

[0155] See also Fig.11 , exemplarily showing a set of scenes for capturing images of videos in a gallery.

[0156] See also Fig.11 In (1), a mobile phone main interface 90a is shown as an example. The main interface 90a includes multiple application icons, including a clock, a calendar, a gallery 90a-1, a notepad, a file management, etc.

[0157] For example, the user clicks on the gallery 90a-1, and the mobile phone responds to the operation and enters the gallery interface, for example Fig.11 Interface 90b shown in (2).

[0158] See also Fig.11 In (2), a gallery interface 90b is shown as an example. In the interface 90b, pictures and videos taken / downloaded by the user are displayed in columns according to the shooting time / download time.

[0159] Exemplarily, the interface 90b includes a video 90b-1 shot by the user yesterday and a guide picture 90b-2 downloaded.

[0160] In some possible implementations, the guide image 90b-2 is used to guide the image capture method provided in the embodiment of the present application to capture the wonderful moments in the video 90b-1.

[0161] It should be noted that a large number of videos and pictures that have been shot by the user are displayed in the gallery interface 90b. These videos and pictures can be shot by the user himself or downloaded by the user online. The user hopes to capture the wonderful moments of the shot video 90b-1 through the downloaded picture 90b-2.

[0162] Continue to see Fig.11 In (2), for example, after the user clicks on the video 90b-1, the mobile phone responds to the operation and enters the video operation interface, for example Fig.11 Interface 90c shown in (3).

[0163] See also Fig.11 In (3), a video operation interface 90c is exemplarily shown. The interface 90c includes a plurality of video operation controls, such as a video play control, a share control, a delete control, a filter control 90c-1 and more function controls.

[0164] Exemplarily, after the user clicks the filter control 90c-1, the mobile phone captures the wonderful moments of the video 90b-1 by executing the image capture method provided in the embodiment of the present application, and updates the captured picture as the video cover to the video 90b-1.

[0165] At this point, the embodiment of the present application has completed the automatic capture of wonderful moments of the video in the gallery based on the guide picture, guide text and the image capture method provided by the embodiment of the present application, which greatly improves the accuracy and versatility of image capture.

[0166] It should be noted that after the user clicks the filter control 90c-1, the gallery application can also display Figure 8 The user can add guide text and guide image through the guide parameter setting interface to capture the wonderful moments of the video 90b-1.

[0167] In addition, it should be noted that in some possible implementations, the gallery application is not associated with the guide parameter setting interface, and the user does not enter the guide parameter setting interface after clicking the filter control 90c-1. After the user clicks the filter control 90c-1, the image capture method provided in the embodiment of the present application captures the wonderful moments of the video according to the preset scene, thereby obtaining images of the wonderful moments.

[0168] In order to better understand the process of capturing wonderful moments by the image capturing method provided by the present application, the following embodiments still take the mobile phone 100 as an example. Figures 12 to 20 The image processing flow during image capture is described.

[0169] For example, Fig.12 As shown, the image capture method comprises the following steps:

[0170] S101. Based on the image capture instruction triggered by the user, a preview video stream and a scene description text are obtained.

[0171] refer to Figures 6 to 10 In the user interface shown, after the mobile phone detects that the user enters the camera photo page and clicks on any of the options in the control 40b-1 and the control 50a-1 in sequence to complete the scene selection, it is determined that the user has triggered the image capture instruction. At this time, the mobile phone obtains the scene description text corresponding to the scene selected by the user and obtains the preview video stream through the camera.

[0172] The scene description text can be Figure 7 The scene description corresponding to the subdivided scene selected by the control 50c-1 in (3); or the scene description corresponding to the subdivided scene selected by the user by clicking Figure 8 The guiding text input after the control 60a-1 in (1) can also be the image scene description text preset in the mobile phone, which is not limited in the embodiment of the present application.

[0173] Exemplarily, the scene description text may be “surfing scene, stretching posture, the moment of rolling up waves”, “jumping / body tilting to wipe off snowflakes”, “shooting scene”, and the like.

[0174] S102. Preprocess the preview video stream to obtain an image frame set.

[0175] The camera of the mobile phone can generate RAW images in real time according to the light in the viewing range, and send the RAW images to the upper-layer application that calls the camera, such as the camera application. Taking the frame rate of 30fps as an example, the camera can report a frame of RAW image to the upper-layer application every 33.3 milliseconds (ms), that is, the upper-layer application can receive a frame of RAW image every 33.3ms.

[0176] Therefore, in order to reduce the computational cost and improve the processing efficiency, the preview video stream can also be preprocessed to reduce the number of image frames involved in subsequent computations. The preprocessing method includes decoding, frame extraction and other operations.

[0177] S103. Perform scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set.

[0178] Among them, the mobile phone can use a multi-label classification model to determine the scene labels corresponding to the images in the image frame set.

[0179] Exemplarily, the scene tag may include but is not limited to a combination of one or more parameters such as the subject in the image, the subject's action, the subject's emotion, the subject's expression, the shooting scene, etc.

[0180] Exemplary scene labels include, but are not limited to, looking back, fireworks, lightning, parent-child, running, jumping, skateboarding, skiing, badminton, basketball, splashing water, flying hair, flapping wings, graduation celebration, romantic warmth, affectionate gaze, passion on the field, cat jumping, dog jumping, etc.

[0181] In some possible implementations, see Fig.13 , input the scene description text and the image frame set into the text encoder (text_encoder) and the image encoder (image_encoder) respectively, and obtain the aligned text vector (text_embedding) and frame vector (frame_embedding). Then calculate the similarity score (scene_similartity_scores) between each frame image and the text in the text vector and frame vector, and determine the scene label corresponding to the image frame set.

[0182] The multi-label classification model can be a pre-trained model. The training process can be found in Fig.14 . Its training data can be image-text data pairs captured from the Internet, manually annotated, and image-text data pairs generated by a large language model. Exemplarily, the image-text data pairs captured from the Internet can be obtained by searching various image and film sharing websites on the Internet, selecting a suitable image-text data source for capture, and capturing the image and text and paired text, and then performing data preprocessing through data cleaning, deduplication, and filtering. For single-modal data such as collected pictures, manual annotation can be performed, such as manually writing description text for the picture or using other text generation tools to assist in generating description text. For existing text data, matching can be performed through keyword search, and then the pictures for claims can be manually selected. The collected data is input into the model for training to obtain a multi-label classification model.

[0183] Specifically, the training process of the multi-label classification model can be trained in the cloud and then sent to the mobile phone, or it can be trained on the mobile phone, which is not limited in the embodiments of the present application.

[0184] For example, the similarity between each frame image and the scene description text in the text vector and the frame vector is calculated by the cosine similarity algorithm, and then the similarity is sorted by the top selector (top_K_selector) to obtain the top-ranked scene labels. The closer the cosine similarity value is to 1, the higher the similarity between the image and the scene description text; the closer the cosine similarity value is to 0, the lower the similarity between the image and the scene description text.

[0185] In some possible implementations, since the confidence of the top-ranked scene labels cannot be determined at this time, it is necessary to input the obtained scene labels and image frames into the image-text matching model (ITModel validation) for binary classification to further determine the matching degree between the scene labels and the image frames, so as to obtain the final scene labels of the image frame set.

[0186] Exemplarily, when determining the degree of match between a scene label and an image frame, it may be determined based on a maximum confidence value, thereby outputting a unique label; it may also be determined based on a confidence range according to business requirements, thereby outputting multiple labels, which is not limited in the embodiments of the present application.

[0187] In some other possible implementations, the electronic device uses a multi-label classification model to average the features of all images in the image frame set to obtain average feature information of the image frame set, where the average feature information includes the features of all images in the image frame set. The mobile phone can use an image scene classification algorithm to determine a scene label corresponding to the average feature information, and use the scene label as the scene label of the image frame set.

[0188] Furthermore, it is not limited to identifying the scene label of an image frame set based on average feature information. For example, the mobile phone can also identify the scene label of the image frame set based on the first image of the image frame set; or, the mobile phone can identify and determine the scene label of an image randomly selected from the image frame set, and use the scene label of the random image as the scene label of the image frame set. The embodiments of the present application are not limited to this.

[0189] S104. Determine guidance parameters based on the scene label, where the guidance parameters include a guidance picture and a guidance text.

[0190] The guidance parameters include but are not limited to one or more of the guidance pictures and guidance texts. The guidance parameters can be used to obtain the image most similar to the guidance parameters from the image frame set, that is, to capture wonderful moments from the image frame set.

[0191] Furthermore, the guidance parameters may be preset in the storage or may be input by the user, which is not limited in the embodiments of the present application.

[0192] For example, see Figure 8In the user interface of the present invention, the guiding text may be one or more of a template text, a text inputted by a user, a text inputted from a local mobile phone, and a text obtained by a network search. The template text may include one or more text examples provided by the mobile phone. The text inputted by a user may be a text typed or voice inputted by a user. The text inputted from a local mobile phone may be a text stored in the mobile phone (e.g., a text of a memo application, a text of a file application, etc.). The text obtained by a network search may be a text retrieved by the mobile phone based on a search term provided by the user, and so on.

[0193] For example, the guide text may be a descriptive sentence in the style of Tang poetry, Song lyrics, Yuan opera, modern poetry, and prose. Based on the guide text, the mobile phone may filter images that meet the description of the guide text from the image collection. For example, when the guide text includes "turn head and smile", the mobile phone may filter images including the subject turning head and smiling from the image collection.

[0194] For example, see Fig. 9 The user interface of the present invention may be a template image, an image inputted by a user, an image inputted locally on the mobile phone, or one or more of images obtained by searching the Internet. The template image may include one or more image examples provided by the mobile phone. The image inputted locally on the mobile phone may be an image stored by the mobile phone (e.g., an image of a gallery application, a video of a video application, etc.). The image inputted by the user may be an image drawn by the user and received by the mobile phone through a touch pad or an input device (e.g., a stylus, a digitizer, etc.). The images obtained by searching the Internet may include images retrieved by the mobile phone based on search terms provided by the user. The mobile phone may filter out images similar to the guide image from an image collection based on the guide image. For example, when the guide image includes “the subject who turns his head and smiles”, the mobile phone may filter out images including “the subject who turns his head and smiles” from the image collection.

[0195] Specifically, the most similar guidance pictures and guidance texts are screened out from the guidance parameters according to the above-mentioned scene labels. The screening method can be referred to step S103.

[0196] In some possible implementations, scene tags may be set for the guide text and the guide image in the guide parameters first, so as to match the scene tags determined by the image frame with the scene tags in the guide parameters, thereby determining the guide parameters.

[0197] In some other possible implementations, there may be multiple guided parameters screened out. After the guided parameters are obtained through the encoder to obtain the guided vector, the mobile phone can also perform pooling operations on the guided vector, such as maxpooling, average pooling, k-max pooling, chunk-max pooling, etc., to sample multiple guided vectors and obtain the reduced-dimensional guided vector. The mobile phone can determine a set of candidate images based on the reduced-dimensional guided vector. In this way, the mobile phone can reduce the calculation parameters and amount of calculation while retaining the main features of the image to prevent overfitting.

[0198] S105. Match the guide image and the guide text with each image frame in the image frame set respectively, and obtain an image-text matching score and an image-image matching score for each image frame.

[0199] Specifically, the guide picture and the image frame set are input into the pre-trained image-image matching model (IIMoel) to calculate the image-image matching score of each frame of image; the guide text and the image frame set are input into the pre-trained image-text matching model (ITMoel) to calculate the image-text matching score of each frame of image.

[0200] Among them, the image-image matching score is used to indicate the similarity between the guide image and the images in the image frame set. The higher the image-image matching score, the higher the similarity between the guide image and the images in the image frame set. The image-text matching score is used to indicate the similarity between the guide text and the images in the image frame set. The higher the image-text matching score, the higher the similarity between the guide text and the images in the image frame set.

[0201] S106. Determine a candidate set of wonderful images from the image frame set according to the image-text matching score and the image-image matching score of each frame of image.

[0202] In some possible implementations, the image-text matching score and the image-image matching score can be screened by a threshold selection method, and when both the image-text matching score and the image-image matching score are greater than a set threshold, the corresponding image is determined as a wonderful image and put into a wonderful image set. The above-mentioned set threshold can be set according to actual needs.

[0203] For other possible implementations, see Fig.14According to the image-text matching score and image-image matching score of each frame in the image frame set, the image-text matching score curve and the image-image matching score curve are constructed in the image frame-similarity coordinate system to obtain the intersection of the matching score curve and the image-image matching score curve. Compared with the guide text, the guide image can better reflect the snapshot result expected by the user. Therefore, the image-image matching score is intercepted here through the above intersection, and the multiple image frames corresponding to the intercepted interval are used as the candidate wonderful image set.

[0204] S107. Perform feature fusion on the guide image and the guide text to obtain target features.

[0205] For example, see Fig.15 , extract features from the guide image, then fuse the extracted features with the text features of the guide text, and finally obtain the target features.

[0206] S108. Calculate the similarity between each frame of the image in the candidate wonderful image set and the target feature.

[0207] Among them, the target features and each frame image in the candidate wonderful image set are input into IIMoel, and the similarity of each frame image is calculated.

[0208] S109. Screening out a wonderful moment image from the candidate wonderful image set according to the similarity, wherein the wonderful moment image is an image frame with the highest similarity in the candidate wonderful image set.

[0209] Specifically, the similarity of each frame of image is sorted, and the image with the highest similarity is output as the wonderful moment image.

[0210] In this way, the mobile phone can process the scene description text input by the user and the preview video stream of the target scene according to the preset or input guidance parameters, and capture the wonderful moment images from the preview video stream, so as to obtain the user-defined scene and the desired wonderful images, thereby improving the user experience.

[0211] In some other possible implementations, since the wonderful moment image in the current preview video stream may not be the best wonderful moment image, multiple wonderful moment image calculations can be performed in the embodiments of the present application, and the wonderful moment image in the current calculation cycle is compared with the wonderful moment image obtained by subsequent calculations, so as to screen out the better quality wonderful moment image as the final wonderful moment.

[0212] The image capture method provided in the embodiment of the present application also includes:

[0213] Step S110: continuously acquiring a plurality of the wonderful moment images, comparing the similarity of each wonderful moment image, and outputting the wonderful moment image with the highest similarity as the final wonderful moment image.

[0214] For example, see Fig.16 , the mobile phone continuously obtains the video stream of the target scene through the camera, and performs three consecutive calculations on the video frames in the video stream, and the calculation cycles are respectively represented as calculation cycle t1, calculation cycle t2 and calculation cycle t3. Among them, the number of image frames involved in the calculation in each calculation cycle is preset; the image frames in each calculation cycle may be repeated with the adjacent calculation cycle, or may not be repeated.

[0215] The wonderful moment image obtained in the calculation cycle t1 is ks1, the wonderful moment image obtained in the calculation cycle t2 is ks2, and the wonderful moment image obtained in the calculation cycle t3 is ks3. The similarities of ks1, ks2 and ks3 in step S108 are compared, and the wonderful moment image ks1 with the highest similarity is output as the final wonderful moment.

[0216] It should be noted that when the current calculation cycle is t1, the search space for optimization can be expanded on the time axis by comparing the similarity with the wonderful moment images obtained at t2 and t3. In order to further improve the accuracy of the capture, when the similarity of ks1 is lower than that of ks2, t2 can be set as the current calculation cycle, and the optimization can be continued on the time axis, so that the wonderful moment images within all calculation cycles can be included in the candidate results, and the wonderful moment images with the highest similarity score can be selected from them. The number of calculation cycles to be expanded can be set according to the performance of the mobile phone.

[0217] In some other possible implementations, aesthetic evaluation or fine-grained image evaluation may be performed on the wonderful moment images in the candidate results, so as to obtain aesthetic scores or attribute scores of various granularities of these wonderful moment images.

[0218] The image capture method provided in the embodiment of the present application also includes:

[0219] Step S111. Score the multiple wonderful moment images based on the trained aesthetic evaluation model, and output the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

[0220] After obtaining the aesthetic score of each wonderful moment image, the aesthetic evaluation threshold (aesthetic evaluation threshold T A =0.5), directly filter out candidate results with values ​​less than T A The remaining wonderful moment images in the candidate results are sorted by aesthetic scores, and the one with the highest score is set as the final wonderful moment image.

[0221] In some other possible implementations, in order to reduce the result output delay, the candidate highlight set in step S106 may be filtered, thereby reducing the image frames involved in subsequent calculations.

[0222] The image capture method provided in the embodiment of the present application also includes:

[0223] Step S106 ′: Filter the candidate wonderful image set according to a preset peak prominence threshold.

[0224] For example, see Fig.17 , when a set of candidate wonderful images is obtained, by setting a peak prominence threshold, for example, a prominence threshold T prominence =0.3, directly filter the image frames with lower matching degree, and sort the video frame images with a prominence greater than the prominence threshold according to the image matching score, and finally output the image frame with the highest score as the highlight moment image.

[0225] In some other possible implementations, the image capture method provided in the embodiment of the present application further includes:

[0226] Step S106 ”: Filter the candidate wonderful image set according to the preset peak prominence threshold and peak width threshold.

[0227] In addition, the prominence of the peak is a measure of the prominence of the peak from the baseline around the curve, and the prominence of the peak is also defined as the vertical distance between the peak and its lowest contour. Therefore, the threshold of the peak width can also be increased, such as the width Width ≥ 4, so as to further filter the image frames that meet the prominence threshold and reduce the image frames involved in subsequent calculations.

[0228] For example, Fig.18 As shown, the mobile phone includes multiple modules, and the mobile phone implements the image capture method provided in the embodiment of the present application through the above multiple modules. Among them, the above multiple modules include but are not limited to one or more guidance modules, one or more encoders, scene detection modules, one or more scene matching modules, one or more matching models, key segment selection modules and key image selection modules.

[0229] One or more guidance modules may be used to determine guidance parameters and send the guidance parameters to the encoder. For example, one or more guidance modules may include but are not limited to a text guidance (guided_texts) module, an image guidance (guided_images) module, etc. The text guidance module may be used to determine the guidance text. The image guidance module may be used to determine the guidance image. The mobile phone may obtain the guidance text and the guidance image through a web search (web_search) and input them into the corresponding guidance module. For example, Fig.18 The arrow from the text guidance module to the image guidance module may indicate that the text guidance module receives keywords input by the user, searches for images based on the keywords, and sends the searched images to the image guidance module.

[0230] Optionally, the text guidance module can receive text input by the user (for example, typed through an electronic device or input by voice), and transmit the input text to the image guidance module. The image guidance module uses the input text as a keyword to search the Internet, searches for images that match the text description, and uses the image selected by the user as the guidance image.

[0231] Among them, one or more encoders can be used to convert the guidance parameters into a guidance vector. For example, one or more encoders may include but are not limited to a text encoder (text_encoder) and an image encoder (image_encoder). The text encoder can be used to process the guidance text to obtain a guided text vector (guided_text_embedding) corresponding to the guidance text, and the guided text vector can be used to represent the characteristics of the guidance text. The image encoder can obtain a vector corresponding to the image set based on an image set. For example, a mobile phone can obtain an image set by decoding, extracting frames, and the like on a video file or a video stream obtained by a camera, and the image set includes one or more image subsets. The image encoder can obtain a frame vector (frame_embeddings) corresponding to the image subset based on the image subset, and the frame vector can be used to represent the characteristics of the image of the image subset. The image encoder can also obtain a guided image vector (guided_image_embedding) based on the guidance image, and the guided image vector can be used to represent the characteristics of the guidance image.

[0232] The scene detection module can be used to calculate the similarity between the frame vector and each scene based on the frame vector of the input video, and determine the scene label corresponding to the frame vector, that is, determine the scene label to which the image subset belongs, such as splashing water, flying hair, flapping wings, etc. The scene detection module can send the scene label of the image frame set to one or more matching modules.

[0233] Among them, one or more matching modules can be used to screen guidance parameters similar to scene labels. One or more matching modules may include but are not limited to text strategy (text_strategy) matching module and image strategy (image_strategy) matching module. Among them, the text matching module is used to determine one or more guidance text vectors similar to the scene label; the image strategy matching module is used to determine one or more guidance image vectors similar to the scene label. It can be understood that when the mobile phone matches the scene label corresponding to the image frame set with the guidance parameter through the matching module, the mobile phone can calculate the similarity between the matched guidance parameter and the image frame set through one or more matching models.

[0234] Wherein, one or more matching models may include but are not limited to an image-text matching model (image-text model, ITMoel) and an image-image matching model (image-image model, IIMoel). Wherein, when the text matching strategy module sends the guide text vector to the image-text matching model, the scene detection module may send the frame vector corresponding to the image frame vector to the image-text matching model. When the image matching strategy module sends the guide image vector to the image-image matching model, the scene detection module may send the frame vector corresponding to the image frame set to the image-image matching model.

[0235] Among them, ITMoel can be used to process the frame vector provided by the scene detection module with the guide text vector to obtain the image-text matching score. The image-text matching score is used to represent the similarity between the frame vector and the guide text vector. The higher the image-text matching score, the higher the similarity between the frame vector and the guide text vector. IIModel is used to process the frame vector provided by the scene detection module with the image vector to obtain the image-image matching score. The image-image matching score is used to represent the similarity between the frame vector and the image vector. The higher the image-image matching score, the higher the similarity between the frame vector and the image vector.

[0236] The key clip selection (key_clip_selector) module can obtain a comprehensive score of the image frame set based on the image-text matching score and the image-image matching score. The key clip selection module can also determine a candidate wonderful image set based on the comprehensive score of the image frame set. The candidate wonderful image set is a subset of target images with the highest comprehensive score in the image frame set. For example, the key clip selection module can set different weight coefficients for the image-text matching score and the image-image matching score to calculate the comprehensive score. For example, the key clip selection module can set a weight coefficient of 0.75 for the image-text matching score and a weight coefficient of 0.25 for the image-image matching score. In this way, since users tend to obtain information through vision, setting a higher weight for the image-image matching score can provide users with result images that better meet user requirements. In some examples, the key clip selection module can also set a preset similarity, and the key clip selection module can filter out image frames with a comprehensive score lower than the preset similarity. Alternatively, the key clip selection module can write image frames with a comprehensive score higher than the preset similarity into the candidate wonderful image set.

[0237] The key image selection (key_frame_selector) module can obtain a wonderful moment image based on the candidate wonderful image set provided by the key segment selection module. Among them, the key image selection module can perform feature fusion of the guide picture and the guide text, and calculate the similarity between the fused target features and the image frames in the candidate wonderful image set, and output the image with the highest similarity as the wonderful moment image. The key image selection module can also obtain the output wonderful moment images multiple times in a row, and compare the similarity scores of multiple wonderful moment images to obtain the optimal wonderful moment image, thereby realizing the expansion and optimization on the timeline. The key image selection module can also perform aesthetic scoring on the wonderful moment images obtained multiple times, so as to output the wonderful moment image with the highest aesthetic score as the final wonderful moment image.

[0238] In this way, the electronic device can process the input video stream according to the guide text and guide image input by the user through the processing of various modules, and filter out the result images that meet the user's personalized and diverse needs from the video stream, further diversifying the number and types of captured scenes.

[0239] In some possible implementations, the aesthetic evaluation model can be sent to the mobile phone after training on the cloud server, or it can be trained locally on the mobile phone. Here, the training of the aesthetic evaluation model on the cloud server is taken as an example to introduce it.

[0240] For example, see Fig.19The cloud server can classify the collected images according to the image scenes. For example, the cloud server can classify the images into scene types such as looking back, lightning, parent-child, running, etc. In this way, the cloud server can obtain images of various scene types and ensure that the number of images of each scene in the acquired images is small, so that the aesthetic evaluation model can reasonably evaluate the images of each scene.

[0241] The cloud server can perform subject detection on the image, that is, determine the subject of the image, and after determining the subject of the image, control the number of images of different subjects so that the difference between the number of images of different subjects is small. Afterwards, the cloud server can receive the scores of the image annotations by the annotators through the measurement indicators of aesthetic evaluation, such as composition, color, light, exposure, and conception. The cloud server can obtain the aesthetic score of the image based on the scores of each measurement indicator. Among them, the subject of the image is different, and the annotators have different standards for scoring the image through the composition indicator. For example, taking the score of each measurement indicator as 10 points, when the annotator scores the composition of the image, the image can be evenly divided into nine grids. When the main body of the image is a person, the area where the person is located is located in the upper six grids of the nine grids, the annotator can mark the image with a score of 6 points or more on the composition factor. When the area where the person is located is located in the lower six grids of the nine grids, the annotator can mark the image with a score of 5 points or less on the composition factor. When the main body of the image is an animal, the area where the animal is located is located in the lower six grids of the nine grids, the annotator can mark the image with a score of 6 points or more on the composition factor. It should be noted that the measurement indicators and the corresponding score values ​​are only examples, and other methods can also be used to determine the score of the image.

[0242] The cloud server can collect the rating results of multiple maintainers for the same image, and use the average of the multiple rating results as the aesthetic score of the image.

[0243] Afterwards, the cloud server can classify the images. Specifically, the cloud server can classify the images according to the scores and determine the number of images of each score. The cloud server can keep the number of images in each score segment consistent. In this way, the cloud server can ensure that the number of images in each score segment is roughly the same, so as to avoid the problem of data category imbalance. Based on the images with similar numbers in each score segment, the cloud server obtains an aesthetic evaluation model that reasonably evaluates the images of each score.

[0244] The cloud server can train an aesthetic evaluation model based on the correspondence between images and their aesthetic scores. For example, the aesthetic evaluation model constructed by the cloud server is as follows: Fig. 20As shown, the aesthetic evaluation model includes an input image module, an encoder and a predicted distribution module. The input image module is used to receive a set of images selected by a user. The encoder includes a backbone network, a fully-connected layer and a normalized exponential function layer (softmax layer). The encoder is used to process the input image set to obtain an image vector for each image in the image set. The image vector is used to represent the scores of various measurement indicators of the image, which can be used to calculate the aesthetic score of the image. The predicted distribution module is used to process the image vector output by the encoder to obtain the aesthetic score of the image. Exemplarily, the predicted distribution module can determine the probability of the image being in each score based on the image vector. The predicted distribution module can add the product of the score and the probability, that is, calculate the mathematical expectation to obtain the aesthetic score of the image.

[0245] The mobile phone can download the aesthetic evaluation model from the cloud server. After the mobile phone obtains the aesthetic evaluation model, it can use the aesthetic evaluation model to score all images in the candidate results and obtain the aesthetic scores of all images in the candidate results. The mobile phone can select a result image from the candidate results based on the aesthetic score, and the result image is one or more images with the highest aesthetic score in the candidate results. Optionally, the final wonderful moment image is one or more images in the candidate results whose aesthetic score is greater than a preset aesthetic score. In this way, the aesthetic evaluation model can score each image in the candidate image set, and use the one or more images with the highest aesthetic score as the final wonderful moment image.

[0246] To better understand Figure 1 The software structure of the electronic device 100 is shown, and the software structure of the electronic device 100 is described below. Before describing the software structure of the electronic device 100, the architecture that can be adopted by the software system of the electronic device 100 is first described.

[0247] Specifically, in actual applications, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

[0248] In addition, it is understandable that the software systems currently used by mainstream electronic devices include but are not limited to Windows systems, Android systems, and iOS systems. For ease of explanation, the present application embodiment takes the layered Android system as an example to exemplify the software structure of the electronic device 100.

[0249] In addition, the image capture solution provided in the subsequent embodiments of the present application is also applicable to other systems in specific implementations.

[0250] See also Fig.21 , which is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0251] like Fig.21 As shown, the layered architecture of the electronic device 100 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some implementations, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and system library, the hardware abstraction (HAL) layer, and the kernel layer.

[0252] The application layer may include a series of application packages. Fig.21 As shown, the application package may include camera, settings, music, Bluetooth, gallery and other applications, which are not listed here one by one and are not limited in this application.

[0253] Specifically in the embodiment of the present application, the opening entrance of the image capture solution can be set in the camera application, for example Figure 6 Control 40b-1 in (2).

[0254] Among them, the application framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. In some implementations, these programming interfaces and programming frameworks can be described as functions. Exemplarily, the application framework layer may include a camera access interface. Among them, the camera access interface may include camera management and camera devices. The camera access interface is used to provide an application programming interface and a programming framework for camera applications.

[0255] The system library and runtime layer include the system library and Android Runtime. Android Runtime includes the core library and the virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0256] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0257] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0258] The system library may include multiple functional modules, such as image capture processing engine, media library, three-dimensional (3D) graphics processing library (such as OpenGL ES), two-dimensional (2D) graphics engine (such as SGL), shape manager, etc.

[0259] Specifically in the embodiment of the present application, the image capture processing engine is pre-installed with the pre-trained models in the embodiment of the present application, including the image-text matching model, the image-image matching model, the aesthetic evaluation model and the multi-label classification model. The camera application passes the preview video stream into the image capture processing engine, so that the image capture processing engine calls the corresponding processing model to process the video stream, thereby outputting the wonderful moment image. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.

[0260] It can be understood that the 2D graphics engine mentioned above is a drawing engine for 2D drawing.

[0261] The HAL layer is an interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: camera hardware abstraction layer, display driver hardware abstraction layer, and power hardware abstraction layer.

[0262] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, power management driver, and sensor driver. Among them, the display driver can include a secondary screen display driver, a main screen display driver, etc. The sensor driver can include drivers corresponding to gyroscope sensors, acceleration sensors, Hall sensors, etc., which are used to detect the angle change between the main screen and the secondary screen of the external folding screen mobile phone. The camera driver is used to drive the corresponding camera to work. The capacitor management driver is used to drive the power management module to power the display screen.

[0263] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation to this embodiment.

[0264] In addition, it can be understood that the division of the above functional modules is only an example listed for better understanding the technical solution of this embodiment, and is not the only limitation to this embodiment. In practical applications, the above functions can also be integrated into one functional module, and this embodiment does not limit this.

[0265] In addition, in actual applications, the above-mentioned functional modules can also be expressed as services or frameworks. For example, the speech recognition module can be expressed as a speech recognition service or a speech recognition framework, etc. This embodiment does not impose any limitation on this.

[0266] The software structure of the electronic device 100 is introduced here. It can be understood that: Fig.21 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0267] In addition, it is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0268] In addition, it should be noted that the image capture method provided by the above embodiments implemented by the electronic device in the actual application scenario can also be executed by a chip system included in the electronic device, wherein the chip system may include a processor. The chip system can be coupled to the memory so that when the chip system is running, the computer program stored in the memory is called to implement the steps executed by the above electronic device. The processor in the chip system can be an application processor or a processor other than an application processor.

[0269] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image capture method in the above-mentioned embodiment.

[0270] In addition, an embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the image capture method in the above-mentioned embodiment.

[0271] In addition, an embodiment of the present application also provides a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the image capture method in the above-mentioned embodiment, so as to control the receiving pin to receive the signal, so as to control the sending pin to send the signal.

[0272] In addition, it can be seen from the above description that the electronic device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0273] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image capture method, characterized in that: Applied to electronic equipment, the method comprises: In response to the image capture instruction, obtaining a preview video stream and a scene description text; Preprocessing the preview video stream to obtain an image frame set; Performing scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set; Determining a guidance parameter from a preset guidance parameter set based on the scene tag, wherein the guidance parameter includes a guidance picture and a guidance text; Matching the guide image and the guide text with each frame of the image in the image frame set respectively, to obtain an image-text matching score and an image-image matching score for each frame of the image; Determining a candidate wonderful image set from the image frame set according to the image-text matching score and the image-image matching score of each frame of image; Performing feature fusion on the guide image and the guide text to obtain target features; Calculating the similarity between each frame of the image in the candidate wonderful image set and the target feature; A wonderful moment image is screened out from the candidate wonderful image set according to the similarity, and the wonderful moment image is an image frame with the highest similarity in the candidate wonderful image set.

2. The method according to claim 1, characterized in that The step of selecting the wonderful moment image from the candidate wonderful image set according to the similarity also includes: A plurality of the wonderful moment images are continuously acquired, the similarity of each wonderful moment image is compared, and the wonderful moment image with the highest similarity is output as the final wonderful moment image.

3. The method according to claim 2, characterized in that The comparing the similarity of each wonderful moment image and outputting the wonderful moment image with the highest similarity as the final wonderful moment image also includes: The multiple wonderful moment images are scored based on the trained aesthetic evaluation model, and the wonderful moment image with the highest aesthetic score is output as the final wonderful moment image.

4. The method according to claim 1, characterized in that: The step of determining a candidate wonderful image set from the image frame set according to the image-text matching score and the image-image matching score of each frame of image comprises: A plurality of image frames whose image-text matching scores and image-image matching scores are both greater than a preset threshold are determined as the candidate wonderful image set.

5. The method according to claim 4, characterized in that The step of determining a candidate wonderful image set from the image frame set according to the image-text matching score and the image-image matching score of each frame of image further includes: Based on the image-text matching score and the image-image matching score, constructing an image-text matching score curve and an image-image matching score curve in the same coordinate system, wherein the coordinate system is an image frame-similarity coordinate system; Obtaining an intersection point of the image-text matching score curve and the image-image matching score curve; The image score curve is intercepted based on the intersection, and a plurality of image frames corresponding to the intercepted image score line segments are determined as a candidate wonderful image set.

6. The method according to claim 5, characterized in that After intercepting the image score curve based on the intersection, and determining a plurality of image frames corresponding to the intercepted image score line segments as a candidate wonderful image set, the method includes: According to a preset peak prominence threshold, the image score line segments are filtered, and a plurality of image frames corresponding to the filtered image score line segments are determined as the candidate wonderful image set.

7. The method according to claim 6, characterized in that According to a preset peak prominence threshold, filtering the image score line segment also includes: The image score line segments are filtered according to a preset peak prominence threshold and a peak width threshold.

8. The method according to claim 1, characterized in that: The performing scene classification on the image frame set based on the scene description text to obtain a scene label corresponding to the image frame set includes: The scene description text and the image frame set are input into a pre-trained multi-label classification model for label classification to obtain one or more scene labels corresponding to the image frame set.

9. The method according to claim 1, characterized in that: Before obtaining the preview video stream and the scene description text based on the image capture instruction triggered by the user, the method further includes: Displaying a first interface, wherein the first interface includes a camera application icon; Detecting a first click operation on the camera application icon; In response to the first click operation, displaying a second interface, wherein the second interface includes a snapshot start icon; Detecting a second click operation on the snapshot start icon; In response to the second click operation, displaying a scene selection bar in the second interface; detecting that a third click operation on the scene selection bar is completed; In response to the third click operation, an image capture instruction is sent, and a preview image is displayed in the second interface.

10. The method according to claim 9, characterized in that In response to the second click operation, displaying a scene selection bar in the second interface, further comprising: In response to the second click operation, finding the display scene definition option in the second interface; detecting a fourth click operation on the scene definition option; In response to the fourth click operation, a third interface is displayed, wherein the third interface includes a first input control and a second input control, wherein the first input control is used to add the guide text, and the second input control is used to add the guide picture.

11. An electronic device, characterized in that: The electronic device comprises: a memory and a processor, wherein the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image capture method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: It comprises a computer program, and when the computer program is run on an electronic device, the electronic device executes the image capture method according to any one of claims 1 to 10.

Images