Image processing method, electronic device, computer program product and storage medium

Abstract

The application provides an image processing method, an electronic device, a computer program product and a storage medium, and relates to the technical field of terminals. The electronic device displays a shooting preview interface, and the shooting preview interface displays preview images frame by frame. During the shooting preview process, the electronic device analyzes whether the preview images match multi-modal scene information of a target scene, finds out a target image matching the multi-modal scene information of the target scene, and obtains a snapshot image of the target scene matched by the target image. When the electronic device shoots the target scene, the multi-modal scene information of the target scene used includes scene reference information of an image mode and a scene prompt sentence of a language mode. The electronic device can automatically shoot a snapshot image corresponding to the target scene without relying on the user's pre-judgment of the wonderful scene and the shooting operation of clicking the shutter. Based on the multi-modal scene information, the automatic shooting is performed, and the scene matching degree of the snapshot image is higher.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device, computer program product, and storage medium. Background Technology

[0002] Smartphones are now used in almost every aspect of people's daily lives, including chatting, entertainment, dining, and travel. With the continuous advancement of smartphone imaging capabilities and the support of various image algorithms, people can easily capture wonderful photos of everyday life, recording precious moments and sharing them widely with friends and family.

[0003] After a mobile phone activates its camera mode, it captures fleeting moments perceived by the user, such as a pet running, people jumping, a badminton smash, or a heartwarming moment between parents and children. The phone receives the user's shutter click and captures the image of that moment. Because these fleeting, random moments require the user to perceive them promptly and click the shutter; otherwise, the device will struggle to effectively capture the highlights of the scene.

[0004] It is evident that existing electronic devices suffer from the technical challenge of relying on user interaction to capture compelling images, which presents significant difficulties. Summary of the Invention

[0005] This application provides an image processing method, an electronic device, a computer program product, and a storage medium to solve the technical problem that obtaining excellent images in electronic devices depends on user operation and is difficult.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, an image processing method is provided, applied to an electronic device. The electronic device displays a shooting preview interface, which displays a preview stream including preview images displayed frame by frame. During the shooting preview process, the electronic device analyzes whether the preview images match the multimodal scene information of the target scene, finds the target image that matches the multimodal scene information of the target scene, and obtains a captured image matching the target scene from the target image.

[0008] When electronic devices capture images of a target scene, they use multimodal scene information of the target scene, which mainly includes scene reference information in the image mode and scene prompts in the language mode. The scene prompts are used to describe the scene attributes of the target scene.

[0009] Electronic devices can automatically capture images of target scenes without relying on users' predictions of exciting scenes or clicking the shutter button. Based on multimodal scene information, automatic capture results in images with higher scene matching accuracy.

[0010] In one possible implementation of the first aspect, the electronic device obtains a preview image matching the target scene from the preview image based on the multimodal scene information of the target scene, which is achieved by analyzing whether the key features of the two frames match.

[0011] Specifically, the electronic device acquires the first key feature of the target scene based on multimodal scene information, and the second key feature of the preview image. The order in which the electronic device acquires the first key feature of the target scene and the second key feature of the preview image is not limited.

[0012] The electronic device analyzes whether the first key feature of the target scene matches the second key feature of the preview image. If the first key feature of the target scene matches the second key feature of the preview image, the preview image is determined to be the target image.

[0013] The electronic device extracts the first key feature from the scene reference image of the target scene and the second key feature from the preview image, respectively. The key features are associated with scene attributes. By analyzing whether the images match through key features, interference from other features can be reduced, improving the accuracy of image matching and thus improving the scene matching degree of the captured images.

[0014] In one possible implementation of the first aspect, the step of the electronic device obtaining the first key feature of the target scene based on multimodal scene information is further specified. The electronic device obtains a first feature vector of a scene reference image based on scene prompts, and then extracts the first key feature of the target scene from the first feature vector.

[0015] The electronic device acquires a first feature vector, which records the mapping relationship between feature values ​​and feature locations. Feature values ​​at some locations have a strong correlation with scene attributes, while those at others have a relatively weak correlation. The electronic device extracts the first key features with the strongest correlation to scene attributes from the first vector.

[0016] In one possible implementation of the first aspect, the step of the electronic device obtaining the first feature vector of the scene reference image based on scene prompts is further specified. The target scene captured by the electronic device uses multimodal scene information, involving the scene reference image of the image modality and the scene prompts of the language modality. The electronic device uses a multimodal large model to extract the first feature vector of the scene reference image.

[0017] Specifically, the electronic device inputs the scene reference image and scene prompts into the multimodal large model to obtain the first feature vector. The multimodal large model is a deep learning-based machine learning technique that combines multimodal information such as text, images, video, and audio, and learns the relationships between different modalities to achieve richer and more accurate multimodal computation. A multimodal large model typically includes a Large Language Mode 1 (LLM) module, an image module, etc., with different modal modules processing data from different modalities. During the extraction of the first feature vector from the scene reference image, the multimodal large model references the LLM module to analyze the features of the scene reference language, ensuring that the extracted first feature vector emphasizes the features indicated by the scene reference statements, thus reducing unnecessary feature interference.

[0018] In one possible implementation of the first aspect, the step of the electronic device acquiring the second key feature of the preview image is further defined. The electronic device first acquires the second feature vector of the preview image, and then acquires the second key feature from the second feature vector.

[0019] The electronic device acquires a second feature vector, which records the mapping relationship between feature values ​​and feature locations. Feature values ​​at some locations have a strong correlation with scene attributes, while those at others have a relatively weak correlation. The electronic device extracts the second key features with the strongest correlation to scene attributes from the second vector, and can then combine this with scene prompts from the target scene to emphasize certain key features.

[0020] In one possible implementation of the first aspect, the step of the electronic device obtaining the second feature vector of the previewed image based on scene prompts is further specified. The target scene for which the electronic device captures the image uses multimodal scene information, involving a scene reference image of the image modality and scene prompts of the language modality. The electronic device uses a multimodal large model to extract the second feature vector of the scene reference image.

[0021] Specifically, the electronic device inputs the preview image and the scene prompts of the matched target scene into the multimodal large model to obtain the second feature vector. During the extraction of the second feature vector from the preview image, the multimodal large model refers to the large language module to analyze the features of the scene reference language, so that the extracted second feature vector focuses more on the features indicated by the scene reference statement, thereby reducing unnecessary feature interference.

[0022] In one possible implementation of the first aspect, the steps for the electronic device to obtain the first key feature from the first feature vector are further specified. The electronic device first uses a key feature location search algorithm to find the first feature location of the scene attribute associated with the target scene from all feature locations in the first feature vector. Then, it extracts the first key feature corresponding to the first feature location from the first feature vector. The key feature location search algorithm includes a genetic algorithm. Genetic algorithms have high accuracy in searching feature locations. Of course, the electronic device can also use other key feature location search algorithms to search for the first feature location, such as reinforcement learning-based search algorithms, exhaustive search, random search, fast search, greedy search strategies, etc., without limitation.

[0023] In one possible implementation of the first aspect, the step of the electronic device acquiring the second key feature from the second feature vector is further specified. The electronic device may also utilize a key feature location lookup algorithm, such as a legacy algorithm, to find the second feature location of the scene attribute associated with the target scene from all feature locations in the second feature vector. Then, the electronic device extracts the second key feature corresponding to the second feature location from the second feature vector.

[0024] In one possible implementation of the first aspect, an alternative scheme is provided for the electronic device to obtain the second key feature from the second feature vector. When performing image processing, the electronic device considers that the extracted key features for two frames of scene matching are typically in the same or similar positions. To determine whether the preview image matches the target scene, the electronic device can directly reference the first feature position of the scene reference image of the target scene as the second feature position of the preview image.

[0025] The electronic device directly extracts the second key feature corresponding to the first feature position from the second feature vector. This saves the operation of finding the second feature position in the preview image and improves the correlation between the extracted second key feature and the target scene, further improving the accuracy of scene matching.

[0026] In one possible implementation of the first aspect, the step of the electronic device analyzing whether a first key feature of the target scene matches a second key feature of the preview image is further defined. The electronic device calculates the feature similarity between the first and second key features, such as cosine similarity and Euclidean distance. If the electronic device analyzes that the feature similarity satisfies a preset matching condition, it determines that the first key feature of the target scene matches the second key feature of the preview image. The preset matching condition may include a feature similarity exceeding a preset similarity.

[0027] In one possible implementation of the first aspect, a scheme is provided where the target scene to be captured is determined based on user selection. The electronic device, in response to the first operation, displays a capture scene tab. While displaying a shooting preview interface, the electronic device can also display a capture control on the shooting preview interface. If the electronic device does not receive a click operation on the capture control, it can execute a regular shooting process, such as a process of taking a picture based on a shutter click. If the electronic device receives a click operation on the capture control, it enters an automatic capture process and displays the capture scene tab.

[0028] The scene capture tab displayed on the electronic device includes at least one scene option control for a first scene. Each scene option control can correspond to and indicate the first scene to be captured, such as a jumping scene, a sunrise scene, etc.

[0029] The electronic device displays multiple scene option controls for a scene capture tab. If it receives a second operation, such as a click, applied to one of the scene option controls, it identifies the first scene corresponding to that scene option control as the target scene to be captured. The electronic device then acquires the multimodal scene information of the target scene and begins automatic scene capture for that scene.

[0030] In one possible implementation of the first aspect, the steps for the electronic device to acquire multimodal scene information of the target scene are further defined. For example, the electronic device receives images and statements input by the user as multimodal scene information.

[0031] Specifically, the electronic device displays a multimodal scene information input interface for the target scene. The input interface includes an image-based input box and a speech-based input box. The electronic device can receive images input from the image-based input box and statements input from the speech-based input box. The electronic device uses the image input from the image-based input box as a scene reference image and the statements input from the speech-based input box as scene prompts.

[0032] Electronic devices provide solutions for users to input multimodal scene information of the target scene, so that the images obtained by performing capture operations based on multimodal scene information can better meet user needs.

[0033] In one possible implementation of the first aspect, an alternative scheme is provided for the electronic device to obtain multimodal scene information of the target scene, namely, the electronic device requests the multimodal scene information from a server. The server mentioned here can be a cloud server or a physical server, without limitation.

[0034] The electronic device sends a query request to the server; the query request is used to request multimodal scene information of the target scene. Upon receiving the multimodal scene information of the target scene sent by the server, the electronic device can then perform the corresponding snapshot operation based on the obtained multimodal scene information.

[0035] In one possible implementation of the first aspect, an alternative scheme is provided for the electronic device to acquire multimodal scene information of a target scene. For example, the electronic device pre-configures multiple first scenes and stores the multimodal scene information of each first scene in local memory. The electronic device can also directly acquire the multimodal scene information of the target scene from local memory, based on the target scene determined by the user's operation as the first scene. Acquiring multimodal scene information directly from local memory is faster and further improves the efficiency of automatic image capture.

[0036] Secondly, an image processing method is provided for application in electronic devices. This method can extract images matching a target scene from multiple frames of a video.

[0037] Specifically, the electronic device acquires the first video to be processed. This video can be captured by the device's camera or acquired from other devices via a network or other communication methods. The first video consists of multiple consecutive frames. The electronic device then acquires multimodal scene information of the target scene to be matched. This information can be obtained from local memory or from a server.

[0038] The electronic device analyzes each frame of the first image to determine if it matches any target scene. This analysis primarily focuses on whether the first image matches the multimodal scene information of the target scene. The electronic device then analyzes and finds target images that match the multimodal scene information and generates a capture image based on these target images. For example, the electronic device can directly use the target image as the capture image, or it can capture a new frame immediately after the target image.

[0039] The electronic device can analyze each frame of the first video frame by frame to determine whether it matches the target scene. Alternatively, the electronic device can analyze a portion of the first video frame by frame skipping to determine whether that portion matches the target scene.

[0040] The electronic device can analyze the first video and extract target images that match the target scene. This saves users the trouble of manually searching for and taking screenshots of images that match the target scene, simplifies the process of extracting key images from the video, and improves the user experience. The electronic device receives a complete video at a time, allowing it to acquire all images simultaneously.

[0041] In another possible implementation of the second aspect, the electronic device obtains a first image matching the target scene from multiple first images based on the multimodal scene information of the target scene by analyzing whether the key features of the two images match.

[0042] Specifically, the electronic device acquires the first key feature of the target scene based on multimodal scene information. The electronic device also acquires the third key feature of the first image. The order in which the electronic device acquires the first key feature of the target scene and the third key feature of the first image is not limited.

[0043] The electronic device analyzes whether the first key feature of the target scene matches the third key feature of the first image. If the first key feature of the target scene matches the third key feature of the first image, the first image is determined to be the target image.

[0044] The electronic device extracts the first key feature from the scene reference image of the target scene and the third key feature from the first image, respectively. These key features are associated with scene attributes. By analyzing whether the images match through key features, interference from other features can be reduced, improving the accuracy of image matching and thus enhancing the scene matching degree of the captured images.

[0045] In one possible implementation of the second aspect, the step of the electronic device obtaining the first key feature of the target scene based on multimodal scene information is further specified. The electronic device obtains the first feature vector of the scene reference image based on scene prompts, and then extracts the first key feature of the target scene from the first feature vector.

[0046] The electronic device acquires a first feature vector, which records the mapping relationship between feature values ​​and feature locations. Feature values ​​at some locations have a strong correlation with scene attributes, while those at others have a relatively weak correlation. The electronic device extracts the first key features with the strongest correlation to scene attributes from the first vector.

[0047] In one possible implementation of the second aspect, the step of the electronic device obtaining the first feature vector of the scene reference image based on scene prompts is further specified. The target scene captured by the electronic device uses multimodal scene information, involving the scene reference image of the image modality and the scene prompts of the language modality. The electronic device uses a multimodal large model to extract the first feature vector of the scene reference image.

[0048] Specifically, the electronic device inputs the scene reference image and scene prompts into the multimodal large model to obtain the first feature vector. The multimodal large model is a deep learning-based machine learning technique that combines multimodal information such as text, images, video, and audio, and learns the relationships between different modalities to achieve richer and more accurate multimodal computation. A multimodal large model typically includes a Large Language Mode 1 (LLM) module, an image module, etc., with different modal modules processing data from different modalities. During the extraction of the first feature vector from the scene reference image, the multimodal large model references the LLM module to analyze the features of the scene reference language, ensuring that the extracted first feature vector emphasizes the features indicated by the scene reference statements, thus reducing unnecessary feature interference.

[0049] In one possible implementation of the second aspect, the step of the electronic device acquiring the third key feature of the first image is further defined. The electronic device first acquires the third feature vector of the first image, and then acquires the third key feature from the third feature vector.

[0050] The electronic device acquires a third feature vector, which records the mapping relationship between feature values ​​and feature locations. Feature values ​​at some locations have a strong correlation with scene attributes, while those at others have a relatively weak correlation. The electronic device extracts the third key features with the strongest correlation to scene attributes from the third vector, and can then combine this with scene prompts from the target scene to emphasize certain key features.

[0051] In one possible implementation of the second aspect, the step of the electronic device obtaining the third feature vector of the first image based on scene prompts is further specified. The target scene for which the electronic device captures images uses multimodal scene information, involving a scene reference image of the image modality and scene prompts of the language modality. The electronic device uses a multimodal large model to extract the third feature vector of the scene reference image.

[0052] Specifically, the electronic device inputs the first image and the scene prompts of the matched target scene into the multimodal big data model to obtain the third feature vector. During the extraction of the third feature vector from the first image, the multimodal big data model refers to the big language module to analyze the features of the scene reference language, so that the extracted third feature vector focuses more on the features indicated by the scene reference statement, thereby reducing unnecessary feature interference.

[0053] Thirdly, an electronic device is provided, which has the function of implementing the image processing method of the first or second aspect described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0054] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, enable the computer to perform the image processing method of any one of the first and second aspects described above.

[0055] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, enables the computer to perform the image processing method of any one of the first and second aspects described above.

[0056] The technical effects of any of the design methods in aspects two through five can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0057] Figure 1 This is an illustration of a highlight image;

[0058] Figure 2 A schematic diagram of the shooting preview interface provided in an embodiment of this application;

[0059] Figure 3 A schematic diagram illustrating the multimodal scene information of each first scene involved in the image processing method provided in the embodiments of this application;

[0060] Figure 4 These are schematic diagrams of the interfaces related to the display of scene capture options on an electronic device involved in the image processing method provided in the embodiments of this application;

[0061] Figure 5 Another schematic diagram of the capture scene tab bar involved in the image processing method provided in the embodiments of this application;

[0062] Figure 6 A schematic flowchart of the image processing method provided in the embodiments of this application;

[0063] Figure 7 This is a schematic diagram illustrating a partial correspondence between feature locations and feature values ​​in an image processing method provided in this embodiment of the application.

[0064] Figure 8 A partial flowchart illustrating the execution of the image processing method by an electronic device involved in the image processing method provided in the embodiments of this application;

[0065] Figure 9 A schematic diagram illustrating how an electronic device using a genetic algorithm finds the location of key features in the image processing method provided in this application embodiment;

[0066] Figure 10A schematic diagram of the interface for displaying a target image on an electronic device involved in the image processing method provided in the embodiments of this application;

[0067] Figure 11 A schematic flowchart of another image processing method provided in an embodiment of this application;

[0068] Figure 12 A schematic diagram of the software flow involved in the image processing method provided in the embodiments of this application;

[0069] Figure 13 This is a hardware structure diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0070] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0071] To facilitate understanding, some technical common sense involved in the embodiments of this application will be introduced first.

[0072] Highlight images, also known as best-of-the-day shots, are single-frame images extracted from videos or images to record the moment a memorable scene occurs. Memorable scenes can include smiling faces, jumping, running pets, badminton smashes, moments of victory, and exciting actions during airplane landings.

[0073] There are two main ways electronic devices acquire highlight images. One method involves posing the subject in a specific, engaging scene, with the electronic device capturing the scene intentionally maintained by the subject. The other method involves the electronic device capturing a fleeting, instantaneous highlight. Using the instantaneous capture method, electronic devices can obtain more realistic images of the highlight, resulting in higher image quality. For example... Figure 1 As shown, the camera of the electronic device captures highlight images corresponding to scenes such as cute pets running, people jumping, badminton smashes, and heartwarming family moments. Since these exciting scenes are fleeting and occur randomly, the user of the electronic device needs to perceive the scene promptly and press the shutter button; otherwise, the electronic device will struggle to effectively capture the highlight images.

[0074] Based on this, this embodiment provides an image processing method applied to an electronic device. The electronic device acquires multimodal scene information of the target scene for automatic capture, including a scene reference image in the image modality and scene prompts in the language modality. The electronic device uses the multimodal scene information to analyze the preview images displayed frame by frame, obtains a target image that matches the multimodal scene information of the target scene for automatic capture, and obtains a capture image corresponding to the target scene for automatic capture based on the target image. In this way, the electronic device can automatically capture the corresponding target scene image without relying on the user's prediction of exciting scenes or clicking the shutter button. Automatic capture based on multimodal scene information results in a higher scene matching degree for the captured images.

[0075] The image processing method provided in this embodiment is applied to electronic devices, which may include personal computers (PCs), tablet computers, laptops, portable computers (such as mobile phones), wearable electronic devices (such as smartwatches), augmented reality (AR) / virtual reality (VR) devices, in-vehicle computers, and other electronic devices equipped with cameras. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0076] The image processing method provided in this embodiment mainly consists of two stages: the electronic device initiating the capture process and the electronic device executing the capture operation. These two stages will be described in detail below.

[0077] In the first stage, the electronic device initiates the image capture process.

[0078] Electronic devices are equipped with a camera application, and the device can display a camera icon corresponding to the camera application on the desktop. When the electronic device receives a click on the camera icon, it can launch the camera application, start the photo-taking process, and display the shooting preview interface.

[0079] like Figure 2U11 shown in (1) is a schematic diagram of a shooting preview interface. The shooting preview interface mainly includes: parameter control area 101, image display area 102, mode control area 103, front and rear camera flip control 104, gallery control 105, shutter control 106, and zoom control 107. Among them, parameter control area 101 includes multiple parameter controls, each of which is used to respond to the user's input shooting parameter adjustment operation. The parameter controls included in the parameter control area may include, but are not limited to: flash control, AI recognition switch control, color standard control, and more detailed camera setting controls. Image display area 102 can be used to display preview images, which are images captured in real time by the electronic device through the camera. The electronic device can refresh the display content in image display area 102 in real time so that the user can preview the image currently captured by the camera. Mode control area 103 may include multiple mode controls corresponding to different shooting modes, such as photo mode control, portrait mode control, video mode control, professional mode control, aperture mode control, night scene mode control, and more mode controls. Each mode control can be labeled with text information, such as "Aperture", "Night Scene", "Portrait", "Photo", "Video", "Pro", "More", or it can be displayed with icons or a combination of text information and icons. The zoom control 107 is used to respond to the user's zoom operation on the electronic device to adjust the field of view of the lens module. The image displayed in the image display area 102 changes as the field of view of the lens module is adjusted.

[0080] In one example, the electronic device can initiate the snapshot process in any shooting mode. For instance, the electronic device can automatically initiate the snapshot process in shooting modes such as still mode, professional mode, and video mode. Figure 2 The shooting mode shown in (1) is the photo mode, and the electronic device can start the snapshot process.

[0081] In another example, the electronic device receives a capture instruction and initiates the capture process.

[0082] There are several ways for electronic devices to receive capture instructions, such as clicking on the capture mode control or the capture control, or using voice control to instruct the capture.

[0083] In one scenario, the snapshot instruction received by the electronic device can be a click operation applied to the snapshot mode control.

[0084] like Figure 2 The shooting preview interface U12 shown in (2) is a schematic diagram of the interface for the newly added capture mode control. Compared to Figure 2The shooting preview interface U11 shown in (1) has a new capture mode control 108 added to the mode control area 103. If the electronic device receives a click operation on the capture mode control 108, such as Figure 2 As shown in (3) U13, the electronic device switches the shooting mode to the capture mode and starts the capture process. The electronic device performs the capture operation of the highlight image in the capture mode.

[0085] In another case, such as Figure 2 The shooting preview interface U14 shown in (4) is a schematic diagram of the interface for adding the snapshot control. Compared to Figure 2 The shooting preview interface U11 shown in (1) has a new capture control 109 added to the image display area 102 in the shooting preview interface U14. If the electronic device receives a click operation on the capture control 109, the electronic device starts the capture process. Figure 2 As shown in (5) U15, the electronic device can switch the color parameter or graphic parameter of the capture control 109 to instruct the electronic device to start the capture process. The electronic device performs the capture operation of the highlight image in the current shooting preview scene.

[0086] In another scenario, the snapshot instruction received by the electronic device can be a voice control operation that includes a "snap command".

[0087] like Figure 2 As shown in (6) U16, the electronic device starts the voice acquisition process in the shooting preview mode to collect the user's voice information. The electronic device can use Natural Language Processing (NLP) algorithms to analyze the collected voice information and search for the existence of keywords corresponding to "capture command". If the keywords corresponding to "capture command" are found, the capture process is started. The keywords for "capture command" searched by the electronic device can be of various types, such as "capture", "start capture", "capture highlight image", "automatic capture", etc., without limitation.

[0088] If the electronic device finds the keyword corresponding to the "capture command," it can also switch the color or graphic parameters of the capture control displayed in the shooting preview interface to instruct the electronic device to start the capture process. For example... Figure 2 As shown in (5) U5, the electronic device can adjust the color parameters or graphic parameters of the capture control 109 to change the display effect, so as to instruct the electronic device to start the capture process. The electronic device performs the capture operation of the highlight image in the current shooting preview scene.

[0089] The above content mainly explains the interface and response operations related to starting the snapshot process on an electronic device. After starting the snapshot process, it is also necessary to determine which exciting scenes to capture. For ease of description, the exciting scenes to be captured are referred to as the first scene, and there can be one or more first scenes.

[0090] In one example, the electronic device can be pre-configured to capture a first scene. For instance, the electronic device can be pre-configured to capture the first scene before leaving the factory or when installing a camera application. The pre-configuration of the first scene can include pre-storing multimodal scene information for each first scene. This multimodal scene information can include scene reference images for the image modality and scene prompts for the language modality. In specific implementations, the multimodal scene information can also include scene information for other modalities besides image and language modalities, such as audio and video modalities, without limitation.

[0091] In other cases, the multimodal scene information for each first scene may also include a scene theme. Electronic devices can associate and store the scene theme, scene reference image, and scene prompts for each first scene. For example... Figure 3 The diagram shown is a schematic representation of the multimodal scene information for each first scenario.

[0092] In one specific implementation, the multimodal scene information of different first scenarios is completely different. For example, the scene theme, scene reference image, and scene prompt statements are all different in different first scenarios.

[0093] like Figure 3 Scene 1 in the image has the theme "jumping". The reference image for the scene is shown in P1. The scene prompt is "Is there a person in the picture whose feet are off the ground and in mid-air with the figure intact?"

[0094] like Figure 3 Scene 2 in the image has the theme "Sunrise". The reference image for the scene is shown on P2. The prompt for the scene is "Is there an orange-red sun hanging in the sky in the picture?"

[0095] like Figure 3 Scene 3 in the picture has the theme of "flying". The reference image for the scene is shown on P3. The prompt for the scene is "Is there a bird in the picture with its wings spread out?"

[0096] In another specific implementation, the multimodal scene information of different first scenes may be partially the same. For example, different first scenes may have the same scene theme, or different first scenes may have the same scene reference image. To facilitate differentiation and accurate identification of each first scene, the scene prompt statements for different scenes are usually different.

[0097] like Figure 3Scene 4 in the picture has the theme of "skateboarding". The reference image of the scene is shown on P4. The prompt of the scene is "Is there a person in the picture who completes a skateboarding trick in the air and the figure is complete?"

[0098] like Figure 3 Scene 5 in the image has the theme "jumping," and the reference image is shown on page 5. The scene prompt is "Is there someone in the picture with both feet in the air?" Scene 5 shares the same theme as Scene 1, and the reference image for Scene 5 is the same as that for Scene 4.

[0099] like Figure 3 Scene 6 in the image has the theme "skateboarding" and the reference image is shown on page 6. The prompt is "Are both feet on the skateboard in the picture?". Scene 5 has the same theme as Scene 4, but the reference image for Scene 5 is different from that for Scene 4.

[0100] It should be noted that the above-described multimodal scene information for various first scenarios is merely illustrative and is not intended to limit the possible implementation methods. The following will explain how electronic devices acquire multimodal scene information for each first scenario in several different cases.

[0101] In one scenario, the electronic device can store the multimodal scene information for each first scene in its local memory. For example, the electronic device can store the multimodal scene information for each first scene before leaving the factory, or it can store the multimodal scene information for each first scene after leaving the factory when the camera application is installed based on the user's operation.

[0102] Alternatively, after installing a camera app, the electronic device can add a custom editing window in the camera app's function settings or editing options to receive user input or selection operations, add multimodal scene information of exciting scenes that need to be captured, and the electronic device receives and stores it as the multimodal scene information of the first scene to be captured.

[0103] In another scenario, electronic devices can store multimodal scene information for each first scenario in a server.

[0104] For example, the electronic device can pre-store the multimodal scene information of each first scene in the server. Each time the capture process is initiated, the electronic device requests the multimodal scene information of each first scene from the server and stores it in its local memory. During the capture process, the capture operation is performed based on the multimodal scene information of each first scene. At the end of the capture process, the electronic device can also promptly clear the multimodal scene information of the first scenes stored in its local memory to release local memory. It should be noted that the server mentioned in this embodiment can include a cloud server or a physical server, and is not limited thereto.

[0105] For example, electronic devices can send captured preview images to a server, allowing the server to analyze the preview images based on pre-stored multimodal scene information of a first scene. The server analyzes the preview images to match the multimodal scene information of the first scene and can then issue a capture command to the electronic device. The electronic device receives the capture command from the server and executes the capture operation.

[0106] For example, electronic devices can store a portion of the multimodal scene information of the first scene in local memory and another portion in a server. The number of first scenes stored in the server can be greater than the number stored in local memory. In this way, the electronic device can perform a snapshot operation offline based on the portion of the multimodal scene information stored in local memory. When connected to the internet, the electronic device can perform a snapshot operation based on the other portion of the multimodal scene information stored in the server. Of course, when connected to the internet, the electronic device can also determine the first scene to be captured based on user actions. The multimodal scene information of the first scene determined by the electronic device may be stored in local memory or in a server, without limitation.

[0107] The examples above illustrate several primary scenarios that electronic devices can capture. The following will further define the scenarios that electronic devices actually need to capture each time the capture process is initiated.

[0108] In one example, the electronic device can capture images of all pre-configured first scenes each time the capture process is initiated.

[0109] For example, electronic devices are pre-configured with features such as Figure 3 The image shows multiple first scenes. Each time the capture process is initiated, the electronic device analyzes whether the captured preview image matches any scene among all the first scenes. If the preview image matches any scene, the capture operation is performed.

[0110] For example, an electronic device can pre-set a threshold for the number of scenes. If the number of first scenes pre-configured by the electronic device is less than or equal to the threshold, it can capture images of all first scenes. If the number of first scenes pre-configured by the electronic device is greater than the threshold, it can capture images of some of the first scenes. In practice, the electronic device can set the scene threshold to 3. Capturing 3 or fewer first scenes requires relatively less computation and has a relatively lower capture latency.

[0111] In another example, the electronic device may capture only a portion of the pre-configured first scenes from the entire set of first scenes. For ease of description, the scene to be captured from the pre-configured first scenes will be referred to as the second scene.

[0112] In one scenario, the electronic device can automatically identify the current scene and capture a portion of the scene that belongs to the first scene, i.e., the second scene.

[0113] Electronic devices can pre-store scene recognition algorithms or models. These algorithms or models can be used to analyze the pixel features of an image to determine the corresponding real-world scene. After initiating the capture process, the electronic device runs the scene recognition algorithm or model to analyze the real-time preview image to determine the real-time scene. The number of real-time scenes may be one or more.

[0114] If the real-time scene determined by the electronic device corresponds to a portion of the entire first scene, then that portion of the first scene is the second scene to be captured, and the electronic device captures that portion of the second scene. The electronic device determines the second scene to be captured based on scene detection; the number of second scenes can be one or more.

[0115] In one specific implementation, after the electronic device initiates the image capture process, it acquires the first one or several preview images. Then, it uses a scene recognition algorithm or model to perform scene recognition, and the recognized scene is the real-time scene. For example, if the electronic device detects pixel features of the ocean in the first one or several preview images, it determines that the real-time scene is an ocean scene.

[0116] The first scene pre-configured by the electronic device may include sports scenes, wave scenes, seabird scenes, sunrise scenes at sea, skateboarding scenes, and family-friendly scenes. Among them, wave scenes, seabird scenes, and sunrise scenes at sea are associated with the ocean scene. The electronic device can record the portion of the first scene that is associated with the ocean scene, such as wave scenes, seabird scenes, and sunrise scenes at sea, as the second scene corresponding to the real-time scene.

[0117] In another specific implementation, considering that the real-time scene of the image captured by the electronic device may change during the shooting process, the electronic device can run a scene recognition algorithm or scene recognition model throughout the entire shooting process after the capture process is initiated to detect the real-time scene corresponding to the preview image. The real-time scene analyzed by the electronic device at different times during the entire shooting process may be the same or different, and the number of real-time scenes analyzed by the electronic device each time may also be different. For example, the electronic device may analyze one or more real-time scenes at the same time.

[0118] Each time the electronic device detects a new real-time scene, it determines whether the new real-time scene is a part of the entire first scene, and determines a new second scene based on the new real-time scene. In other words, the electronic device can continuously update the second scene that needs to be captured.

[0119] During the image capture process, the electronic device can analyze whether the currently captured preview image matches the current second scene. If a match is found, an automatic image capture operation is performed. The electronic device can continuously perform multiple automatic image capture operations during the camera application's operation. The second scenes corresponding to different automatic image capture operations can be the same or different.

[0120] In another scenario, the electronic device can also determine the scene to be captured based on user input. For clarity, the scene determined by user input can be referred to as the third scene, and there can be one or more third scenes. Before or after initiating the capture process, the electronic device receives user interaction input indicating the third scene to be captured. This user interaction input can include input, clicks, or voice control.

[0121] The electronic device determines the third scene to be captured based on user input, which can be one or more scenes selected from a pre-configured set of first scenes. The electronic device can also add custom scenes based on user input. This ensures that the images acquired by the electronic device during automatic capture are more closely aligned with user needs, improves user satisfaction with automatic capture operations, and reduces unnecessary computational resource consumption and memory usage associated with automatic capture operations.

[0122] In one specific implementation, the third scene determined by the electronic device based on the user's operation is one or more of the first scenes pre-configured by the electronic device. The electronic device can display a capture scene tab in the shooting preview interface, which includes scene option controls for multiple first scenes. The electronic device receives a click operation on a scene option control of a certain first scene and uses the first scene that received the click operation as the user-indicated newly added custom scene, i.e., the third scene.

[0123] In one example, the process by which the electronic device determines the capture scene based on user actions and acquires multimodal scene information of the capture scene is as follows:

[0124] In response to the first operation, the electronic device displays a scene capture tab. The scene capture tab includes at least one scene option control for a first scene; the shooting preview interface includes a capture control, and the first operation is a click operation performed on the capture control.

[0125] The electronic device responds to a second operation on the scene selection control in the scene capture tab and determines a third scene; the third scene is the first scene corresponding to the scene selection control that was activated by the second operation.

[0126] Electronic devices acquire multimodal scene information from a third scenario.

[0127] like Figure 4 The image shown is a schematic diagram of the interface used by an electronic device to receive user input and determine the scene to be captured. Figure 2 As shown in (4), the electronic device receives a first operation, which can be a user's click operation on the capture control. The electronic device can display as shown in [example diagram]. Figure 4 The shooting preview interface U41 shown in (1) includes a scene capture tab 110, which includes multiple scene option controls for a first scene. Figure 4 As shown in (1), the scene capture tab includes four scene option controls for the first scene, namely scene option controls for scene 1 to scene 4. Furthermore, as... Figure 4 As shown in (1), the electronic device can also display the scene theme of each scene to prompt the user to select the type of scene to capture.

[0128] Electronic devices can also be configured with different selection modes for scene option controls, such as single-selection or multi-selection. Electronic devices can differentiate between selected and unselected scene option controls by highlighting the selected one and not highlighting the unselected one, thus indicating to the user which scene has been selected and which has not.

[0129] like Figure 4 In the shooting preview interface U41 shown in (1), the electronic device sets selection status prompt symbols, such as prompt circle 111, in front of each snapshot scene option control. Figure 4 In the shooting preview interface U42 shown in (2), the electronic device receives a second operation, which is a click operation applied to scene 1. The electronic device then uses scene 1, which receives the click operation, as the third scene to be captured. Figure 4 In the shooting preview interface U42 shown in (2), the electronic device can select the prompt before scene 1. In other implementations, the electronic device can also highlight the user-selected capture scene option by emphasizing the font or displaying a "√", without limitation.

[0130] In practice, the electronic device can also limit the selection and deselection operations for each scene capture option. For example, if the electronic device receives a single click on the Scene 1 option, it will display Scene 1 as selected. If the electronic device receives two or more clicks or consecutive double-clicks on the Scene 1 option, it will deselect Scene 1. When the electronic device selects Scene 1, it will highlight the corresponding scene option control. If the electronic device determines that Scene 1 is not selected, it will not highlight the corresponding scene option control.

[0131] like Figure 4 In the shooting preview interface U42 shown in (2), the electronic device can also display an "OK" control 112 on the shooting preview interface. If the electronic device receives a click operation on the "OK" control, it will use the currently selected capture scene as the third scene. In specific implementation, the electronic device can limit the number of third scenes to one or more.

[0132] The electronic device determines the third scene that needs to be captured and obtains multimodal scene information of the third scene. There are various ways to achieve this.

[0133] For example, electronic devices can receive multimodal scene information input by users. Specifically, the electronic device displays an input interface for multimodal scene information of the target scene, which includes an input box for image mode and an input box for language mode.

[0134] The electronic device receives an image input from an image-modal input box and a statement input from a language-modal input box. The image input from the image-modal input box serves as a scene reference image, and the statement input from the language-modal input box serves as a scene prompt.

[0135] For example, an electronic device sends a query request to a server; the query request is used to request multimodal scene information of the target scene. The electronic device receives the multimodal scene information of the target scene sent by the server.

[0136] For example, electronic devices can also directly obtain multimodal scene information of the target scene from local memory.

[0137] Electronic devices determine a third scene based on user actions and can display the multimodal scene information of this determined third scene on the shooting preview interface, such as displaying the scene theme on the shooting preview interface. Figure 4 In the shooting preview interface U43 shown in (3), the electronic device can adjust the display parameters of the capture control 109 to indicate to the user that it is currently in the automatic capture process. The electronic device can also display the scene theme 113 of the determined third scene to indicate to the user that the third scene is currently being automatically captured.

[0138] In practical implementation, the electronic device can also receive click operations from the user on scene theme 113 of the third scene, and update the third scene that needs to be automatically captured. For example, if the third scene currently being captured by the electronic device is scene 1, the electronic device receives a click operation on the currently displayed scene 1 and switches the currently captured scene 1 to another scene.

[0139] In addition, the electronic device can also receive click operations from the user on the capture control 109 to exit the automatic capture process.

[0140] In another specific implementation, the electronic device can also receive the user's voice control operation to determine the third scene that the user instructs to capture.

[0141] If the electronic device determines that the snapshot scene indicated by the user is any of the pre-configured first scenes, it will determine the snapshot scene indicated by the user as the third scene and perform an automatic snapshot operation for the third scene.

[0142] like Figure 4 As shown in (4), the electronic device receives the user's voice control operation, analyzes the user's voice input command using natural language processing technology, and determines the snapshot scene indicated by the user. For example, if the electronic device analyzes the user's voice input command as "capture a jumping scene," it can determine that the snapshot scene indicated by the user is a "jumping scene." The electronic device determines the "jumping scene" as... Figure 3 From one of the multiple first scenarios shown, the third scenario to be captured can be determined as a "jumping scenario". It should be noted that the electronic device receives the user's voice control command and determines the user-indicated capture scenario by analyzing the scenario theme contained in the voice control command. Of course, the electronic device can also determine the user-indicated capture scenario by analyzing the scenario prompts contained in the voice control command; there is no limitation. The electronic device can further determine the user-indicated third scenario by receiving user uploads or requests from the server, or by using multimodal scene information of the third scenario.

[0143] In the aforementioned scenarios, based on user input, the electronic device selects all or part of a pre-configured set of scenes as the third scene to be captured. The electronic device can directly retrieve the multimodal scene information corresponding to the third scene from its local memory, or it can retrieve this information from a server. The multimodal scene information includes a scene reference image and scene prompts. Based on the acquired multimodal scene information, the electronic device captures the image corresponding to the third scene.

[0144] In another specific implementation, the electronic device receives user input and adds a custom scene as a third scene that needs to be captured.

[0145] like Figure 5 The shooting preview interface U51 shown in (1) of the electronic device, the scene option control displayed in the snapshot scene tab 110, may also include a custom scene option control. When the electronic device receives a click operation from the user on the custom scene option control, it enters the process of creating a user-defined snapshot scene. For example... Figure 5 In the shooting preview interface U52 shown in (2), the electronic device can receive the scene theme of the user-input custom scene, such as "smile". The electronic device can define the user-input custom scene as the third scene by receiving the user's click operation on the "OK" control after inputting the custom scene.

[0146] Or such as Figure 4 As shown in (4), the electronic device can also receive the user's voice control operation, analyze the user's instruction to capture the custom scene, and determine the user's instruction to capture the custom scene as the third scene.

[0147] If the third scene selected by the user received by the electronic device is a custom new scene, the electronic device also needs to obtain the multimodal scene information of the custom third scene.

[0148] For example, electronic devices can receive multimodal scene information input by users. Specifically, the electronic device displays an input interface for multimodal scene information of the target scene, which includes an input box for image mode and an input box for language mode.

[0149] The electronic device receives an image input from an image-modal input box and a statement input from a language-modal input box. The image input from the image-modal input box serves as a scene reference image, and the statement input from the language-modal input box serves as a scene prompt.

[0150] For example, an electronic device sends a query request to a server; the query request is used to request multimodal scene information of the target scene. The electronic device receives the multimodal scene information of the target scene sent by the server.

[0151] Furthermore, the storage of scene reference images by the electronic device mentioned in the above embodiments may refer to the electronic device directly storing the original image data of the scene reference image. In other cases, considering that when the electronic device analyzes whether the preview image matches the scene reference image of the capture scene, it actually analyzes the feature vectors and key feature positions of the preview image in real time, and whether they match the feature vectors and key feature positions of the scene reference image of the capture scene. Based on this, the electronic device can also pre-process and calculate the scene reference images to obtain the image feature vectors and key feature positions of each scene reference image, and store the corresponding feature vectors and key feature positions of each scene reference image locally. In this way, when the electronic device subsequently analyzes whether the preview image matches the scene reference image, it can only obtain the feature vectors and key feature positions of the preview image and analyze whether they match the feature vectors and key feature positions of the previously stored scene reference image. The electronic device's use of image feature vectors and key feature positions instead of scene reference images saves the step of temporarily obtaining the feature parameters of the reference image in the automatic capture process, reduces image analysis time, and improves automatic capture efficiency.

[0152] In the second stage, the electronic device performs the snapshot operation.

[0153] The scene that the electronic device needs to capture is denoted as the target scene. The target scene can be the first scene, second scene, or third scene mentioned above. The electronic device determines the target scene to be captured and acquires the multimodal scene information of each target scene. The electronic device acquires a preview stream in real time and displays multi-frame preview images of the preview stream frame by frame on the preview interface. The electronic device analyzes whether the frame preview images match the multimodal scene information of the target scene to be captured, and then performs the capture operation.

[0154] In one specific implementation, reference is made to... Figure 6 The electronic device performs the snapshot operation in three main steps:

[0155] S61: The electronic device displays a shooting preview interface; the shooting preview interface is used to display a preview stream, which includes preview images displayed frame by frame.

[0156] The electronic device initiates the snapshot process and displays a shooting preview interface. The shooting preview interface displays preview images of the preview stream frame by frame. In this embodiment, the electronic device analyzes the preview images and performs the snapshot operation based on the analysis results.

[0157] It should be noted that in various shooting modes, such as photo mode, shooting mode, and night mode, the inherent workflow of a camera application includes continuously acquiring preview images and updating their display on the shooting preview interface. This embodiment only analyzes the preview images acquired using the inherent workflow of the electronic device; it does not involve adding a new operation of acquiring preview images.

[0158] S62: Obtain the target image; wherein, the target image is a preview image of multimodal scene information matching the target scene; the multimodal scene information includes a scene reference image of the image mode and a scene prompt statement of the language mode, and the scene prompt statement is used to describe the scene attributes of the target scene.

[0159] The electronic device initiates the image capture process to acquire multimodal scene information of the target scene to be captured. The electronic device can obtain the multimodal scene information of the target scene from its local memory or from a server.

[0160] The number of target scenes can be one or more. The multimodal scene information of the target scenes may include scene reference images and scene prompts, or scene themes. The scene reference images acquired by the electronic device can be data in full image format or data in image feature format, without limitation.

[0161] Electronic devices can compare and analyze the acquired preview images with the multimodal scene information of the target scene to determine whether the preview image matches the target scene. In practice, the electronic device analyzes whether the acquired preview images match the target scene by analyzing all acquired preview images frame by frame, analyzing whether each frame of the preview image matches the multimodal scene information of the target scene, so as to detect the preview image that matches the target scene more promptly.

[0162] In other cases, electronic devices can also analyze only a portion of the captured preview images. For example, they can perform frame-skipping analysis on the preview images at intervals of one or several frames to analyze whether the partial preview images match the multimodal scene information of the target scene. This allows for timely detection of images matching the target scene while reducing image analysis operations and time consumption, further improving the automatic capture efficiency of the electronic device.

[0163] There are various methods for electronic devices to analyze whether a preview image matches the multimodal scene information of the target scene. The main methods are to analyze the pixel features of the preview image and whether it matches the scene reference image of the target scene and the scene features indicated by the scene prompts.

[0164] In one specific implementation, the electronic device may pre-store image analysis algorithms. The electronic device can use the pre-stored image analysis algorithms to extract pixel features from a preview image, and also use the pre-stored image analysis algorithms to extract pixel features from a scene reference image of the target scene. The electronic device compares the pixel features of the preview image with the pixel features of the scene reference image of the target scene, analyzing whether the pixel features of the two frames match. If the electronic device determines that the pixel features of the preview image match the pixel features of the scene reference image of the target scene, that is, it determines that the preview image matches the multimodal scene information of the target scene.

[0165] In another specific implementation, the electronic device can also determine whether the preview image matches the target scene by analyzing the key features of the scene reference image of the target scene and whether they match the key features of the preview image.

[0166] In this context, key features of the target scene refer to those features that accurately reflect the scene attributes of the target scene and enable the feature objects in the image to be distinguished from feature objects in other images. Key features can refer to pixel features of the image modality or parametric features of the language modality.

[0167] Correspondingly, the key features of a preview image refer to the pixel features that can accurately reflect the scene attributes of the preview image and enable the feature objects in the preview image to be distinguished from the feature objects in other images.

[0168] like Figure 3 The multimodal scene information for each scenario shown includes scene reference images and scene prompts that reflect the scene attributes of each scenario and distinguish them from other scenarios. For example, in scenario 1, the scene reference image P1 includes pixel features corresponding to the feet of the person in the background sky, and the scene prompt includes the parameter feature "feet off the ground and in mid-air". As another example, in scenario 2, the scene reference image P2 includes pixel features corresponding to the sun and the sky, with an orange-red color attribute. The pixel region containing the sun's pixel features is located in the central region of the pixel region containing the sky's pixel features, and the scene prompt includes the parameter feature "the orange-red sun is suspended in mid-air".

[0169] In this embodiment, the electronic device combines scene prompts to obtain pixel features in the scene reference image, especially some pixel features that are closely related to scene attributes.

[0170] Image pixel features mainly include color features, texture features, shape features, and spatial relationship features. Color features are global features, describing the surface properties of the color attributes of objects within an image or image region. Texture features are also global features, describing the surface properties of the texture attributes of objects within an image or image region. Shape features have two representation methods: contour features and region features. Image contour features mainly target the outer boundaries of objects, while image region features relate to the entire shape region. Spatial relationship features refer to the spatial positions or relative directions between multiple objects segmented from an image. Spatial relationships mainly include connectivity / adjacency, overlap / intersection, and containment / enclosure relationships. In other words, the essence of an image is a collection of multi-dimensional features.

[0171] Electronic devices analyze and process images by typically breaking them down into feature vectors. These feature vectors are then arranged to create feature vectors, which record the mapping between feature locations and feature values. Different feature locations correspond to different feature values. In fact, each feature location contains feature values ​​with multiple dimensions, such as the multidimensional feature values ​​of color attributes, texture attributes, and shape attributes mentioned earlier.

[0172] like Figure 7 The diagram shown illustrates a partial correspondence between feature locations and feature values ​​obtained from image analysis and processing by an electronic device. Figure 7 It can be seen that among the pixel features of an image extracted by an electronic device, some locations record information that constitutes basic image information, such as image size, sharpness, and brightness. These locations can be recorded as basic feature locations (e.g., ...). Figure 7 (As shown in (A)). Additionally, some locations record key information closely related to the image content, such as jumping, airborne, badminton, and people; these locations can be designated as key locations. Typically, an image can have hundreds, thousands, or even more feature locations, with the majority being basic feature locations (such as...). Figure 7 (As shown in (B)). Among all the feature locations corresponding to the feature vector of an image, only a small number of locations are key feature locations. The features of these few key feature locations can accurately reflect the key content of the image. These key feature locations are recorded as key features.

[0173] In this embodiment, key features of various types of images can be obtained during image processing. The matching degree of the images can be analyzed through key features, resulting in less interference information and higher matching accuracy.

[0174] In one specific implementation, the electronic device analyzes whether the preview image matches the target scene by analyzing key features of the target scene and key features of the preview image. Specifically, this may include the following steps:

[0175] S621: The electronic device obtains the first key feature of the target scene based on multimodal scene information.

[0176] The electronic device identifies the features at key locations within the entire feature set of the scene reference image and designates them as the first key feature of the target scene. It's important to note that the first key feature of the target scene acquired by the electronic device is not limited to a single location; it can refer to a set of features from multiple locations. The number of feature locations is typically three, five, or even more.

[0177] A scheme for an electronic device to acquire the first key feature of a target scene may include the following steps:

[0178] S6211: The electronic device obtains the first feature vector of the scene reference image based on the scene prompt statement.

[0179] The electronic device acquires a feature vector from the scene reference image, denoted as the first feature vector. It should be noted that the first feature vector acquired by the electronic device is not limited to a single dimension of the scene reference image, but rather is a set of feature vectors representing pixel features across multiple dimensions. The electronic device can directly extract the first feature vector from the scene reference image using feature extraction algorithms.

[0180] In one specific implementation, the electronic device can utilize a multimodal large model to extract a first feature vector from a scene reference image.

[0181] Multimodal large models are a deep learning-based machine learning technique that combines multimodal information such as text, images, videos, and audio. By learning the relationships between different modalities, it achieves richer and more accurate multimodal computation. Multimodal large models typically include a Large Language Model (LLM) module, an image module, etc., with different modal modules processing data of different modalities.

[0182] In this embodiment, the multimodal scene information of the target scene includes a scene reference image and a scene prompt statement (i.e., a prompt). The scene reference image is in the image modality, and the scene prompt statement is in the language modality, involving multimodal information and suitable for a large multimodal model. The electronic device can pre-train a large multimodal model and use the large multimodal model to extract the first feature vector of the scene reference image of the target scene.

[0183] The multimodal large model is essentially a neural network, comprising a large language module for handling language modalities and an image processing module for handling image modalities. The image processing module is essentially a hidden layer containing multiple convolutional layers and normalization layers, which performs feature vector extraction on the input scene reference image. During the extraction of the first feature vector from the scene reference image, the multimodal large model references the large language module to analyze the features of the scene reference language, ensuring that the extracted first feature vector emphasizes the features indicated by the scene reference statement, thus reducing unnecessary feature interference.

[0184] The scheme of pre-training a multimodal large model for electronic devices can be a commonly used neural network training scheme. The electronic device acquires multiple sets of sample data. Each set of sample data includes a first class of sample data and a second class of sample data. The first class of sample data contains input sample data with features to be classified, and the second class contains target sample data with target features. It should be noted that the features to be classified refer to features that need to be processed for recognition, classification, labeling, etc., such as multi-dimensional features or prompt statement features. Target features are one or more classes of features that need to be identified or processed, such as feature vectors that match prompt statements. Feature association is established between the input sample data and the target sample data.

[0185] In this model, the input sample data serves as the input values ​​for the multimodal large-scale model, while the target sample data serves as the target values ​​for the neural network. The input sample data includes sample images and prompt statements, while the target sample data contains feature vectors. The target sample data can be the main feature vectors extracted from all feature vectors in the sample images, focusing on the features indicated by the prompt statements. In other words, the elements of the input sample data and the target sample data are essentially the same, but their feature attributes may differ.

[0186] In this embodiment, the electronic device can train a multimodal neural network to obtain a large multimodal model with feature extraction capabilities, and then use the trained large multimodal model to perform the feature vector extraction operation in image processing. Alternatively, the electronic device can directly acquire a large multimodal model trained by other devices to perform the feature vector extraction operation in the image processing method provided in this embodiment.

[0187] The process of training a multimodal neural network with electronic devices to obtain a large multimodal model with feature extraction capabilities mainly includes: the electronic device acquiring sample data and a multimodal neural network without setting relevant parameters of the weight matrix; inputting the sample data into the multimodal neural network for iterative training; gradually optimizing the weight matrix in the multimodal neural network according to the loss function until the loss function converges, so that the weight matrix in the multimodal neural network reaches the optimal value; at this point, the multimodal neural network can be used as a large multimodal model.

[0188] Specifically, the electronic device uses a loss function to calculate the difference between the predicted value and the true value, and then uses the difference to backpropagate and modify the weight matrix. After that, the training sample data is input into the multimodal neural network after the weight matrix is ​​modified, and multiple iterations of training are performed until the difference is minimized, that is, the multimodal neural network converges.

[0189] When a multimodal neural network (MNN) undergoes iterative training using input and target values, it employs a backpropagation algorithm to correct the parameters of the initial weight matrix during training, thereby minimizing the reconstruction error loss. Specifically, the difference between the predicted and target values ​​obtained in each iteration is calculated and then propagated back into the MNN to modify its weight matrix. The modified MNN is then used for iterative training again, and the difference from each iteration is used to update the weight matrix. This process of iterative training and weight matrix modification continues until the difference between the predicted and target values ​​falls within the acceptable loss range of the loss function, indicating convergence of the loss function. This is how the MNN achieves convergence through multiple iterative training iterations. The backpropagation algorithm, primarily driven by error loss, is used to obtain the optimal parameters for a large multimodal model. Of course, other similar algorithms can also be used for iterative optimization, without limitation.

[0190] like Figure 8 The diagram shows a partial flowchart of the image processing method performed by the electronic device. The electronic device inputs the scene reference image and scene prompts of the target scene into a multimodal large model. Using the hidden layer of the large language module of the multimodal large model, it extracts the first feature vector of the scene reference image and calculates the feature mean. It should be noted that in specific implementations, the hidden layer of the large language module typically has 32 layers, such as LLaMA2 and Qwen, with the last 16 layers extracting features containing the main feature locations of key positions. In this embodiment, the electronic device can complete the extraction of pixel features required for this implementation using only the last 16 layers.

[0191] In addition, the multimodal neural network used by electronic devices to train multimodal large models can be open-source multimodal neural networks such as MiniGPT-4[1], so as to train a multimodal large model with feature vector extraction function.

[0192] S6212: Electronic device acquires the first key feature in the first feature vector.

[0193] The electronic device inputs a scene reference image and scene prompts for the target scene into a multimodal large model to obtain the first feature vector of the target scene. The electronic device then extracts the first key feature associated with the scene attributes of the target scene from the first feature vector, which may include the following steps:

[0194] S62121: Electronic devices use a key feature location search algorithm to find the first feature location of the scene attribute associated with the target scene from all feature locations of the first feature vector; wherein, the key feature location search algorithm includes a genetic algorithm.

[0195] Electronic devices can utilize key feature location search algorithms to find the primary feature location of a key feature. For example, electronic devices can use genetic algorithms, reinforcement learning-based search algorithms, exhaustive search, random search, fast search, greedy search strategies, etc., to find the location of key features.

[0196] In one specific implementation, such as Figure 9 As shown, the electronic device uses a genetic algorithm to locate the key feature positions. The genetic algorithm mainly includes the following steps:

[0197] 1. The electronic device initializes individuals and the population. The electronic device randomly initializes any dimension position as the initial generation of individuals (10 individuals), which are then placed into the population to form the initial population.

[0198] 2. Fitness assessment of electronic devices. Fitness assessment is used to evaluate the adaptability of an individual to the current environment and is a basic indicator for evaluating the quality of an individual. In this method, fitness assessment can use the Top 3 accuracy on a video test set with manually labeled information, that is, to determine whether the best 3 frames calculated using corresponding location features are manually labeled highlights.

[0199] 3. Electronic devices undergo positional mutations. Each round of evolution involves one mutation, primarily either a change or an addition. The electronic device may change any position among all the positions recorded by a single individual (50% probability), or randomly add a new position (50% probability).

[0200] 4. The electronic device performs crossover, for example, every 5 rounds. The electronic device can select two superior individuals as parents through a tournament selection method. Shared positions between the two parents are retained, while non-shared positions are randomly retained (with a 50% probability). The electronic device can first randomly select 10% of individuals as a candidate subset, and then select the individual with the highest fitness from this subset.

[0201] 5. Electronic devices undergo elimination. No elimination occurs if the population size has not reached the preset number (e.g., 30 individuals). Once the population size reaches the preset number, for each new individual added, the individual with the lowest fitness will be eliminated from the population.

[0202] 6. The electronic device stops searching. When the highest fitness level in the population remains unchanged for 30 rounds of evolution, the evolutionary process stops, and the individual with the highest fitness level is output as the search result. In this way, the electronic device can obtain the location of the first feature containing the key feature.

[0203] In another specific implementation, the electronic device can also use a greedy search algorithm to find the location of the first feature where the key feature is located, which mainly includes the following process:

[0204] 1. The electronic device iterates through all one-dimensional features, calculates the test set accuracy based on the usage effect of each feature, and retains only the one-dimensional feature with the highest accuracy.

[0205] 2. Each time, the electronic device adds one dimension to the existing features and calculates the accuracy based on the features of the new dimension. For example, if there are 20 dimensions in total and the 7th dimension performs best, then the candidate set for the second search is [7,1], [7,2], ..., [7,,20]. The optimal two-dimensional feature is selected, and the location of the third dimension is explored based on the new two-dimensional feature.

[0206] 3. The electronic device continues searching until all dimensions have been traversed, and the combination of positions with the highest accuracy is taken as the search result.

[0207] It should be noted that the most accurate results obtained by electronic devices do not necessarily appear in all dimensions, but rather in a subset of all dimensions. This subset is precisely the key features contained in the feature vector of the scene reference image.

[0208] In another specific implementation, the electronic device can also use a fast search algorithm to find the location of the first feature where the key feature is located, which mainly includes the following process:

[0209] 1. The electronic device iterates through all one-dimensional features, calculates the test set accuracy based on the usage effect of each feature, and sorts all dimensions based on the accuracy of each single dimension.

[0210] 2. The electronic device sequentially extracts top1, top2, and so on features from all dimensions to form candidate position information, calculates the accuracy of each feature, and takes the feature position with the highest accuracy as the search result.

[0211] Electronic devices can use fast search methods to adapt to new open scenes that users expect to capture relatively quickly.

[0212] Of course, in practice, electronic devices can also use other algorithms to obtain the first feature position in the feature vector of the scene reference image, without limitation.

[0213] S62122: Extract the first key feature from the electronic device; wherein, the first key feature is the feature corresponding to the first feature position in the first feature vector.

[0214] Based on the mapping relationship between feature positions and feature values ​​recorded in the first feature vector, and the first feature position, the electronic device can extract the first key feature from the first feature vector. For example, if the feature dimension of the first feature vector of the scene reference image obtained by the electronic device through a multimodal large model is 1×N×4096, and the key feature position determined by the electronic device through a genetic algorithm is a set of discrete sequences [4,7,27], then the first key feature corresponding to the key feature is a feature vector with a dimension of 1×3×4096. In essence, this is similar to the electronic device extracting the feature vector corresponding to the key position from the first feature vector of the scene reference image as the first key feature of the target scene.

[0215] Table 1 below shows the differences in the representational ability of features at different locations on the image, where the difference indicates the number of Top 1 correct answers / total number of samples. Table 1 shows that the representational ability of key features determined by combining scene reference statements and key locations is relatively strong. Electronic devices utilize the first key feature of the scene reference image to represent the scene attributes of the scene reference image, resulting in stronger representational ability, easier matching of the target scene image, and reduced interference from features at other locations, thus improving matching accuracy.

[0216] Table 1

[0217]

[0218] It should be noted that after the electronic device starts the capture process, it can use a multimodal large model to extract the first feature vector of the scene reference image of the target scene, then use a genetic algorithm to obtain the key feature positions of the scene reference image of the target scene, and then combine the key feature positions to extract the first key feature from the first feature vector of the scene reference image.

[0219] In other cases, electronic devices can also use a multimodal large model to extract the first feature vector of the scene reference image of the target scene in advance when the target scene is pre-configured, then use a genetic algorithm to obtain the key feature positions of the scene reference image of the target scene, and then combine the key feature positions to extract the first key feature from the first feature vector.

[0220] The electronic device pre-stores the first feature vector and first feature position of the scene reference image of the target scene as multimodal scene information locally, or directly stores the first key feature and first feature position of the scene reference image of the target scene locally. In this way, after starting the capture process, the electronic device can directly obtain the first key feature and key feature position of the scene reference image of the target scene without having to perform feature extraction operations on the scene reference image on a temporary basis, thus improving capture efficiency.

[0221] S622: The second key feature for electronic devices to acquire preview images.

[0222] The electronic device extracts key features from the preview image, known as the second key features, which are then compared with the first key features of the scene reference image of the target scene. It should be noted that the second key features of the preview image acquired by the electronic device are not limited to a single location; they can refer to a set of features from multiple locations. The number of feature locations is typically three, five, or even more.

[0223] The second key feature of an electronic device acquiring a preview image mainly includes the following steps:

[0224] S6221: Electronic device acquires the second feature vector of the preview image.

[0225] The electronic device acquires the feature vector of the preview image, which is denoted as the second feature vector. It should be noted that the second feature vector of the preview image acquired by the electronic device is not limited to a single dimension of the preview image, but rather is a set of feature vectors representing pixel features across multiple dimensions. The electronic device can then use feature extraction algorithms to directly extract the second feature vector from the scene reference image.

[0226] In one specific implementation, the electronic device can utilize a multimodal large model to extract a second feature vector from the preview image.

[0227] Continue as Figure 8 As shown, the electronic device inputs the preview image and the scene prompt statement of the target scene to be matched into the multimodal large model respectively. Using the hidden layer of the large language module of the multimodal large model, the second feature vector of the preview image is extracted and the feature mean is calculated.

[0228] It should be noted that the image capture method for S61 and S62 provided in this embodiment is mainly an image processing method for matching a preview image with a single target scene. If the electronic device needs to capture multiple target scenes, it can input each preview image frame and the scene prompt statement for each target scene into a multimodal large model to extract the second feature vector for each preview image to match each target scene. During the extraction of the second feature vector from the preview image using the multimodal large model, the electronic device will consider the feature attributes indicated by the scene prompt statement of the target scene, emphasizing the extraction of certain feature vectors. When matching the same preview image frame with different target scenes, the second feature vectors obtained from extracting the same preview image frame based on the scene prompt statements of different target scenes may not be completely identical, mainly due to different emphasized feature attributes, which is more conducive to the subsequent matching of the emphasized feature attributes.

[0229] S6222: Electronic device acquires the second key feature from the second feature vector.

[0230] The electronic device inputs the scene prompts and preview images of the target scene into a multimodal large model to obtain the second feature vector of the preview image. The electronic device then extracts the second key features associated with the scene attributes of the target scene from the second feature vector, which may include the following steps:

[0231] S62221: Electronic devices use a key feature location search algorithm to find the second feature location of the scene attribute associated with the target scene from all feature locations of the second feature vector; wherein, the key feature location search algorithm includes a genetic algorithm.

[0232] Electronic devices can utilize key feature lookup algorithms to locate the key feature positions, denoted as the second feature positions. For example, electronic devices can use genetic algorithms, reinforcement learning-based search algorithms, exhaustive search, random search, fast search, greedy search strategies, etc., to find the second feature positions of key features in a preview image.

[0233] In other embodiments, the electronic device may also directly use the first feature position of the key feature of the scene reference image of the matched target scene as the second feature position of the preview image.

[0234] Continue as Figure 7 As shown, when an electronic device performs image processing, the key features extracted from two frames of images for scene matching are usually in the same or similar positions. To determine whether the preview image matches the target scene, the electronic device can directly refer to the first feature position of the target scene's scene reference image as the second feature position of the preview image.

[0235] In cases where electronic devices store the first feature vector and first feature position of the scene reference image of the target scene as multimodal scene information of the target scene locally in advance, the electronic device can directly obtain the first feature vector and first feature position of the scene reference image of the target scene from local memory after starting the capture process.

[0236] S62222: Extract the second key feature from the electronic device; wherein, the second key feature is the feature corresponding to the second feature position in the second feature vector.

[0237] Based on the second feature vector of the acquired preview image, the electronic device can extract the second key feature from the second feature vector by means of the recorded mapping relationship between feature position and feature value, as well as the key feature position of the target scene.

[0238] S623: The electronic device analyzes whether the first key feature of the target scene matches the second key feature of the preview image.

[0239] S624: If the first key feature of the target scene matches the second key feature of the preview image, the electronic device determines that the preview image is the target image.

[0240] Electronic devices can calculate the feature similarity between the first key feature of the target scene and the second key feature of the preview image to determine whether the preview image matches the target scene. For example, electronic devices can use parameters such as cosine similarity and Euclidean distance of each key feature as feature parameters for the corresponding feature similarity.

[0241] In one specific implementation, the electronic device uses cosine similarity as a parameter for calculating feature similarity. The electronic device acquires the first key feature f of the scene reference image of the target scene. proto And the second key feature f of the preview image v According to the cosine similarity calculation formula:

[0242]

[0243] To calculate the feature similarity between the two, where cos sim This is the obtained feature similarity.

[0244] The electronic device obtains the feature similarity between the first key feature of the scene reference image of the target scene and the second key feature of the preview image, and then determines whether the feature similarity meets the preset matching conditions, thereby determining whether the preview image matches the target scene.

[0245] In one specific implementation, the preset matching conditions set by the electronic device can be of various types.

[0246] For example, the preset matching conditions set by an electronic device may include whether the feature similarity exceeds a preset similarity.

[0247] The electronic device pre-sets a fixed similarity threshold as a preset matching condition. By acquiring the feature similarity between each frame of the preview image and the scene reference image of the target scene, the electronic device can determine whether the feature similarity exceeds the preset similarity threshold. If the feature similarity exceeds the preset similarity threshold, the preview image is determined to match the target scene.

[0248] In one scenario, the electronic device can set the same preset similarity for different target scenes. The electronic device compares the feature similarity of each frame of the preview image with the scene reference image of each target scene to the preset similarity, and determines the preview image that exceeds the preset similarity as a match with the scene reference image of the target scene.

[0249] In this case, after the electronic device determines that the feature similarity of the preview image exceeds the preset similarity, it can distinguish which target scene the preview image matches, or it can not distinguish which target scene the preview image matches. It only needs to determine that the feature matching degree between the preview image and the scene reference image of a certain target scene exceeds the preset similarity.

[0250] In another scenario, electronic devices can also be configured with different preset similarity levels for different target scenarios.

[0251] For example, an electronic device determines whether a preview image matches Scene 1. The electronic device inputs the preview image and the scene prompts for Scene 1 into a multimodal large model, extracts the second feature vector of the preview image, and combines this with the key feature locations of the scene reference image for Scene 1 to determine the second key feature. The electronic device calculates the feature similarity between the second key feature of the preview image and the first key feature of the scene reference image for Scene 1. If the feature similarity exceeds a preset similarity for Scene 1, the preview image is determined to match Scene 1; otherwise, the preview image is determined not to match Scene 1.

[0252] S63: The electronic device generates a captured image; the captured image is obtained based on the target image.

[0253] Once the electronic device determines the target image that matches the multimodal scene information of the target scene, it can acquire a captured image of the target scene based on the target image, which is then used as the image obtained by performing a capture operation on the target scene.

[0254] Electronic devices can acquire captured images of matching target scenes in various ways based on target images of the target scene.

[0255] In one example, the electronic device can directly use the target image that matches the target scene as the captured image.

[0256] Alternatively, the electronic device can use the next frame of the target image that matches the target scene as the captured image. The next frame of the target image, as referred to here, means a frame captured by the electronic device's camera at a fixed frame rate during the preview stream acquisition process, at a time following the acquisition time of the target image.

[0257] In another example, an electronic device can automatically open the shutter to capture a frame as a snapshot when it detects a target image that matches the target scene.

[0258] After an electronic device acquires a captured image, it can output the captured image in various ways.

[0259] In one example, the electronic device can directly switch the camera's shooting preview interface to an image display interface, where the captured image is displayed.

[0260] For example, electronic devices displaying such as Figure 10 During the process of capturing the preview interface U101 shown in (1), if a capture image P0 matching the target scene is obtained based on the preview image, such as Figure 10 As shown in (2), the electronic device can directly switch the shooting preview interface U101 to the image display interface U102, and the electronic device will display the captured image on the image display interface U102.

[0261] Or such as Figure 10 As shown in (3), the image display interface U103 of the electronic device can also be equipped with a save control 1001 and a cancel control 1002. If the electronic device receives a click operation on the save control 1001, it can save the captured image, for example, by saving the captured image to the album. If the electronic device receives a click operation on the cancel control 1002, it can choose not to save the captured image to the album to avoid unnecessary memory usage.

[0262] In another example, the electronic device can also update the album thumbnail display area of ​​the shooting preview interface with a thumbnail of the captured image.

[0263] For example Figure 10 As shown in (4), after the electronic device performs a snapshot operation to obtain a snapshot image, it saves the snapshot image to the album. In addition, the electronic device obtains a thumbnail of the snapshot image and updates and displays the thumbnail of the snapshot image in the album thumbnail display area 1003 of the shooting preview interface U104. If the electronic device receives a click operation from the user on the album thumbnail display area, it can display the snapshot image, such as... Figure 10 As shown in (2) of the text.

[0264] In summary, the image processing method provided in this embodiment utilizes a multimodal large model to extract the first key features of the target scene to be captured and the second key features of each preview image. This allows for accurate analysis of whether the preview images match each target scene, enabling automatic capture of highlight images of corresponding exciting scenes. By searching for the location of key features, the electronic device can eliminate interference information from all features. The key feature search strategy based on a genetic algorithm can effectively locate the position of key features and find more representative features with better capture discrimination. Furthermore, the electronic device provides solutions based on various preset capture scenes, mainly including fixed scenes and general open scenes, offering good scalability and applicability to both fixed preset scenes and general open scenes. By leveraging the understanding and knowledge fusion capabilities of the large language module of the multimodal large model, as well as the representation capabilities of models such as ViT and CLIP, the electronic device achieves more efficient computation, further improving the matching degree of the acquired images. This enables the capture of highlight images from multiple angles and scenes, making the acquired images more in line with user needs.

[0265] The image processing method provided in the above embodiments is mainly a scheme for automatically capturing target images matching the target scene during the photo preview process of an electronic device. In other embodiments, the electronic device can also directly perform image processing operations on multiple consecutive frames of images in the acquired video, and automatically extract the target image matching the target scene from the multiple consecutive frames of images in the video.

[0266] like Figure 11 The diagram shown illustrates another process flow for an electronic device to execute display control methods. The scheme for an electronic device to automatically extract a target image matching a target scene from a video can mainly include the following steps:

[0267] S111: The electronic device acquires a first video; the first video includes multiple consecutive frames of first images.

[0268] The electronic device acquires the first video to be processed, which can be a video captured by the electronic device's camera, or a video acquired by the electronic device from another device via a network or other communication methods. The electronic device executes the image processing method provided in this embodiment to extract the target image from the video. This can be a function implemented based on the camera application of the electronic device, or a function implemented based on an image processing application or video editing application within the electronic device, and is not limited thereto.

[0269] The electronic device determines the scene to be matched, denoted as the target scene. The target scene can be the first scene, the third scene, or the third scene mentioned above, which will not be elaborated further here. The electronic device obtains the multimodal scene information of the target scene to be matched. The electronic device can obtain the multimodal scene information of the target scene from its local memory or from a server.

[0270] The electronic device may identify one or more target scenes. The multimodal scene information of the target scenes may include scene reference images and scene prompts, or it may include scene themes. The scene reference images acquired by the electronic device may be data in full image format or data in image feature format, without limitation.

[0271] The electronic device can receive user input and determine the first video to be processed. The first video acquired by the electronic device includes multiple consecutive frames. The electronic device can analyze each frame of the first video frame by frame to determine whether it matches the target scene. The electronic device can also analyze a portion of the first video by skipping frames to determine whether that portion matches the target scene. For ease of description, the image that needs to be analyzed among the multiple frames of the first video is referred to as the first image.

[0272] The order in which the electronic device acquires the first video and acquires multimodal scene information of the target scene is not limited.

[0273] S112: The electronic device acquires the target image; the target image is the first image of multimodal scene information matching the target scene; the multimodal scene information includes a scene reference image of the image mode and a scene prompt statement of the language mode, the scene prompt statement being used to describe the scene attributes of the target scene.

[0274] The electronic device analyzes each frame of the first image to determine if the first image matches any target scene. This analysis primarily involves checking the multimodal scene information of the target scene. Various methods can be used to analyze this multimodal scene information, but the analysis mainly focuses on the pixel features of the first image and whether it matches the scene reference image and scene features indicated by scene prompts.

[0275] In one specific implementation, the electronic device may pre-store an image analysis algorithm. The electronic device extracts pixel features from a first image, as well as scene reference images of the target scene and scene features indicated by scene prompts. The electronic device compares the pixel features of the first image with the scene features of the target scene, analyzes whether the similarity between the two exceeds a preset threshold, and determines to preview the multimodal scene information of the first matching target scene if the similarity exceeds the preset threshold.

[0276] In another specific implementation, the electronic device can also determine whether the first image matches the target scene by analyzing the key features of the target scene and the key features of the first image, and by analyzing whether the key features of the two frames match. Here, the key features of the target scene refer to features that accurately reflect the scene attributes of the target scene and enable the main object to be distinguished from other objects. Key features can refer to pixel features of the image modality or parametric features of the language modality.

[0277] In this embodiment, key features of each frame can be obtained during image processing. The matching degree of the images can be analyzed through key features, resulting in less interference information and higher matching accuracy.

[0278] The operation of an electronic device to determine whether a first image matches a target scene based on key features may include the following steps:

[0279] S1121: The electronic device obtains the first key feature of the target scene based on multimodal scene information.

[0280] The electronic device obtains the first key feature of the scene attribute associated with the target scene from all feature locations corresponding to the scene reference image.

[0281] S1122: The electronic device acquires the third key feature of the first image.

[0282] Similarly, the key features acquired by the electronic device from the first image are denoted as the third key features.

[0283] In practical implementation, the electronic device utilizes a multimodal large model to obtain the first key feature of the target scene and the third key feature of the first image. Specifically, the electronic device executes step S1121, which mainly includes the following process:

[0284] S11211: The electronic device inputs the scene reference image and scene prompt statement of the target scene into the multimodal large model to obtain the first feature vector.

[0285] like Figure 8 The diagram shows a partial flowchart of the image processing method performed by the electronic device. The electronic device inputs the scene reference image of the target scene and the scene prompt statement into the multimodal large model, and uses the hidden layer of the large language module of the multimodal large model to extract the first feature vector of the scene reference image.

[0286] S11212: The electronic device acquires the first key feature in the first feature vector.

[0287] Electronic devices can utilize key feature location lookup algorithms to find the location of a key feature, denoted as the first feature location. For example, electronic devices can use genetic algorithms, reinforcement learning-based search algorithms, exhaustive search, random search, fast search, greedy search strategies, etc., to find the first feature location.

[0288] Based on the mapping relationship between the feature position and the feature value recorded in the first feature vector, and the first feature position, the electronic device can extract the first key feature from the first feature vector.

[0289] The specific implementation process for electronic devices to extract the first key feature of the target scene can be found in the aforementioned implementation processes S6211-S6213, and will not be repeated here.

[0290] Correspondingly, the electronic device executes step S1122, which mainly includes the following process:

[0291] S11221: The electronic device inputs the first image and the scene prompt statement of the target scene into the multimodal large model to obtain the third feature vector.

[0292] S11222: Obtain the third key feature from the third feature vector.

[0293] The electronic device acquires the feature vector of the first image, denoted as the third feature vector. Alternatively, the electronic device can utilize a multimodal large model to acquire the third feature vector of the first image.

[0294] like Figure 8 As shown, the electronic device inputs the first image and the scene prompt statement of the target scene to be matched into the multimodal large model respectively. Using the hidden layer of the large language module of the multimodal large model, the third feature vector of the first image is extracted and the feature mean is calculated.

[0295] It should be noted that when the electronic device needs to capture multiple target scenes, it can input each frame of the first image and the scene prompts for each target scene separately into a multimodal large model to extract the third feature vector for each frame of the first image used to match each target scene. During the extraction of the third feature vector from the first image using the multimodal large model, the electronic device will consider the feature attributes indicated by the scene prompts for the target scenes, emphasizing the extraction of certain feature vectors. When matching the same frame of the first image with different target scenes, the third feature vectors obtained from the extraction of the same frame of the first image based on the scene prompts for different target scenes may not be exactly the same, mainly because different feature attributes are emphasized, which is more conducive to the matching of subsequent emphasized feature attributes.

[0296] Electronic devices can utilize key feature location lookup algorithms to find the location of key features, denoted as the third feature location. For example, electronic devices can use genetic algorithms, reinforcement learning-based search algorithms, exhaustive search, random search, fast search, greedy search strategies, etc., to find the location of key features in the first image.

[0297] In other embodiments, the electronic device may also directly use the first feature position of the scene reference image of the matched target scene as the third feature position of the first image.

[0298] Based on the third feature vector of the acquired first image, the electronic device can extract the third key feature from the third feature vector by means of the recorded mapping relationship between feature position and feature value, and the first feature position of the target scene.

[0299] S1123: The electronic device analyzes whether the first key feature of the target scene matches the third key feature of the first image.

[0300] S1124: If the first key feature of the target scene matches the third key feature of the first image, the electronic device determines that the first image is the target image.

[0301] Electronic devices can calculate the feature similarity between the first key feature of the target scene and the third key feature of the first image to determine whether the first image matches the target scene. For example, electronic devices can use parameters such as cosine similarity and Euclidean distance of each key feature as feature parameters for the corresponding feature similarity.

[0302] The electronic device acquires the feature similarity between the first key feature of the scene reference image of the target scene and the third key feature of the first image, and then determines whether the feature similarity meets the preset matching conditions to determine whether the first image matches the target scene. The electronic device can set various preset matching conditions.

[0303] In one specific implementation, the preset matching conditions set by the electronic device may include whether the feature similarity exceeds a preset similarity.

[0304] If the electronic device determines that the third key feature of the first image has a preset similarity to the first key feature of the target scene that exceeds a preset similarity, then it determines that the first image matches the target scene and uses the first image as the target image.

[0305] In one scenario, the electronic device can set the same preset similarity for different target scenes. The electronic device compares the feature similarity of each frame of the first image with the scene reference image of each target scene with the preset similarity, and determines the first image that exceeds the preset similarity as a match with the scene reference image of the target scene.

[0306] In another scenario, the electronic device can also set different preset similarity values ​​for different target scenes. If the electronic device determines that the feature similarity between the first image and a certain target scene exceeds the preset similarity value of the target scene, then it determines that the first image matches the target scene.

[0307] In another specific implementation, the preset matching conditions set by the electronic device may also include the highest or relatively high feature similarity score.

[0308] For example, an electronic device can acquire the feature similarity between several consecutive frames of the first image and the target scene, and select the first image with the highest feature similarity as the image matching the target scene. Alternatively, the electronic device can extract N consecutive first images sequentially according to the frame sequence of all first images in the first video, in a fixed-length window, and select the image that matches the target scene from the N consecutive first images extracted in each window.

[0309] Specifically, the electronic device uses the first frame with the highest feature similarity score among these N consecutive first images, or the first image of M frames, as the image to be matched with the target scene, where N and M are both positive integers, and N is greater than M.

[0310] S113: The electronic device generates a captured image; the captured image is obtained based on the target image.

[0311] The electronic device matches each frame of the first image with the target scene, and uses the first image that matches the multimodal scene information of the target scene as the target image.

[0312] In one specific implementation, after the electronic device performs image processing operations to obtain the target image, it can directly save the target image as a captured image to the photo album.

[0313] In another specific embodiment, the electronic device can also apply the image processing method provided in this embodiment to video editing scenarios, such as a "one-click blockbuster" scenario. The electronic device uses the image processing method provided in this embodiment to process video clips and extract partial target images as the highlight images corresponding to the video clips.

[0314] In one scenario, the electronic device can directly save the captured image. Later, when editing video, the captured image can be applied to the video editing process based on user actions.

[0315] In another scenario, electronic devices can directly extract snapshots from the video during the video editing process and apply the extracted snapshots directly to the next stage of video generation.

[0316] In other cases, electronic devices can also be used to automatically generate frame number labels corresponding to highlights in video data. The generated video-highlight frame data pairs can be used to train small and medium-sized models to reduce the cost of manual annotation and improve the accuracy of annotation.

[0317] The image processing method provided in this embodiment allows an electronic device to analyze a first video and extract target images that match the target scene. This saves users the trouble of manually searching for and taking screenshots of images that match the target scene, simplifies the process of extracting key images from a video, and improves the user experience. The electronic device receives a complete video at a time, allowing it to obtain all the images simultaneously.

[0318] The above embodiments explain the specific implementation process of the image processing method from the perspective of electronic devices. The following will explain the specific implementation process of the image processing method from the perspective of the internal software architecture of electronic devices.

[0319] like Figure 12 The diagram shown illustrates the internal hardware and software architecture of the electronic device. The following section will explain in detail the image processing flow of the electronic device, taking into account its internal architecture.

[0320] Specifically, the internal architecture of an electronic device can be divided into four layers, from top to bottom: the application layer (APP), the framework layer (FWK), the hardware abstraction layer (HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included, without limitation.

[0321] The application layer may include a series of application packages, such as the camera application involved in this embodiment. In addition, the application layer also includes applications such as a gallery and image processing applications with camera functionality. Application packages may also include applications such as calling, calendar, maps, navigation, music, video, and text messaging.

[0322] The framework layer provides the application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0323] The framework layer runs a camera service, which can be called by camera applications to implement shooting-related functions. In addition, the framework layer may include a window manager, content providers, a view system, a resource manager, and a notification manager. The window manager manages window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to applications. Data can include video, images, and audio. The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. The display interface can consist of one or more views. The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, notifications can be used to announce download completion or message alerts. Notifications can also appear as icons or scrollbar text in the system's top status bar, such as notifications from background applications, or as dialog windows on the screen. Notification messages can also include text messages displayed in the status bar, notification sounds, vibrations of the electrical terminal, flashing indicator lights, etc.

[0324] The kernel layer is the layer between hardware and software. The kernel layer contains at least a camera driver. This camera driver can be used to drive hardware modules with shooting capabilities, such as a camera sensor. In other words, the camera driver is responsible for data interaction with the camera sensor. The kernel layer may also include display drivers, audio drivers, sensor drivers, etc., but this embodiment does not impose any limitations on this.

[0325] The Hardware Abstraction Layer (HAL) can encapsulate drivers in the kernel layer and provide an interface for the framework layer to call them, shielding the underlying hardware implementation details. For example... Figure 12 As shown, the aforementioned hardware abstraction layer may include a camera access processing module (Camera HAL), etc. The camera access processing module is the core software framework of the camera, and it includes an interface module, a sensor access module (Sensor Node), a preview processing module, and a snapshot processing module. These interface module, sensor access module, preview processing module, and snapshot processing module are components in the image data and control command transmission pipeline within the camera access processing module.

[0326] Specifically, the sensor invocation module can be a control node oriented towards the camera sensor, which can control the camera sensor through the camera driver. The interface module can be a software interface oriented towards the application framework layer, used for data interaction with the application framework layer. Of course, the interface module can also interact with the preview processing module, the snapshot processing module, and the high-frequency enhancement processing module in the camera invocation processing module.

[0327] like Figure 12 As shown, at the application layer, the camera application can receive user-initiated capture commands, such as clicks on the capture mode control or the capture control itself. The application layer then passes the capture mode to the camera service in the framework layer.

[0328] At the framework layer, the camera service generates preview and capture requests based on the capture mode and acquires multimodal scene information of the target scene to be captured. The multimodal scene information acquired by the camera service may include scene reference images and scene cue statements, or the first key feature and key feature locations of the target scene. The camera service then sends the preview request, the capture request, and the multimodal scene information of the target scene to the interface module of the hardware abstraction layer.

[0329] At the hardware abstraction layer, the interface module instructs the sensor calling module, which in turn calls the camera sensor and camera driver to acquire raw images.

[0330] The hardware call module mainly involves the processing of two requests: the preview request and the snapshot request.

[0331] The interface module sends preview requests to the preview processing module. The preview processing module receives raw image data acquired by the camera sensor and generates a preview image. The preview processing module uploads the preview image to the camera service for image rendering processing, and then transmits it to the camera application for display.

[0332] In addition, the interface module sends the capture request and the multimodal scene information of the target scene to the capture processing module. The capture processing module also receives the preview image generated by the preview processing module and analyzes whether the preview image matches the multimodal scene information of the target scene. If the capture processing module determines that the preview image matches the multimodal scene information of the target scene, it generates a capture image based on the preview image. The capture processing module uploads the capture image to the camera service for image rendering processing, and then transmits it to the camera application for display.

[0333] The specific implementation process of the capture processing module analyzing whether the preview image matches the multimodal scene information of the target scene can be found in the aforementioned document. Figure 6 and Figure 11 The image processing steps shown will not be described in detail here.

[0334] Understandable Figure 12 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose limitations.

[0335] Furthermore, it is understood that the electronic device, in order to implement the image processing method of this embodiment, includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0336] Furthermore, embodiments of this application also provide an electronic device, including a display screen, a memory, and a processor;

[0337] The memory stores the instructions that the computer executes;

[0338] The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the image processing method provided in the above embodiments. In addition to these main components, the electronic device also includes components for implementing basic functions, which will be discussed below. Figure 13 Please provide a detailed explanation.

[0339] like Figure 13 The diagram shown is a structural schematic of an electronic device 1300 provided in an embodiment of this application. The electronic device 1300 may include a processor 1310, an external memory interface 1320, an internal memory 1321, a Universal Serial Bus (USB) interface 1330, a charging management module 1340, a power management module 1341, a battery 1342, antenna 1, antenna 2, a mobile communication module 1350, a wireless communication module 1360, an audio module 1370, a speaker, a receiver, a microphone, a headphone jack, a sensor module 1380, buttons 1390, a motor 1391, an indicator 1392, a camera 1393, a display screen 1394, and a SIM card module 1395, etc. The sensor module 1380 may include a pressure sensor 1380A, a gyroscope sensor 1380B, an ambient light sensor 1380C, a touch sensor 1380D, etc.

[0340] The structure illustrated in this embodiment of the invention does not constitute a limitation on the electronic device 1300. It may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0341] Processor 1310 may include one or more processing units. For example, processor 1310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0342] The aforementioned controller can be the decision-maker that directs the various components of the electronic device 1300 to work in a coordinated manner according to instructions. It is the nerve center and command center of the electronic device 1300. The controller generates operation control signals based on the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0343] The processor 1310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1310 is a cache memory, which can store instructions or data that the processor 1310 has just used or that are used repeatedly. If the processor 1310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1310, and thus improves the efficiency of the system.

[0344] In some embodiments, the processor 1310 may include interfaces. These interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0345] The interface connection relationships between the modules illustrated in this embodiment of the invention are merely illustrative and do not constitute a structural limitation on the electronic device 1300. The electronic device 1300 may employ different interface connection methods or a combination of multiple interface connection methods as described in this embodiment of the invention.

[0346] The charging management module 1340 can be a rechargeable battery or a disposable battery. If it is a rechargeable battery, it can receive charging input through a charger. The power management module 1341 is used to connect the battery 1342, the charging management module 1340, and the processor 1310. The power management module 1341 receives input from the battery 1342 and / or the charging management module 1340 to power the processor 1310, memory 1321, external memory interface 1320, display 1394, camera 1393, and wireless communication module 1360, etc.

[0347] The wireless communication function of electronic device 1300 can be realized through antenna 1, antenna 2, mobile communication module 1350, wireless communication module 1360, modem and baseband processor, etc.

[0348] Electronic device 1300 implements display functions through a GPU, a display screen 1394, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1394 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 1310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0349] Display screen 1394 is used to display images, videos, etc. Display screen 1394 includes a display panel. The display panel may be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), Minied, MicroLED, Micro-OLED, Quantum Dot Light-Emitting Diodes (QLED), etc. In some embodiments, electronic device 1300 may include one or N displays 1394, where N is a positive integer greater than 1.

[0350] Electronic device 1300 can perform shooting functions through ISP, camera 1393, video codec, GPU, display and application processor.

[0351] The ISP (Image Signal Processor) is used to process data fed back from the camera 1393. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set within the camera 1393.

[0352] Camera 1393 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 1300 may include one or N cameras 1393, where N is a positive integer greater than 1.

[0353] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 1300 is selecting a frequency point, the DSP is used to perform Fourier transforms on the frequency energy.

[0354] Video codecs are used to compress or decompress digital video. Electronic device 1300 can support one or more video codecs. Thus, electronic device 1300 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0355] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0356] The external storage interface 1320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1300. The external memory card communicates with the processor 1310 through the external storage interface 1320 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0357] Internal memory 1321 can be used to store executable program code, including instructions. Processor 1310 executes various functional applications and data processing of electronic device 1300 by running the instructions stored in internal memory 1321. Memory 1321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 1300 (such as audio data, phonebook, etc.). Furthermore, memory 1321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, other volatile solid-state storage devices, Universal Flash Storage (UFS), etc.

[0358] Electronic device 1300 can implement audio functions, such as music playback and recording, through audio module 1370, speaker, receiver, microphone, headphone jack, and application processor.

[0359] The audio module 1370 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 1370 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 1370 may be located in the processor 1310, or some functional modules of the audio module 1370 may be located in the processor 1310.

[0360] Indicator 1392 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0361] The SIM card module 1395 is used to realize the communication function of the SIM card. The SIM card module 1395 may include a SIM card interface, a SIM card circuit and related auxiliary devices.

[0362] The image processing methods described in the foregoing embodiments can all be implemented in the electronic device 1300 having the above-described hardware structure.

[0363] Based on the above embodiments, this application also provides an image processing apparatus, which includes a processor for executing the image processing method provided in the above embodiments.

[0364] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method provided in the above embodiments.

[0365] This application also provides a computer program product containing instructions that, when run on a computer, enable the computer to perform the image processing method provided in the above embodiments.

[0366] The specific implementation methods and technical effects of the electronic devices, computer-readable storage media, and computer program products containing instructions provided in this application can be found in the specific implementation process and technical effects of the image processing method provided in the foregoing embodiments, which will not be repeated here.

[0367] In some embodiments, as described above, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the functional modules described above is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0368] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0369] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0370] The above are merely specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An image processing method, characterized in that, The image processing method, applied to electronic devices, includes: The shooting preview interface is used to display a preview stream, which includes preview images displayed frame by frame. A target image is acquired; wherein the target image is a preview image of multimodal scene information matching a target scene; the multimodal scene information includes a scene reference image of the image modality and a scene prompt statement of the language modality, the scene prompt statement being used to describe the scene attributes of the target scene; the matching of the target image with the multimodal scene information of the target scene includes: the feature similarity between the second key feature of the target image and the first key feature of the target scene satisfies a preset matching condition; the first key feature of the target scene is obtained based on the scene reference image and the scene prompt statement of the target scene; Generate a captured image; the captured image is obtained based on the target image.

2. The image processing method according to claim 1, characterized in that, The step of acquiring the target image includes: Based on the multimodal scene information, the first key feature of the target scene is obtained; Obtain the second key feature of the preview image; Analyze whether the first key feature of the target scene matches the second key feature of the preview image; If the first key feature of the target scene matches the second key feature of the preview image, the preview image is determined to be the target image.

3. The image processing method according to claim 2, characterized in that, The step of obtaining the first key feature of the target scene based on the multimodal scene information includes: Based on the scene prompt statement, obtain the first feature vector of the scene reference image; Obtain the first key feature from the first feature vector.

4. The image processing method according to claim 3, characterized in that, The step of obtaining the first feature vector of the scene reference image based on the scene prompt statement includes: The scene reference image and the scene prompt statement are input into the multimodal large model to obtain the first feature vector.

5. The image processing method according to any one of claims 3 to 4, characterized in that, The step of obtaining the second key feature of the preview image includes: Obtain the second feature vector of the preview image; Obtain the second key feature from the second feature vector.

6. The image processing method according to claim 5, characterized in that, The step of obtaining the second feature vector of the preview image includes: The preview image and the scene prompt statement are input into the multimodal large model to obtain the second feature vector.

7. The image processing method according to claim 5, characterized in that, The step of obtaining the first key feature in the first feature vector includes: A key feature location search algorithm is used to find the first feature location associated with the scene attribute of the target scene from all feature locations of the first feature vector; wherein, the key feature location search algorithm includes a genetic algorithm; Extract the first key feature; wherein the first key feature is the feature corresponding to the first feature position in the first feature vector.

8. The image processing method according to claim 5, characterized in that, The step of obtaining the second key feature in the second feature vector includes: Using a key feature location search algorithm, the second feature location associated with the scene attribute of the target scene is found from all feature locations of the second feature vector; wherein, the key feature location search algorithm includes a genetic algorithm; Extract the second key feature; wherein the second key feature is the feature corresponding to the second feature position in the second feature vector.

9. The image processing method according to claim 7, characterized in that, The step of obtaining the second key feature in the second feature vector includes: Extract the second key feature corresponding to the first feature position from the second feature vector.

10. The image processing method according to any one of claims 2 to 4, characterized in that, The step of analyzing whether the first key feature of the target scene matches the second key feature of the preview image includes: Calculate the feature similarity between the first key feature and the second key feature; If the feature similarity meets a preset matching condition, the first key feature of the target scene is determined to match the second key feature of the preview image; the preset matching condition includes the feature similarity exceeding a preset similarity.

11. The image processing method according to claim 10, characterized in that, The feature similarity includes at least one of cosine similarity and Euclidean distance.

12. The image processing method according to any one of claims 1 to 4, characterized in that, Prior to the step of acquiring the target image, the image processing method further includes: In response to the first operation, a snapshot scene tab is displayed; the snapshot scene tab includes at least one scene option control for a first scene; the shooting preview interface includes a snapshot control, and the first operation is a click operation performed on the snapshot control; In response to a second operation on the scene option control in the capture scene tab, the target scene is determined; the target scene is the first scene corresponding to the scene option control affected by the second operation. Obtain the multimodal scene information of the target scene.

13. The image processing method according to claim 12, characterized in that, The step of acquiring the multimodal scene information of the target scene includes: An input interface that displays the multimodal scene information of the target scene, the input interface including an input box for the image mode and an input box for the language mode; The system receives an image input from the image modality input box and a statement input from the language modality input box; wherein the image input from the image modality input box serves as the scene reference image, and the statement input from the language modality input box serves as the scene prompt statement.

14. The image processing method according to claim 12, characterized in that, The step of acquiring the multimodal scene information of the target scene includes: A query request is sent to the server; the query request is used to request the multimodal scene information of the target scene; Receive the multimodal scene information of the target scene sent by the server.

15. The image processing method according to claim 12, characterized in that, The step of acquiring the multimodal scene information of the target scene includes: Obtain the multimodal scene information of the target scene stored in local memory.

16. The image processing method according to any one of claims 1 to 4, characterized in that, The step of generating the captured image includes: The target image is used as the captured image.

17. An image processing method, characterized in that, The image processing method, applied to electronic devices, includes: Acquire a first video; the first video includes multiple consecutive frames of a first image; A target image is acquired; wherein the target image is a first image that matches multimodal scene information of a target scene; the multimodal scene information includes a scene reference image of the image modality and a scene prompt statement of the language modality, the scene prompt statement being used to describe the scene attributes of the target scene; the matching of the target image with the multimodal scene information of the target scene includes: the feature similarity between the second key feature of the target image and the first key feature of the target scene satisfies a preset matching condition; the first key feature of the target scene is acquired based on the scene reference image and the scene prompt statement of the target scene; Generate a captured image; the captured image is obtained based on the target image.

18. The image processing method according to claim 17, characterized in that, The step of acquiring the target image includes: Based on the multimodal scene information, the first key feature of the target scene is obtained; Obtain the third key feature of the first image; Analyze whether the first key feature of the target scene matches the third key feature of the first image; If the first key feature of the target scene matches the third key feature of the first image, the first image is determined to be the target image.

19. The image processing method according to claim 18, characterized in that, The step of obtaining the first key feature of the target scene based on the multimodal scene information includes: Input the scene reference image and the scene prompt statement into the multimodal large model to obtain the first feature vector; Obtain the first key feature from the first feature vector; The step of obtaining the third key feature of the first image includes: Input the first image and the scene prompt statement into the multimodal large model to obtain the third feature vector; Obtain the third key feature from the third feature vector.

20. An electronic device, characterized in that, The electronic device includes a display screen, a memory, and a processor, wherein the display screen and the memory are both coupled to the processor; The memory stores computer-executed instructions; The processor performs the image processing method according to any one of claims 1 to 19.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the image processing method as described in any one of claims 1 to 19.

22. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 19.

Images