Image processing method, electronic device and cloud server

Abstract

The application discloses an image processing method, an electronic device and a cloud server, and belongs to the technical field of artificial intelligence. The method is executed by the electronic device, and the method comprises the following steps: obtaining identity recognition information of a first object in a first image and image description information of the first image; outputting object description information of the first image based on the identity recognition information and the image description information; the object description information is used for describing the identity and visual attribute information of the first object; wherein the identity recognition information is obtained by performing identity recognition on the first object by the electronic device; and the image description information is obtained by performing image recognition on the first image by the cloud server.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to an image processing method, an electronic device, and a cloud server. Background Technology

[0002] With the rapid iteration of artificial intelligence technology, multimodal models have acquired the ability to understand image content and interact with it using natural language. They can be applied to various scenarios to provide convenience for users: for example, in assistive scenarios for the visually impaired, they can be used to describe the environment and provide video question answering, offering environmental information to visually impaired individuals.

[0003] However, most multimodal models are currently deployed on cloud servers, so images must be uploaded to these servers for interpretation. The image descriptions obtained in this way typically only include objective descriptions of the image, which are insufficient and cannot meet user needs. Another approach involves uploading the identity information of multiple objects to the cloud server. This allows the cloud server to combine the uploaded object identity information with the image interpretation to obtain object descriptions that include the object identities. However, this method carries the risk of privacy breaches. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, electronic device, cloud server, storage medium, and program product that can reduce the risk of privacy leakage in image processing and extract richer information from images.

[0005] In a first aspect, embodiments of this application provide an image processing method, executed by an electronic device, the method comprising: Obtain the identification information of the first object in the first image and the image description information of the first image; Based on identity recognition information and image description information, output object description information of the first image; the object description information is used to describe the identity and visual attribute information of the first object. The identity recognition information is obtained by identifying the first object through an electronic device; the image description information is obtained by image recognition of the first image through a cloud server.

[0006] The above scheme involves using an electronic device to identify a first object in a first image, obtaining the object's identification information. A cloud server then performs image recognition on the first image to obtain the object's image description information. Finally, based on the object's identification information and the image description information, object description information for the first image is generated and output through the electronic device. According to this embodiment, object identification is performed locally on the electronic device, eliminating the need to upload an object feature library for identification to a cloud server, thus reducing the risk of object feature privacy leakage. Furthermore, the final object description information includes not only objective image description information but also the object's identification information, resulting in richer output information.

[0007] As one possible implementation, before outputting the object description information of the first image based on identity recognition information and image description information, the method further includes: The first object is identified to obtain its identification information. Send the first image to the cloud server; Receive image description information of the first image sent by the cloud server; Based on the identity recognition information and the image description information of the first image, the object description information of the first image is generated.

[0008] With the above solution, only a small amount of data interaction is needed between the electronic device and the cloud server to generate object description information for the first image, thus improving data transmission efficiency.

[0009] As one possible implementation, before outputting the object description information of the first image based on identity recognition information and image description information, the method further includes: The first object is identified to obtain its identification information. Send the first image and identification information to the cloud server; Receive object description information of the first image sent by the cloud server.

[0010] With the above solution, electronic devices only need to perform identity recognition, thereby reducing the consumption of computing resources on electronic devices. In addition, generating object description information through cloud servers with abundant computing resources improves generation efficiency.

[0011] As one possible implementation, the first object is identified to obtain its identification information, including: Extract the first object feature vector of the first object from the first image; Based on the first object feature vector, a second object feature vector with a similarity greater than a similarity threshold is determined from the pre-entered object identity information database; the object identity information database includes object feature vectors and object identity information associated with the object feature vectors. The object identity information associated with the feature vector of the second object is determined as the identity recognition information of the first object.

[0012] Using the above methods, object identification can be completed quickly based on the entered object identity information database.

[0013] As one possible implementation, where the first object is a person, the method further includes, before extracting the first object feature vector from the first image: If the first image includes at least one face, extract the face image corresponding to each face from the first image to obtain at least one face image; From at least one face image, select face images that meet the first image selection criteria as the face images to be identified; From the first image, extract the first object feature vector of the first object, including: Extract facial feature information from the image of the face to be identified; The facial feature information is used to determine the first object feature vector of the first object.

[0014] The above methods can yield high-quality facial images for identity recognition, thereby improving the accuracy of identity recognition.

[0015] As one possible implementation, the first image filtering criteria include at least one of the following: The resolution of the face image is greater than the first resolution threshold; The blurriness of the face image is less than the first blurriness threshold.

[0016] As one possible implementation, the method also includes: Receive the user's first input when selecting a second image from the album interface; In response to the first input, the feature vector of the third object of the second object is extracted from the second image; Displays the interface for entering the object's identity information; Receive the object identity information of the second object entered by the user on the object identity information entry interface; The feature vector of the third object is associated with the object identity information of the second object and stored in the object identity information database.

[0017] Using the above method, you can quickly enter the identity information of an individual based on images in your photo album, even if the individual is not physically present.

[0018] As one possible implementation, the second object is a person, and in response to the first input, a third object feature vector of the second object is extracted from the second image, including: The faces and facial landmarks in the second image are detected to obtain the second face detection result and the first facial landmark detection result; the second face detection result includes at least one candidate face image, and the first facial landmark detection result includes the facial landmark coordinates of each candidate face image; If the second face detection result indicates that the second image includes a face, face alignment is performed on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one first aligned image. From at least one first aligned image, select the face image of the second object; from the face image of the second object, extract the third object feature vector of the second object.

[0019] The above method can accurately extract the object feature vector of the second object.

[0020] As one possible implementation, selecting the face image of the second object from at least one first aligned image includes: if the number of first aligned images is one, using the first aligned image as the face image of the second object; If the number of first aligned images is at least two, and the area ratio of the first candidate image to the second candidate image in the at least two first aligned images is greater than or equal to a ratio threshold, the first candidate image is used as the face image of the second object. The first candidate image is an image with an area greater than a first threshold among at least two first aligned images; the second candidate image (face) is an image with an area greater than a second threshold among at least two first aligned images; the first threshold is greater than the second threshold.

[0021] The above method can accurately identify the facial image of the second object, avoiding information entry errors.

[0022] As one possible implementation, before extracting the feature vector of the third object from the face image of the second object, the method further includes: Determine whether the face image of the second object meets the second image filtering criteria; From the face image of the second object, extract the feature vector of the third object, including: If the face image of the second object meets the second image filtering criteria, the feature vector of the third object of the second object is extracted from the face image of the second object.

[0023] The above methods can improve the quality and accuracy of the final obtained third-object feature vector.

[0024] As one possible implementation, the second image filtering criteria include at least one of the following: The area of ​​the face image in the original image is greater than the first proportion threshold and less than the second proportion threshold; The pose information of the face in the face image satisfies the pose condition; The blurriness of the face image is less than the second blurriness threshold.

[0025] As one possible implementation, the method also includes: Displays the shooting preview interface; Receive voice information input by the user; When the voice information is intended to record object identity information, the object identity information of the third object is extracted from the voice information. Extract the feature vector of the fourth object from the preview image displayed on the shooting preview interface; The fourth object feature vector of the third object and the object identity information of the third object are associated and stored in the object identity information database.

[0026] The above methods allow for the convenient input of an individual's identity information via video Q&A.

[0027] As one possible implementation, the third object is a person. From the preview image displayed on the shooting preview interface, the feature vector of the fourth object of the third object is extracted, including: The faces and facial landmarks in the preview image are detected to obtain the third face detection result and the second facial landmark detection result; the second face detection result includes at least one candidate face image, and the second facial landmark detection result includes the facial landmark coordinates of each candidate face image; If the third face detection result indicates that the preview image includes a face, face alignment is performed on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one second aligned image. From at least one second-aligned image, select the face image of the third object; Extract the feature vector of the fourth object from the face image of the third object.

[0028] The above method can accurately extract the feature vector of the object to be entered.

[0029] Secondly, embodiments of this application provide an image processing method executed by a cloud server, the method comprising: Receive a first image and object information of a first object in the first image sent by an electronic device, wherein the object information includes the identification information of the first object; Based on the first image and object information, generate object description information for the first image; Send object description information to electronic devices.

[0030] The above scheme involves using an electronic device to identify a first object in a first image, obtaining the object's identification information. A cloud server then performs image recognition on the first image to obtain the object's image description information. Finally, based on the object's identification information and the image description information, object description information for the first image is generated and output through the electronic device. According to this embodiment, object identification is performed locally on the electronic device, eliminating the need to upload an object feature library for identification to a cloud server, thus reducing the risk of object feature privacy leakage. Furthermore, the final object description information includes not only objective image description information but also the object's identification information, resulting in richer output information.

[0031] As one possible implementation, the number of first objects is one. Based on the first image and object information, object description information of the first image is generated, including: Based on the object information, generate the object identity semantic information of the first object; The object identity semantic information and the first image are input into the multimodal model to obtain the object description information output by the multimodal model.

[0032] Through the above method, the cloud server employs object identity semantic injection to invoke the multimodal model. In this way, the multimodal model in the cloud server only receives the semantic conclusion of the object's identity, without accessing the original object identity features or being able to infer the object's identity features in reverse. This resolves the privacy leakage risk associated with storing object identity features on the cloud server.

[0033] As one possible implementation, there are multiple first objects, and the identity information also includes the pixel coordinates of each first object in the first image; Based on the first image and object information, generate object description information for the first image, including: Based on the pixel coordinates of each first object, determine the image region of each first object in the first image; In the first image, a label is added to the image region of each first object to obtain an object index map; wherein, the labeling pattern of the labels for different image regions is different; Based on the identity information and corresponding tagging pattern of each first object, generate the object identity semantic information of each first object; The object identity semantic information and object index graph of each first object are input into the multimodal model to obtain the object description information output by the multimodal model.

[0034] By using the above method, it can be ensured that the object description information of the first image is accurately generated when the first image contains multiple first objects.

[0035] Thirdly, embodiments of this application provide an electronic device, including: The acquisition module is used to acquire the identity information of the first object in the first image and the image description information of the first image; The output module is used to output object description information of the first image based on identity recognition information and image description information; the object description information is used to describe the identity and visual attribute information of the first object. The identity recognition information is obtained by identifying the first object through an electronic device; the image description information is obtained by image recognition of the first image through a cloud server.

[0036] As one possible implementation, the electronic device also includes: The processing module is used to perform identity recognition on the first object before outputting the object description information of the first image based on the identity recognition information and image description information, so as to obtain the identity recognition information of the first object; The communication module is used to send the first image to the cloud server; The communication module is also used to receive image description information of the first image sent by the cloud server; The processing module is also used to generate object description information of the first image based on the identity recognition information and the image description information of the first image.

[0037] As one possible implementation, the electronic device also includes: The processing module is used to perform identity recognition on the first object before outputting the object description information of the first image based on the identity recognition information and image description information, so as to obtain the identity recognition information of the first object; The communication module is used to send the first image and identification information to the cloud server; The communication module is also used to receive object description information of the first image sent by the cloud server.

[0038] As one possible implementation, the processing module is specifically used for: Extract the first object feature vector of the first object from the first image; Based on the first object feature vector, a second object feature vector with a similarity greater than a similarity threshold is determined from the pre-entered object identity information database; the object identity information database includes object feature vectors and object identity information associated with the object feature vectors. The object identity information associated with the feature vector of the second object is determined as the identity recognition information of the first object.

[0039] As one possible implementation, the first object is a person, and the processing module is also used for: Before extracting the first object feature vector of the first object from the first image, if the first image includes at least one face, extract the face image corresponding to each face from the first image to obtain at least one face image; From at least one face image, select face images that meet the first image selection criteria as the face images to be identified; From the first image, extract the first object feature vector of the first object, including: Extract facial feature information from the image of the face to be identified; The facial feature information is used to determine the first object feature vector of the first object.

[0040] As one possible implementation, the first image filtering criteria include at least one of the following: The resolution of the face image is greater than the first resolution threshold; The blurriness of the face image is less than the first blurriness threshold.

[0041] As one possible implementation, the electronic device also includes: The receiving module is used to receive the first input from the user selecting a second image from the album interface; The processing module is also configured to extract the third object feature vector of the second object from the second image in response to the first input; The display module is used to display the object identity information input interface; The receiving module is also used to receive the object identity information of the second object entered by the user on the object identity information input interface; The storage module is used to associate and store the feature vector of the third object and the object identity information of the second object into the object identity information database.

[0042] As one possible implementation, the second object is a character, and the processing module is specifically used for: The faces and facial landmarks in the second image are detected to obtain the second face detection result and the first facial landmark detection result; the second face detection result includes at least one candidate face image, and the first facial landmark detection result includes the facial landmark coordinates of each candidate face image; If the second face detection result indicates that the second image includes a face, face alignment is performed on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one first aligned image. Select the face image of the second object from at least one first aligned image; Extract the feature vector of the third object from the face image of the second object.

[0043] As one possible implementation, the processing module is specifically used for: If there is only one first aligned image, the first aligned image is used as the face image of the second object; If the number of first aligned images is at least two, and the area ratio of the first candidate image to the second candidate image in the at least two first aligned images is greater than or equal to a ratio threshold, the first candidate image is used as the face image of the second object. The first candidate image is an image with an area greater than a first threshold among at least two first aligned images; the second candidate image (face) is an image with an area greater than a second threshold among at least two first aligned images; the first threshold is greater than the second threshold.

[0044] As one possible implementation, the processing module is also used for: Before extracting the feature vector of the third object from the face image of the second object, it is determined whether the face image of the second object meets the second image screening conditions; If the face image of the second object meets the second image filtering criteria, the feature vector of the third object of the second object is extracted from the face image of the second object.

[0045] As one possible implementation, the second image filtering criteria include at least one of the following: The area of ​​the face image in the original image is greater than the first proportion threshold and less than the second proportion threshold; The pose information of the face in the face image satisfies the pose condition; The blurriness of the face image is less than the second blurriness threshold.

[0046] As one possible implementation, the electronic device also includes: The display module is used to display the shooting preview interface; The receiving module is used to receive voice information input by the user; The processing module is also used to extract the object identity information of a third object from the voice information when the voice information has the intention to input object identity information; The processing module is also used to extract the fourth object feature vector of the third object from the preview image displayed on the shooting preview interface; The storage module is used to associate and store the fourth object feature vector of the third object and the object identity information of the third object in the object identity information database.

[0047] As one possible implementation, the third object is a character, and the processing module is specifically used for: The faces and facial landmarks in the preview image are detected to obtain the third face detection result and the second facial landmark detection result; the second face detection result includes at least one candidate face image, and the second facial landmark detection result includes the facial landmark coordinates of each candidate face image; If the third face detection result indicates that the preview image includes a face, face alignment is performed on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one second aligned image. From at least one second-aligned image, select the face image of the third object; Extract the feature vector of the fourth object from the face image of the third object.

[0048] Fourthly, embodiments of this application provide a cloud server, including: The communication module is used to receive a first image and object information of a first object in the first image sent by an electronic device. The object information includes the identification information of the first object. The processing module is used to generate object description information of the first image based on the first image and object information; The communication module is also used to send object description information to electronic devices.

[0049] As one possible implementation, the first object has one instance, and the processing module is specifically used for: Based on the object information, generate the object identity semantic information of the first object; The object identity semantic information and the first image are input into the multimodal model to obtain the object description information output by the multimodal model.

[0050] As one possible implementation, there are multiple first objects, and the identity information also includes the pixel coordinates of each first object in the first image; The processing module is specifically used for: Based on the pixel coordinates of each first object, determine the image region of each first object in the first image; In the first image, a label is added to the image region of each first object to obtain an object index map; wherein, the labeling pattern of the labels for different image regions is different; Based on the identity information and corresponding tagging pattern of each first object, generate the object identity semantic information of each first object; The object identity semantic information and object index graph of each first object are input into the multimodal model to obtain the object description information output by the multimodal model.

[0051] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first or second aspect.

[0052] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0053] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the first or second aspect.

[0054] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first or second aspect.

[0055] Understandably, the electronic device provided in the third aspect, the cloud server provided in the fourth aspect, the electronic device provided in the fifth aspect, the readable storage medium provided in the sixth aspect, the chip provided in the seventh aspect, and the computer program product provided in the eighth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0056] Figure 1 This is a schematic flowchart of an image processing method provided in some embodiments of this application; Figure 2 These are schematic diagrams of first images provided in some embodiments of this application; Figure 3 These are schematic diagrams of first images provided in some embodiments of this application; Figure 4 This is a schematic diagram of an object input interface provided in some embodiments of this application; Figure 5 These are schematic diagrams of second images provided in some embodiments of this application; Figure 6 This is a schematic diagram of an object identity information input interface provided in some embodiments of this application; Figure 7 These are schematic diagrams of the shooting preview interface provided in some embodiments of this application; Figure 8 These are schematic diagrams illustrating the interaction between an electronic device and a cloud server, provided in some embodiments of this application. Figure 9 These are schematic diagrams illustrating the interaction between an electronic device and a cloud server, provided in other embodiments of this application. Figure 10 These are schematic diagrams of electronic devices provided in some embodiments of this application; Figure 11 These are schematic diagrams of cloud servers provided in some embodiments of this application; Figure 12 These are block diagrams illustrating electronic devices provided in some embodiments of this application; Figure 13 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0058] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0059] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained. The terminology used in the implementation section of this application is only used to explain the specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application will be explained below.

[0060] Mobile terminal: refers to mobile phones or other terminal devices with local computing capabilities; Cloud server multimodal model: A general multimodal model deployed on a cloud server that can jointly process visual and textual information; Facial recognition: refers to the process by which users create a local identity profile for a specific person through image capture and semantic input on the terminal side; Feature vector: A high-dimensional vector representation output by the terminal-side face feature extraction model, used for person comparison; the feature is an irreversible representation. Identity semantic information: refers to non-biological semantic information related to a person's identity, such as name and social relationships; Video Q&A: refers to a multimodal interactive question-and-answer process based on real-time video streams.

[0061] The purpose of this application is to provide an image processing solution based on edge-cloud collaboration, which enables the understanding of images by performing identity detection of people and other objects on the mobile terminal without storing or inferring the user's biometric features on the cloud server.

[0062] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0063] The image processing method provided in this application can be applied to object recognition scenarios. One specific application scenario is recognizing people captured in real-time by a user through the camera of an electronic device. Another specific application scenario is recognizing people displayed in an application while the user is browsing it. The image processing method provided in this application will be described in detail below with reference to the accompanying drawings. It should be noted that the image processing method provided in this application can be executed by an image processing system, which includes an electronic device and a cloud server. This application uses an image processing system executing the image processing method as an example to illustrate the image processing method provided in this application.

[0064] See Figure 1 The above is a flowchart illustrating the image processing method provided in some embodiments of this application, such as... Figure 1 As shown, the method includes steps 110-120, which will be explained in detail below.

[0065] Step 110. The electronic device acquires the identification information of the first object in the first image and the image description information of the first image.

[0066] In some embodiments of this application, the first image is the image that needs to be processed in this application.

[0067] In some embodiments of this application, the first image may be an image displayed on an electronic device. For example, the first image may be an image displayed in any application on the electronic device. Applications may include, but are not limited to, photo album apps, gallery apps, camera apps, image processing software, browsers, social media apps, etc.

[0068] For example, after a user opens image processing software on their electronic device, they process images in the device's photo album app. While the user is browsing the photo album app, the image processing software can capture images displayed in the app through methods such as screenshots, and then use these captured images as the first image for subsequent image processing. For example, such as... Figure 2 As shown, if the user is browsing image 200 in the photo album app, then image 200 can be selected as the first image.

[0069] In some embodiments of this application, the first image may be a preview image captured by the camera of an electronic device. A preview image refers to an image captured in real time by the camera of an electronic device.

[0070] For example, a user opens image processing software on an electronic device. Upon startup, the software automatically invokes the camera preview interface, displaying the live feed captured by the camera on the software page. In this case, a frame from the real-time camera preview can be captured as the first image according to a preset frame-skipping interval. For example, such as... Figure 3 As shown, the software page displays a camera preview interface 300, which can capture the image displayed in real time as the first image. The frame extraction interval can be set according to actual needs; for example, the frame extraction interval can be set to 1 second, meaning that one frame is captured from the camera preview interface 300 every second as the first image. In some embodiments of this application, the first object is an object contained in the first image. Objects may include, but are not limited to, people, animals, plants, vehicles, and items. The number of first objects in the first image can be one or more. For example, Figure 2 The first object in the first image 200 shown is a person, and there is only one such first object. For example, Figure 3 The first object in the first image 310 is also a person, and there are two first objects.

[0071] In some embodiments of this application, in order to facilitate users' understanding of the first image, after obtaining the first image, image processing is performed on the first image to obtain image description information of the first image and identity recognition information of the first object contained in the first image.

[0072] In some embodiments of this application, the image description information of the first image is information used to describe the visual attributes in the first image. Visual attributes refer to the semantically meaningful, human-understandable and nameable visual features of an image or a specific region in an image, such as an object or scene, such as color, shape, texture, type, posture, clothing, etc.

[0073] For example, for such Figure 2 The first image 200 shown may be described as "a person wearing a blue denim jacket and black trousers".

[0074] For example, for such Figure 3 The first image shown can be described as "There are two people, one wearing a blue denim jacket and black trousers, and the other wearing a red jacket and black trousers".

[0075] In some embodiments of this application, the identity information of the first object is information used to describe the identity of the first object. This identity information may include at least one of the following: object name, and the relationship between the object and the user. The relationship between the object and the user includes, but is not limited to, social relationships and ownership relationships. Social relationships include, but are not limited to, friends, spouse, husband, father, mother, daughter, son, etc., while ownership relationships include, but are not limited to, the user's pet, vehicles registered under the user's name, etc.

[0076] In some embodiments of this application, in order to avoid the leakage of privacy information caused by identifying the identity information of an object through a cloud server, the electronic device can perform identity recognition on the first object in the first image locally, thereby obtaining the identity recognition information of the first object.

[0077] In some embodiments of this application, considering the limited local resources and computing power of the electronic device, in order to improve image processing efficiency, the first image can be image-recognized by a cloud server to obtain the image description information of the first image.

[0078] By using the aforementioned edge-cloud collaboration approach, the risk of object feature privacy leakage can be reduced while improving image processing efficiency.

[0079] Step 120. Based on the identity recognition information and image description information, output the object description information of the first image.

[0080] In some embodiments of this application, after obtaining the image description information of the first image and the identification information of the first object, object description information of the first image is generated based on the aforementioned image description information and identification information, and then output to the user. The object description information generated based on the image description information and identification information of the first image can not only describe the visual attribute information displayed in the first image, but also describe the identification information of objects not displayed in the first image, thereby providing the user with richer information. This facilitates the user in identifying objects, understanding relationships, and participating in interactions based on the output information.

[0081] For example, based on such Figure 2 The object description information generated from the first image 200 shown may include "Friend Xiaoming is wearing a blue denim jacket and black trousers".

[0082] Based on such Figure 3 The object description information generated from the first image shown may include "There are two people in front of you, namely Xiaoming, who is wearing a blue denim jacket and black trousers, and Xiaotian, who is wearing a red jacket and black trousers."

[0083] In some embodiments of this application, the method of outputting object description information of the first image may include, but is not limited to: outputting object description information by voice broadcasting, or outputting object description information by text.

[0084] For example, an electronic device can invoke a TTS (Text To Speech) module to convert object description information into speech and read it aloud. Alternatively, an electronic device can invoke a display module to convert object description information into text and display it on the screen.

[0085] In this embodiment, an electronic device identifies a first object in a first image to obtain the object's identification information. A cloud server then performs image recognition on the first image to obtain the object's image description information. Finally, based on the object's identification information and the image description information, object description information for the first image is generated and output through the electronic device. According to this embodiment, object identification is performed locally on the electronic device, eliminating the need to upload an object feature library for identification to a cloud server, thus reducing the risk of object feature privacy leakage. Furthermore, the final object description information includes not only objective image description information but also the object's identification information, resulting in richer output information.

[0086] In some embodiments of this application, the image processing method supports a question-and-answer mode. In this mode, before step 110 above, user-inputted query information can be obtained first, and step 110 above can be executed in response to the query information.

[0087] For example, in question-and-answer mode, when a user raises their electronic device to take a picture of a friend and asks "Who is the person in the picture?", the first object to be identified can be determined based on this question information. At this time, the image processing software can acquire the real-time image of the picture as the first image, take the person in the first image as the first object, perform image processing on the first image, and finally output object description information such as "He is your friend Xiaoming, wearing a white T-shirt, a blue denim jacket and black trousers."

[0088] In some embodiments of this application, an object identity information database can be pre-set in the electronic device. The electronic device can then identify the first object based on this database, thereby obtaining the object's identity information. The object identity information database includes an object feature vector and object identity information associated with that feature vector. Based on this, the electronic device can identify the first object through steps 210-230.

[0089] Step 210. Extract the first object feature vector of the first object from the first image.

[0090] In some embodiments of this application, the first object feature vector of the first object can be extracted from the first image using image feature extraction technology.

[0091] In some embodiments of this application, when extracting the first object feature vector of the first object from the first image, the first image can be segmented to obtain a local image containing the first object, and then feature extraction can be performed on the local image, and the extracted feature vector can be used as the first object feature vector of the first object.

[0092] In some embodiments of this application, considering that not all objects in the first image are the objects to be identified by the user, or that not all objects in the first image meet the identification conditions, before extracting the first object feature vector, the local images of each object in the first image are first filtered, and the local images that meet the image filtering conditions are selected as the local images corresponding to the first object to be identified, and then feature extraction is performed. The image filtering conditions can be set according to actual needs. For example, image filtering may include restrictions on information such as object size, clarity, and pose. Different types of objects can correspond to different image filtering conditions.

[0093] In some embodiments of this application, when the first object to be identified in the first image is a person, image filtering can be performed through the following steps 310-320.

[0094] Step 310. If the first image includes at least one face, extract the face image corresponding to each face from the first image to obtain at least one face image.

[0095] In some embodiments of this application, a face detection model can be used to perform face detection on the first image, thereby obtaining the first face detection result corresponding to the first image.

[0096] In some embodiments of this application, if a face is detected in the first image, the first face detection result may include the face detection bounding box information (x, y, w, h) for each face and the confidence level of the corresponding face detection bounding box. Among them, confidence level This refers to the certainty score given by a face detection model when it detects a face in a region of an image. The score is a value between 0 and 1, reflecting the model's confidence that the region indeed contains a valid face. A higher confidence score indicates greater certainty that the model has detected a real face; a lower confidence score indicates a higher likelihood of a false detection.

[0097] In some embodiments of this application, if a face is not detected in the first image, the first face detection result is empty.

[0098] In some embodiments of this application, a lightweight object detection model (such as YOLOv8) can be used as a detector to perform face detection on the first image.

[0099] In some embodiments of this application, one face detection box in the first face detection result corresponds to one face. Therefore, the number of face detection boxes in the first face detection result is taken as the number of faces contained in the first image, thereby determining whether the first image contains a face. If it is determined that the first image contains at least one face, based on the face detection box information of each face, a partial image within the face detection box of each face is extracted from the first image as the face image of the face, resulting in at least one face image for subsequent image filtering.

[0100] Step 320. From at least one face image, select face images that meet the first image selection criteria as face images to be identified.

[0101] In some embodiments of this application, the first image filtering condition is a pre-set condition for filtering out face images whose quality meets the requirements for identity recognition. After obtaining at least one face image, the quality index information of each face image is acquired, and the quality index information of each face image is compared with the set first image filtering condition, thereby filtering out face images that meet the first image filtering condition as face images to be recognized. The quality index information includes, but is not limited to, resolution, blurriness, etc.

[0102] In some embodiments of this application, the first image filtering condition includes at least one of the following: The resolution of the face image is greater than the resolution threshold; The blurriness of the face image is less than the first blurriness threshold.

[0103] The resolution threshold ranges from 20×20 to 40×40, and the first ambiguity threshold ranges from 10 to 20. The specific values ​​for the resolution threshold and the first ambiguity threshold can be set based on actual recognition accuracy requirements. For example, the resolution threshold can be set to 30×30, and the first ambiguity threshold can be set to 10.

[0104] In some embodiments of this application, when the first image filtering condition only includes "the resolution of the face image is greater than the resolution threshold", the resolution of each face image is determined, the resolution of each face image is compared with the resolution threshold, and face images with a resolution greater than the resolution threshold are selected as face images to be identified.

[0105] In some embodiments of this application, when the first image filtering condition only includes "the blur of the face image is less than the first blur threshold", the blur of each face image is determined, the blur of each face image is compared with the first blur threshold, and face images with blur less than the first blur threshold are selected as face images to be identified.

[0106] In some embodiments of this application, when the first image screening condition includes "the resolution of the face image is greater than the resolution threshold" and "the blur of the face image is less than the first blur threshold", the resolution and blur of each face image are determined respectively, the resolution of each face image is compared with the resolution threshold, and the blur of each face image is compared with the first blur threshold, and face images with a resolution greater than the resolution threshold and a blur less than the first blur threshold are selected as face images to be identified.

[0107] Accordingly, after selecting the face image to be identified, the face image to be identified is used as the face image of the first image to be identified, thereby extracting face feature information from the face image to be identified; the extracted face feature information is determined as the first object feature vector of the first object to be identified, which is used for subsequent identity recognition.

[0108] By filtering facial images, poor-quality images can be eliminated, thereby improving the accuracy and success rate of subsequent identity recognition.

[0109] In some embodiments of this application, for each face image, the blurriness of the face image can be calculated based on the following formula (1): In the above formula (1), Indicates the blurriness of an image. , The images are respectively in x and y Sobel gradient in the direction. In some embodiments of this application, the face image to be identified can be fed into a feature encoder for face feature extraction, thereby obtaining a first object feature vector.

[0110] In some embodiments of this application, the feature encoder may employ a lightweight feature extraction network, such as MobileNetV3, and the feature encoder may output feature vectors of a preset dimension.

[0111] In some embodiments of this application, the feature encoder may also employ an irreversible mapping design, meaning that the original face image cannot be reconstructed from the feature vectors. This prevents the leakage of the original face image and enhances information security.

[0112] Step 220. Based on the first object feature vector, determine the second object feature vector from the pre-entered object identity information database, which has a similarity greater than the similarity threshold with the first object feature vector.

[0113] In some embodiments of this application, the first image may contain one or more first objects that meet the identification criteria; therefore, the number of first object feature vectors that can be obtained may be one or more. For each first object feature vector, information is matched with an object identity information database to obtain the identification information of each first object.

[0114] In some embodiments of this application, for each first object feature vector, the similarity between the first object feature vector and each object feature vector in the object identity information database can be calculated. The maximum similarity among the calculated similarities is determined, and the maximum similarity is compared with a similarity threshold. If the maximum similarity is greater than the similarity threshold, the object feature vector in the object identity information database corresponding to the maximum similarity is determined as the second object feature vector matching the first object feature vector. The similarity threshold ranges from 0.5 to 1, and the specific value can be set according to the recognition accuracy requirements.

[0115] In some embodiments of this application, the cosine similarity between the first object feature vector and the feature vector of each object in the object identity information database can be calculated based on the following formula (2): In the above formula (2), Represents cosine similarity. Represents the feature vector of the first object. This represents the feature vector of each object in the object identity information database.

[0116] Step 230. Determine the object identity information associated with the feature vector of the second object as the identity recognition information of the first object.

[0117] In some embodiments of this application, for each first object feature vector, after determining the second object feature vector that matches it from the object identity information database, the object identity information associated with the second object feature vector in the object identity information database is determined as the identity recognition information of the first object corresponding to the first object feature vector.

[0118] In some embodiments of this application, for each first object feature vector, if no matching second object feature vector is determined from the object identity information database, the identity information of the first object corresponding to the first object feature vector can be determined to be empty.

[0119] The above method can be used to quickly identify the identity information of the first object.

[0120] In some embodiments of this application, the object feature vector and the object identity information associated with the object feature vector in the object identity information database can be entered by the user according to actual needs. The user can obtain an image containing the object to be entered through an electronic device, and input the identity information of the object to be entered into the electronic device, thereby enabling the electronic device to enter the object feature vector and the associated object identity information of the object to be entered into the object identity information database based on the obtained image and identity information of the object to be entered.

[0121] In some embodiments of this application, the electronic device supports different object entry modes, which may include, but are not limited to, album mode and scanning mode. Album mode is a mode for entering information based on images in an album, while scanning mode is a mode for entering information based on real-time captured video streams. Users can select different object entry modes according to their actual needs to enter object feature vectors and associated object identity information into the object identity information database. The electronic device can then use the corresponding entry strategy to enter the object identity information into the database based on the user-selected entry mode.

[0122] For example, see Figure 4 Electronic devices can display, for example Figure 4 The object entry interface 400 shown includes a first control 401 corresponding to the album mode and a second control 402 corresponding to the scanning mode. Users can trigger different first controls 401 or second controls 402 according to actual needs. When the first control 401 is triggered, the electronic device uses the entry strategy corresponding to the album mode to enter the object identity information database. When the second control 402 is triggered, the electronic device uses the entry strategy corresponding to the scanning mode to enter the object identity information database.

[0123] The input strategies for album mode and scan mode are explained below.

[0124] In some embodiments of this application, in album mode, a user can select a second image containing the second object to be entered from the album app, so that the electronic device can enter relevant information of the second object into the object identity information database based on the second image selected by the user. Specifically, the object feature vector and the object identity information associated with the object feature vector can be entered into the object identity information database through the following steps 410-450.

[0125] Step 410. Receive the first input from the user to select the second image from the album interface.

[0126] In some embodiments of this application, the second image is an image containing the second object to be entered. For example, if the second object is the user's friend Xiaoming, then the second image is an image containing Xiaoming's face.

[0127] In some embodiments of this application, the first input is an input for selecting a second image from a photo album app. Exemplarily, the first input includes, but is not limited to: a user clicking on the second image in the photo album interface using a touch device such as a finger or stylus, a voice command input by the user, a specific gesture input by the user, or other feasible inputs. The specific input can be determined according to actual usage needs, and this embodiment of the invention does not limit it. The specific gesture in this application embodiment can be any one of a single-click gesture, a swipe gesture, a drag gesture, a pressure-recognition gesture, a long-press gesture, an area-change gesture, a double-press gesture, or a double-tap gesture; the click input in this application embodiment can be a single-click input, a double-tap input, or any number of clicks, and can also be a long-press input or a short-press input. For example, as... Figure 5 The first input shown above can be: the user clicking on the second image 501 in the album interface 500 with their finger.

[0128] Step 420. In response to the first input, extract the third object feature vector of the second object from the second image.

[0129] In some embodiments of this application, after receiving a first input for selecting a second image, the electronic device performs feature extraction on the second image, thereby extracting a feature vector from the second image as the object feature vector of the second object, which is referred to as the third object feature vector for easy distinction.

[0130] In some embodiments of this application, the electronic device may use the same feature encoder as step 210 above to extract features from the second image.

[0131] In some embodiments of this application, in order to improve the accuracy of the final obtained third object feature vector, when extracting the third object feature vector, a local image of the second object can be segmented from the second image, and the third object feature vector can be obtained by extracting features from the local image.

[0132] In some embodiments of this application, the feature vector of the third object can be extracted using the following steps 4201-4204.

[0133] Step 4201. Detect the face and facial landmarks in the second image to obtain the second face detection result and the first facial landmark detection result.

[0134] In some embodiments of this application, the second face detection result includes at least one candidate face image, and the first facial landmark detection result includes the facial landmark coordinates of each face in the second image.

[0135] In some embodiments of this application, a face detection model can be used to perform face detection on a second image, thereby obtaining a second face detection result corresponding to the second image. The second face detection result may include the face detection bounding box information (x, y, w, h) of each detected candidate face and the confidence level of the corresponding face detection bounding box.

[0136] In some embodiments of this application, after obtaining the second face detection result, based on the face detection box information of each candidate face in the second face detection result, a face image of each candidate face can be extracted from the second image to obtain at least one candidate face image. Then, a facial landmark detection model can be used to detect facial landmarks in each candidate face image to obtain the facial landmark coordinates of each candidate face in the second image. The facial landmark coordinates may include, but are not limited to, the coordinates of the left eye (…). , ), right eye coordinates ( , ), nose tip coordinates ( , ), coordinates of the left corner of the mouth ( , ), coordinates of the right corner of the mouth ( , ).

[0137] In some embodiments of this application, when performing facial landmark detection on candidate face images, the size of each candidate face image can first be adjusted to the standard face size (e.g., 112×112). Then, for each candidate face image after size adjustment, a landmark detection model is used to perform facial landmark detection, thereby obtaining the facial landmark coordinates of each candidate face image. , ), i = 1, 2,…5]. By adjusting the candidate's face to the standard face size, it is easier to perform pose analysis based on the facial landmark coordinates.

[0138] Step 4202: If the second face detection result indicates that the second image includes a face, perform face alignment on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one aligned image.

[0139] In some embodiments of this application, the shooting angle of the face has a certain impact on feature extraction and comparison. Therefore, in order to eliminate the influence of the shooting angle, after obtaining at least one candidate face image, face alignment can be performed on each candidate face image to map the facial key points of the candidate face image to the standard facial key point positions.

[0140] In some embodiments of this application, when performing face alignment on a candidate face image, the affine matrix M of the face key point transformation corresponding to the candidate face image can be calculated. The calculation formula of the affine matrix M is shown in the following formula (3): In the above formula (3), This represents the standard facial landmark matrix.

[0141] For example, a standard facial landmark matrix As shown in equation (4): Based on the standard facial landmark matrix and the calculated coordinates of the facial landmarks in the candidate face image, an overdetermined system of equations can be obtained. Solving this system of equations using the least squares method yields the affine matrix. M Affine matrix M As shown in equation (5): Then, through affine transformation calculations, the face image after aligning the entire candidate face image to the standard coordinate system can be obtained, i.e., the aligned image.

[0142] In practice, OpenCV's affine transformation library functions can be directly called to calculate the affine matrix. M An affine transformation is performed on the candidate face image to obtain an aligned image of the candidate face image.

[0143] Step 4203. Select the face image of the second object from at least one aligned image.

[0144] In some embodiments of this application, to avoid information entry errors, the relevant information of one object is entered at a time when entering object information. Since the acquired second image may contain two or more faces, at least one aligned image is filtered to obtain the face image of the second object to be entered accurately.

[0145] In some embodiments of this application, when the number of obtained first aligned images is one, the first aligned image can be directly used as the face image of the second object.

[0146] In some embodiments of this application, when at least two first aligned images are obtained, a first candidate image and a second candidate image can be determined from the at least two first aligned images. The areas of the first candidate image and the second candidate image are determined respectively. The area ratio of the first candidate image and the second candidate image is calculated, and the area ratio is compared with a preset ratio threshold. If the area ratio is greater than or equal to the ratio threshold, the first candidate image is used as the face image of the second object. Wherein, the first candidate image is the image with an area greater than the first threshold among the at least two first aligned images; the second candidate image (face) is the image with an area greater than the second threshold among the at least two first aligned images; the first threshold is greater than the second threshold. The ratio threshold is a value greater than 1. The first threshold and the second threshold can be set based on the areas of the at least two first aligned images, wherein the first threshold is used to filter the image with the largest area among the at least two first aligned images, and the second threshold is used to filter the image with the second largest area among the at least two first aligned images.

[0147] For example, the ratio threshold is 2, the number of first aligned images is at least two, the aligned image with the largest area is used as the first candidate image, and the aligned image with an area only smaller than the first candidate image is used as the second candidate image. When the area of ​​the first candidate image is more than twice the area of ​​the second candidate image, the first candidate image is determined to be the face image of the second object and is then processed.

[0148] In some embodiments of this application, if the area ratio of the first candidate image to the second candidate image is less than or equal to a ratio threshold, it is considered that there is no obvious second object in the second image. In this case, a prompt message such as "Multiple faces detected, please focus on a single person's face for entry" can be entered to remind the user that the face image of the second object cannot be filtered out from the second image. This allows the user to adjust the image used for entry in a timely manner.

[0149] In some embodiments of this application, if face detection determines that the second image does not contain a face, a prompt message such as "No face detected, please aim at the person's face to capture the image" can be output to prompt the user to adjust the image used for input.

[0150] Step 4204. Extract the feature vector of the third object from the face image of the second object.

[0151] In some embodiments of this application, after selecting the face image of the second object, facial features are extracted from the face image of the second object, and the extracted facial feature vector is used as the third object feature vector of the second object.

[0152] In some embodiments of this application, to improve the accuracy of the feature vector of the third object, before performing step 4204 above, it can be determined whether the face image of the second object meets the second image filtering conditions. If it is determined that the face image of the second object meets the second image filtering conditions, then step 4204 above is performed.

[0153] In some embodiments of this application, the second image filtering criteria include at least one of the following: The size of the face image in the original image is greater than the first proportion threshold and less than the second proportion threshold; The pose information of the face in the face image satisfies the pose condition; The blurriness of the face image is less than the second blurriness threshold.

[0154] The first and second proportion thresholds are in the range of 0.1 to 0.9, the second ambiguity threshold is in the range of 10 to 20, and the pose conditions include pitch angle threshold and yaw angle threshold. The pitch angle threshold and yaw angle threshold are both in the range of 25 to 45 degrees. The specific values ​​of the above thresholds can be set according to the recognition accuracy requirements.

[0155] In some embodiments of this application, when the second image filtering condition includes "the size ratio of the face image in the original image is greater than a first ratio threshold and less than a second ratio threshold", the area of ​​the face image of the second object and the area of ​​the second image can be determined. Then, the ratio of the area of ​​the face image of the second object to the area of ​​the second image is used as the size ratio of the face image of the second object in the original image. This size ratio is compared with the first ratio threshold and the second ratio threshold respectively to determine whether the face image of the second object satisfies the condition "the size ratio of the face image in the original image is greater than the first ratio threshold and less than the second ratio threshold". Based on this condition, it is possible to avoid the face image of the second object being too large or too small.

[0156] In some embodiments of this application, the pose condition may include a face yaw angle less than a yaw angle threshold and a face pitch angle less than a pitch angle threshold. When the second image filtering condition includes "the pose information of the face in the face image satisfies the pose condition," the pose information of the face in the face image of the second object can be determined. This pose information includes, but is not limited to, the face yaw angle and the face pitch angle. The determined pose information is compared with the pose condition to determine whether the face image of the second object satisfies the condition "the pose information of the face in the face image satisfies the pose condition." Based on this condition, excessively large face yaw angles and pitch angles in the face image of the second object can be avoided.

[0157] In some embodiments of this application, the face yaw angle and face pitch angle in the face image of the second object can be solved using a pose estimation algorithm (such as solvePnP). Assuming standard face 3D coordinates... As shown in equation (6): The actual imaging model of the face in the camera is shown in the following equation (7): The parameter symbols are shown in Table 1 below: Table 1 Where K is an artificially constructed camera intrinsic parameter matrix, for example: focal length: ; Main point: ; Equation (8) is obtained through calculation: The face deflection angle can be calculated as: yaw = atan2( ), and face pitch angle: pitch = asin(- ).

[0158] When the face deflection angle exceeds the deflection angle threshold (e.g., 30 degrees), it is considered too large, and the algorithm returns a message indicating that the face deflection angle is too large. The electronic device then outputs a message such as "The left and right deflection angles of the captured face are too large; please record a straight face." Similarly, when the face tilt angle exceeds the tilt angle threshold (e.g., 25 degrees), it is considered too large, and the algorithm returns a message indicating that the face tilt angle is too large. The electronic device then outputs a message such as "The up and down tilt angles of the captured face are too large; please record a straight face." The deflection angle threshold and tilt angle threshold can be adjusted based on actual testing results. For non-face recording, these thresholds can be relaxed somewhat.

[0159] In some embodiments of this application, when the second image filtering condition includes "the blurriness of the face image is less than the second blurriness threshold", the blurriness of the face image of the second object can be calculated, and the blurriness can be compared with the second blurriness to determine whether the face image of the second object meets the condition "the blurriness of the face image is less than the second blurriness threshold". Based on this condition, the face image of the second object can be prevented from being too blurry.

[0160] In some embodiments of this application, when the blurriness of the face image of the second object is greater than or equal to a second blurriness threshold, the electronic device may output a prompt message such as "The image is too blurry, please hold the camera steady to record the face" to remind the user to adjust the image to be recorded so that the recording can be performed again based on the new image.

[0161] Step 430. Display the object identity information entry interface.

[0162] In some embodiments of this application, after successfully extracting the third object feature vector of the second object, the electronic device can display an object identity information input interface on the screen, and the user can input the object identity information of the second object in the object identity information input interface.

[0163] For example, the second object is a person, and the electronic device can display such a person as... Figure 6 The object identity information input interface 600, as described above... Figure 6 As shown, users can enter the name of the second object and its relationship to the user in the object identity information entry interface 600. The relationship has several preset options to choose from, or it can be entered manually.

[0164] Step 440. Receive the object identity information of the second object entered by the user on the object identity information entry interface.

[0165] Step 450. Associate the feature vector of the third object with the object identity information of the second object and store it in the object identity information database.

[0166] In some embodiments of this application, after the user enters the object identity information of the second object, they can click as shown. Figure 6 The save control 601 shown allows the electronic device to automatically associate and store the feature vector of the third object and the object identity information of the second object into the object identity information database according to the structure shown in Table 2 below.

[0167] Table 2 In some embodiments of this application, to enhance information security, the electronic device may encrypt and save the feature vector of the third object and the object identity information of the second object into an object identity information database. The object identity information database may be a private space file of the image processing software.

[0168] In some embodiments of this application, after saving is completed, the electronic device may output a prompt message such as "Personal information successfully stored".

[0169] At this point, the object feature vector and identity information of the second object have been successfully entered into the object identity information database. Subsequently, based on this information, the identity of the second object contained in other images can be identified.

[0170] In some embodiments of this application, in scanning mode, the object feature vector and the object identity information associated with the object feature vector can be entered into the object identity information database through the following steps 510-550.

[0171] Step 510. Display the shooting preview interface.

[0172] In some embodiments of this application, when a user enters information using a scanning mode, the electronic device responds to the user's selection of the scanning mode by displaying a shooting preview interface and turning on the camera to capture surrounding images.

[0173] For example, see Figure 7 In response to the user's selection of the scanning mode, the following display is shown: Figure 7 The shooting preview interface 700 shown displays images captured by the camera in real time.

[0174] Step 520. Receive voice information input by the user.

[0175] In some embodiments of this application, the electronic device supports user voice input of various commands and questions without wake-up while displaying a shooting preview interface. The electronic device can receive user-inputted voice information in real time through sensors such as microphones.

[0176] Step 530. If the voice information has the intention to input object identity information, extract the object identity information of the third object from the voice information.

[0177] In some embodiments of this application, users can trigger the information input process through voice information of specific phrases. For example, specific phrases may include, but are not limited to, "Remember XX," "Help me remember XX," "This person is XX," etc. For instance, the user inputs voice information. "Remember this person's name is Xiaoming, he is my friend."

[0178] In some embodiments of this application, after receiving voice information input by a user, the electronic device can invoke its local ASR (Automatic Speech Recognition) module to convert the user-input voice information into text. For example, the voice information... Transcribe to text "Remember this person's name is Xiaoming, he is my friend."

[0179] In some embodiments of this application, the electronic device can convert the obtained text. The Large Language Model (LLM) on the electronic device side, which serves as the user's query input, determines whether it represents an intent to input object identity information. If it is determined to be an intent to input object identity information, the LLM module can extract object identity information such as name and relational terms as the object identity information of the third object to be input.

[0180] In some embodiments of this application, the large language model can determine whether the voice information has the intent to input object identity information by judging whether the user query contains the aforementioned specific utterances. If it is determined that the user query contains the specific utterances, it is determined that there is an intent to input object identity information. Otherwise, it is determined that there is no intent to input object identity information.

[0181] For example, in text For example, if the message is “Help me remember this person’s name is Xiaoming, and he is my friend,” the large language model will return: {“Is the intention of the person recorded?”: “Yes”, [{“Name”: “Xiaoming”, “Relationship”: “Friend”}]}.

[0182] If the user's input voice information does not contain a specific phrase, it is determined that there is no intention to input object identity information. For example, if the user asks, "What is that in front of me?", the large language model will return: {"Intention to input person": "No", [{"Name": "", "Relationship": ""}]}.

[0183] In the current implementation, the semantic parsing and extraction of semantic information of the person are performed by a large language model on the electronic device. In practical applications, the transcribed text can also be sent to a cloud server language understanding model to complete the semantic parsing, and then the parsing result can be sent back to the electronic device. This application does not limit this.

[0184] Step 540. Extract the fourth object feature vector from the preview image displayed on the shooting preview interface.

[0185] In some embodiments of this application, when it is determined that the user's input voice information has the intent to record object identity information, the electronic device can trigger an automatic frame capture operation, according to a preset frequency (e.g., 100). ms / f Image frames are captured from the video stream displayed on the shooting preview interface, i.e., preview images. Then, the object feature vector of the third object is extracted from the captured preview image. For ease of distinction, it is referred to as the fourth object feature vector below.

[0186] In some embodiments of this application, the method of extracting the fourth object feature vector of the third object from the preview image is consistent with the method of extracting the third object feature vector of the second object from the second image.

[0187] For example, if the third object is a person, the feature vector of the fourth object can be extracted through the following steps 5401-5404.

[0188] Step 5401. Detect the faces and facial landmarks in the preview image to obtain the third face detection result and the second facial landmark detection result; the second face detection result includes at least one candidate face image, and the second facial landmark detection result includes the facial landmark coordinates of each candidate face image; Step 5402. If the third face detection result indicates that the preview image includes a face, perform face alignment on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one second aligned image. Step 5403. Select the face image of the third object from at least one second aligned image; Step 5404. Extract the fourth object feature vector from the face image of the third object.

[0189] The implementation methods of steps 5401-5404 above can be found in the relevant descriptions in steps 4201-4204 above. To avoid repetition, they will not be elaborated further here.

[0190] In some embodiments of this application, to improve the accuracy of the fourth feature vector, before step 5404 above, it can be determined whether the face image of the third object meets the second image filtering conditions. If it is determined that the face image of the third object meets the second image filtering conditions, then step 5404 above is executed. The implementation methods of the second image filtering conditions and the determination of whether the face image of the third object meets the second image filtering conditions can be found in the relevant descriptions in the above embodiments, and will not be repeated here to avoid repetition.

[0191] Using the above method, images of the images to be entered can be acquired in real time, and object information can be entered based on the input voice information, which is convenient and fast.

[0192] In some embodiments of this application, the electronic device and the cloud server may employ two different collaborative methods to generate object description information for the first image. The two collaborative methods are described below.

[0193] In some embodiments of this application, see Figure 8 Electronic devices and cloud servers can generate object description information for the first image by following steps 810-870.

[0194] Step 810. The electronic device performs identity verification on the first object and obtains the identity verification information of the first object.

[0195] Step 820. The electronic device sends the first image to the cloud server.

[0196] In some embodiments of this application, the electronic device can upload the first image to a cloud server via a network.

[0197] Step 830. The cloud server receives the first image sent by the electronic device.

[0198] Step 840. The cloud server performs image description on the first image using a multimodal model to obtain the image description information of the first image.

[0199] In some embodiments of this application, a multimodal model is deployed on the cloud server. After receiving the first image, the cloud server inputs the first image into the multimodal model for image modeling, thereby obtaining the image description information of the first image output by the multimodal model.

[0200] Step 850. The cloud server sends the image description information of the first image to the electronic device.

[0201] In some embodiments of this application, the cloud server returns image description information of the first image to the electronic device via a network.

[0202] Step 860. The electronic device receives image description information sent by the cloud server.

[0203] Step 870. The electronic device generates object description information for the first image based on the identity recognition information and image description information.

[0204] In some embodiments of this application, the electronic device can semantically splice the identity information of the first object and the image description information returned by the cloud server to obtain the object description information of the first image.

[0205] The final output is illustrated using examples of the first image containing one object and the first image containing two objects.

[0206] For example, see Figure 2The first image 200 contains a person, named "Xiaoming," whose object identification information includes: "Name": "Xiaoming," and "Relationship to User": "Friend." The image description information returned by the cloud server includes: a person wearing a blue denim jacket and black trousers. The electronic device can replace "a person" in the image description information with the corresponding object identification information according to the fixed template output by the model, thus obtaining the object description information. For example, the final object description information obtained is: Xiaoming wearing a blue denim jacket and black trousers.

[0207] For example, see Figure 3 The first image 310 contains two people: one is a pre-recorded friend, Xiaoming, and the other is a friend, Xiaotian. Therefore, the electronic device identifies the following object identity information: "Name": "Xiaoming", "Relationship with user": "Friend"; "Name": "Xiaotian", "Relationship with user": "Friend". The image description information returned by the cloud server includes: There are two people in front, one wearing a blue denim jacket and black trousers, and the other wearing a red jacket and black trousers. The electronic device can semantically concatenate the image description information and the object identity recognition information to provide the following object description information: There are two people in front, namely, Xiaoming, who is wearing a blue denim jacket and black trousers, and Xiaotian, who is wearing a red jacket and black trousers.

[0208] In some embodiments of this application, see Figure 9 Electronic devices and cloud servers can generate object description information for the first image by following steps 910-970.

[0209] Step 910. The electronic device performs identity verification on the first object and obtains the identity verification information of the first object.

[0210] Step 920. The electronic device sends a first image and object information of the first object to the cloud server. The object information includes the identity information of the first object.

[0211] In some embodiments of this application, the electronic device can package and upload the first image and object information of the first object to a cloud server.

[0212] In some embodiments of this application, in addition to including the identification information of the first object, the object information may also include detection information generated when the first object is identified, wherein the detection information includes at least the pixel coordinates of the first object in the first image.

[0213] For example, the first object is a person, and the identity information may include not only the identity recognition information of the first object, but also information such as the face detection result and person ID of each first object. In addition, the identity information uploaded to the cloud server may also include the image name of the first image.

[0214] For example, taking a person as the first object, the identity information sent by the electronic device to the cloud server is shown in Table 3 below: Table 3 In some embodiments of this application, if there are multiple first objects in the first image, the identity information of each first object can be sent to the cloud server.

[0215] Step 930. The cloud server receives the first image and identification information sent by the electronic device.

[0216] Step 940. The cloud server generates object description information for the first image based on the first image and object identification information.

[0217] In some embodiments of this application, a multimodal model is deployed on the cloud server. The cloud server can generate object description information of the first image based on the object information of the first image and the first object through the multimodal model.

[0218] In some embodiments of this application, when the number of first objects in the first image is 1, the above step 940 may include the following steps 9411-9412.

[0219] Step 9411. Based on the object information, generate the object identity semantic information of the first object.

[0220] In some embodiments of this application, structured object identity semantic information containing object information of the first object can be generated according to a preset format.

[0221] For example, for such Figure 2 The first image 200 shown can generate the following object identity semantic information: {"Face detection box color": "green", "Name": "Xiaoming", "Relationship": "Friend"}.

[0222] Step 9412. Input the object identity semantic information and the first image into the multimodal model to obtain the object description information output by the multimodal model.

[0223] In some embodiments of this application, the object identity semantic information of the generated first object can be encoded with the first image to obtain multimodal data. For example, the first image can be encoded into base64 format and then combined with the object identity semantic information to form multimodal data. Then, the multimodal data is input into a multimodal model to obtain the object description information output by the multimodal model.

[0224] In some embodiments of this application, multimodal data is input into a multimodal model to provide the first image and object identity semantic information to the multimodal model on the cloud server, thereby enabling the multimodal model to output object description information of the first image based on the first image and object identity semantic information.

[0225] In some embodiments of this application, when the number of the first objects is at least two, step 940 above may include the following steps 9421-9423: Step 9421. Based on the pixel coordinates of each first object, determine the image region of each first object in the first image.

[0226] Step 9422. In the first image, add a label to the image region of each first object to obtain the object index map.

[0227] In some embodiments of this application, when there are multiple first objects in the first image, in order to facilitate the differentiation of different first objects, for each first object, based on the detection information of the object, a mark is added at the image region position corresponding to the first object in the first image, thereby obtaining an index map containing object marks; based on the identity recognition information of each first object and the corresponding mark pattern, object identity semantic information of each first object is generated.

[0228] In some embodiments of the application, the markings may include, but are not limited to, bounding boxes. Different image regions may have different marking patterns. For example, different colored bounding boxes may be added for different image regions. Thus, image regions corresponding to different objects can be distinguished based on the markings in the index map.

[0229] For example, taking a person as the first object, for the uploaded first image 310, the cloud server can add a bounding box of different colors for different faces in the first image 310 according to the position information (x, y, w, h) of each face in the first image 310, thereby obtaining an object index map containing object markers.

[0230] Step 9423. Based on the identity information and corresponding tagging pattern of each first object, generate the object identity semantic information of each first object.

[0231] In some embodiments of this application, the markers in the object index map correspond one-to-one with the first objects, and each first object has its own corresponding identity information. Therefore, the markers correspond one-to-one with the identity information of the first objects. In this way, the cloud server can establish a one-to-one mapping relationship between identity information and corresponding image regions. Then, based on this mapping relationship, structured object identity semantic information corresponding to each first object is constructed according to a specific format.

[0232] For example, for such Figure 3 The first image 310 shown can generate the following object identity semantic information: {[{"Face detection box color": "blue", "Name": "Xiaotian", "Relationship": "Friend"}, {"Face detection box color": "green", "Name": "Xiaoming", "Relationship": "Friend"}]}.

[0233] Step 9424. Input the object identity semantic information and object index graph of each first object into the multimodal model to obtain the object description information output by the multimodal model.

[0234] In some embodiments of this application, after obtaining the object identity semantic information of each first object, the object identity semantic information of all first objects and the object index map are combined into multimodal data and input into the multimodal model on the cloud server, thereby generating object description information of the first image based on the multimodal data through the multimodal model.

[0235] Through the above method, the cloud server employs object identity semantic injection to invoke the multimodal model. In this way, the multimodal model in the cloud server only receives the semantic conclusion of the object's identity, without accessing the original object identity features or being able to infer the object's identity features in reverse. This resolves the privacy leakage risk associated with storing object identity features on the cloud server.

[0236] In some embodiments of this application, the model's response strategy can also be guided by system-level constraint rules in steps 9412 and 9424 above.

[0237] Step 950. The cloud server sends object description information to the electronic device.

[0238] Step 960. The electronic device receives object description information of the first image sent by the cloud server.

[0239] In this way, the cloud server directly generates object description information based on the first image and the identity information of the first object. The electronic device can directly output the object description information sent by the cloud server, which reduces the computational load of the electronic device.

[0240] Based on the image processing method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Please refer to the following embodiments.

[0241] See Figure 10 Some embodiments of this application provide an electronic device 1000 that includes the following modules: The acquisition module 1001 is used to acquire the identity information of the first object in the first image and the image description information of the first image; The output module 1002 is used to output object description information of the first image based on identity recognition information and image description information; the object description information is used to describe the identity and visual attribute information of the first object; wherein, the identity recognition information is obtained by the electronic device to recognize the first object; the image description information is obtained by the cloud server to recognize the first image.

[0242] In some embodiments of this application, the electronic device 1000 further includes: The processing module is used to perform identity recognition on the first object before outputting the object description information of the first image based on the identity recognition information and image description information, so as to obtain the identity recognition information of the first object; The communication module is used to send the first image to the cloud server; The communication module is also used to receive image description information of the first image sent by the cloud server; The processing module is also used to generate object description information of the first image based on the identity recognition information and the image description information of the first image.

[0243] In some embodiments of this application, the electronic device 1000 further includes: The processing module is used to perform identity recognition on the first object before outputting the object description information of the first image based on the identity recognition information and image description information, so as to obtain the identity recognition information of the first object; The communication module is used to send the first image and identification information to the cloud server; The communication module is also used to receive object description information of the first image sent by the cloud server.

[0244] In some embodiments of this application, the processing module is specifically used for: extracting a first object feature vector from a first image; determining a second object feature vector from a pre-recorded object identity information database based on the first object feature vector, wherein the similarity to the first object feature vector is greater than a similarity threshold; the object identity information database includes object feature vectors and object identity information associated with the object feature vectors; and determining the object identity information associated with the second object feature vector as the identity recognition information of the first object.

[0245] In some embodiments of this application, the first object is a person, and the processing module is further configured to: before extracting the first object feature vector of the first object from the first image, if the first image includes at least one face, extract face images corresponding to each face from the first image to obtain at least one face image; select face images that meet the first image selection conditions from the at least one face image as face images to be identified; extract face feature information from the face images to be identified; and determine the face feature information as the first object feature vector of the first object.

[0246] In some embodiments of this application, the first image filtering condition includes at least one of the following: the resolution of the face image is greater than a first resolution threshold; the blurriness of the face image is less than a first blurriness threshold.

[0247] In some embodiments of this application, the electronic device 1000 further includes: The receiving module is used to receive the first input from the user selecting a second image from the album interface; The processing module is also configured to extract the third object feature vector of the second object from the second image in response to the first input; The display module is used to display the object identity information input interface; The receiving module is also used to receive the object identity information of the second object entered by the user on the object identity information input interface; The storage module is used to associate and store the feature vector of the third object and the object identity information of the second object into the object identity information database.

[0248] In some embodiments of this application, the second object is a person, and the processing module is specifically used for: detecting the face and facial key points in the second image to obtain a second face detection result and a first facial key point detection result; the second face detection result includes at least one candidate face image, and the first facial key point detection result includes the facial key point coordinates of each candidate face image; when the second face detection result indicates that the second image includes a face, based on the facial key point coordinates corresponding to each candidate face image, performing face alignment on each candidate face image to obtain at least one first aligned image; filtering out the face image of the second object from the at least one first aligned image; and extracting the third object feature vector of the second object from the face image of the second object.

[0249] In some embodiments of this application, the processing module is specifically configured to: when the number of first aligned images is one, use the first aligned image as the face image of the second object; when the number of first aligned images is at least two, and the area ratio of the first candidate image to the second candidate image among the at least two first aligned images is greater than or equal to a ratio threshold, use the first candidate image as the face image of the second object; wherein, the first candidate image is an image among the at least two first aligned images whose area is greater than a first threshold; the second candidate image face is an image among the at least two first aligned images whose area is greater than a second threshold; and the first threshold is greater than the second threshold.

[0250] In some embodiments of this application, the processing module is further configured to: determine whether the face image of the second object meets the second image filtering conditions before extracting the third object feature vector of the second object from the face image of the second object; and extract the third object feature vector of the second object from the face image of the second object if the face image of the second object meets the second image filtering conditions.

[0251] In some embodiments of this application, the second image filtering condition includes at least one of the following: the area ratio of the face image in the original image is greater than a first ratio threshold and less than a second ratio threshold; the pose information of the face in the face image satisfies the pose condition; and the blurriness of the face image is less than a second blurriness threshold.

[0252] In some embodiments of this application, the electronic device 1000 further includes: The display module is used to display the shooting preview interface; The receiving module is used to receive voice information input by the user; The processing module is also used to extract the object identity information of a third object from the voice information when the voice information has the intention to input object identity information; The processing module is also used to extract the fourth object feature vector of the third object from the preview image displayed on the shooting preview interface; The storage module is used to associate and store the fourth object feature vector of the third object and the object identity information of the third object in the object identity information database.

[0253] In some embodiments of this application, the third object is a person, and the processing module is specifically used for: The system detects faces and facial landmarks in the preview image to obtain a third face detection result and a second facial landmark detection result. The second face detection result includes at least one candidate face image, and the second facial landmark detection result includes the coordinates of the facial landmarks in each candidate face image. If the third face detection result indicates that the preview image contains a face, the system aligns each candidate face image based on the coordinates of the facial landmarks corresponding to each candidate face image to obtain at least one second aligned image. From the at least one second aligned image, the system selects the face image of the third object. From the face image of the third object, the system extracts the fourth object feature vector of the third object.

[0254] Based on the image processing method provided in the above embodiments, this application also provides a specific implementation method for a cloud server. Please refer to the following embodiments.

[0255] See Figure 11 The above is a schematic diagram of the structure of a cloud server provided in some embodiments of this application, such as... Figure 11 As shown, the cloud server 1100 includes the following modules: Communication module 1101 is used to receive a first image and object information of a first object in the first image sent by an electronic device, wherein the object information includes the identification information of the first object; Processing module 1102 is used to generate object description information of the first image based on the first image and object information; The communication module 1101 is also used to send object description information to the electronic device.

[0256] In some embodiments of this application, the number of first objects is one, and the processing module 1102 is specifically used to: generate object identity semantic information of the first object based on object information; input the object identity semantic information and the first image into a multimodal model to obtain object description information output by the multimodal model.

[0257] In some embodiments of this application, the number of first objects is multiple, and the identity information also includes the pixel coordinates of each first object in the first image; The processing module 1102 is specifically used for: determining the image region of each first object in the first image based on the pixel coordinates of each first object; adding a marker to the image region of each first object in the first image to obtain an object index map; wherein the markers of different image regions have different marking patterns; generating object identity semantic information of each first object based on the identity recognition information of each first object and the corresponding marking pattern; and inputting the object identity semantic information and object index map of each first object into the multimodal model to obtain the object description information output by the multimodal model.

[0258] The electronic device in this application embodiment can also be a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the electronic device.

[0259] The electronic device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0260] The electronic device provided in this application embodiment can achieve... Figures 1 to 9 To avoid repetition, each process implemented by the electronic device in the method embodiment will not be described again here.

[0261] Optionally, such as Figure 12 As shown, this application embodiment also provides an electronic device 1200, including a processor 1201 and a memory 1202. The memory 1202 stores a program or instructions that can run on the processor 1201. When the program or instructions are executed by the processor 1201, they implement each step of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0262] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0263] Figure 13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0264] The electronic device 1300 includes, but is not limited to, the following components: radio frequency unit 1, network module 1302, audio output unit 1303, input unit 1304, sensor 1305, display unit 1306, user input unit 1307, interface unit 1308, memory 1309, and processor 1310.

[0265] Those skilled in the art will understand that the electronic device 1300 may also include a power supply (such as a battery) for each component, which may be logically connected to the processor 1310 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 13 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0266] Among them, electronic device 1300 can be a mobile terminal cloud server such as a mobile phone.

[0267] In some embodiments of this application, when the electronic device 1300 is a mobile device: Processor 1310 is used to acquire the identification information of the first object in the first image and the image description information of the first image; The display unit 1306 or the audio output unit 1303 is used to output object description information of the first image based on identity recognition information and image description information; the object description information is used to describe the identity and visual attribute information of the first object; wherein, the identity recognition information is obtained by the electronic device to recognize the first object; the image description information is obtained by the cloud server to recognize the first image.

[0268] In some embodiments of this application, the processor 1310 is configured to perform identity recognition on the first object before outputting object description information of the first image based on identity recognition information and image description information, and obtain identity recognition information of the first object. Network module 1302 is used to send the first image to the cloud server; Network module 1302 is also used to receive image description information of the first image sent by the cloud server; The processor 1310 is also used to generate object description information of the first image based on the identity recognition information and the image description information of the first image.

[0269] In some embodiments of this application, the processor 1310 is configured to perform identity recognition on the first object before outputting object description information of the first image based on identity recognition information and image description information, and obtain identity recognition information of the first object. Network module 1302 is used to send the first image and identity recognition information to the cloud server; The network module 1302 is also used to receive object description information of the first image sent by the cloud server.

[0270] In some embodiments of this application, the processor 1310 is specifically configured to: extract a first object feature vector from a first image; determine a second object feature vector from a pre-recorded object identity information database based on the first object feature vector, wherein the similarity to the first object feature vector is greater than a similarity threshold; the object identity information database includes object feature vectors and object identity information associated with the object feature vectors; and determine the object identity information associated with the second object feature vector as the identity recognition information of the first object.

[0271] In some embodiments of this application, the first object is a person, and the processor 1310 is further configured to: before extracting the first object feature vector of the first object from the first image, if the first image includes at least one face, crop the face image corresponding to each face from the first image to obtain at least one face image; select face images that meet the first image selection conditions from the at least one face image as face images to be identified; extract the first object feature vector of the first object from the first image, including: extracting face feature information from the face image to be identified; and determining the face feature information as the first object feature vector of the first object.

[0272] In some embodiments of this application, the first image filtering condition includes at least one of the following: the resolution of the face image is greater than a first resolution threshold; the blurriness of the face image is less than a first blurriness threshold.

[0273] In some embodiments of this application, the user input unit 1307 is used to receive a first input from a user selecting a second image from the album interface; The processor 1310 is also configured to, in response to the first input, extract a third object feature vector of the second object from the second image; Display unit 1306 is used to display the object identity information input interface; User input unit 1307 is also used to receive object identity information of a second object entered by the user on the object identity information input interface; The memory 1309 is used to associate and store the feature vector of the third object and the object identity information of the second object into the object identity information database.

[0274] In some embodiments of this application, the second object is a person, and the processor 1310 is specifically used for: detecting the face and facial key points in the second image to obtain a second face detection result and a first facial key point detection result; the second face detection result includes at least one candidate face image, and the first facial key point detection result includes the facial key point coordinates of each candidate face image; when the second face detection result indicates that the second image includes a face, based on the facial key point coordinates corresponding to each candidate face image, performing face alignment on each candidate face image to obtain at least one first aligned image; filtering out the face image of the second object from the at least one first aligned image; and extracting the third object feature vector of the second object from the face image of the second object.

[0275] In some embodiments of this application, the processor 1310 is specifically configured to: when the number of first aligned images is one, use the first aligned image as the face image of the second object; when the number of first aligned images is at least two, and the area ratio of the first candidate image to the second candidate image among the at least two first aligned images is greater than or equal to a ratio threshold, use the first candidate image as the face image of the second object; wherein, the first candidate image is an image among the at least two first aligned images whose area is greater than a first threshold; the second candidate image face is an image among the at least two first aligned images whose area is greater than a second threshold; and the first threshold is greater than the second threshold.

[0276] In some embodiments of this application, the processor 1310 is further configured to: determine whether the face image of the second object meets the second image filtering conditions before extracting the third object feature vector of the second object from the face image of the second object; and extract the third object feature vector of the second object from the face image of the second object if the face image of the second object meets the second image filtering conditions.

[0277] In some embodiments of this application, the second image filtering condition includes at least one of the following: the area ratio of the face image in the original image is greater than a first ratio threshold and less than a second ratio threshold; the pose information of the face in the face image satisfies the pose condition; and the blurriness of the face image is less than a second blurriness threshold.

[0278] In some embodiments of this application, the display unit 1306 is used to display a shooting preview interface; User input unit 1307 is used to receive voice information input by the user; The processor 1310 is also used to extract the object identity information of a third object from the voice information when the voice information has the intention to input object identity information; and to extract the fourth object feature vector of the third object from the preview image displayed on the shooting preview interface. The memory 1309 is used to associate and store the fourth object feature vector of the third object and the object identity information of the third object into the object identity information database.

[0279] In some embodiments of this application, the third object is a person, and the processor 1310 is specifically used to: detect the face and facial key points in the preview image to obtain a third face detection result and a second facial key point detection result; the second face detection result includes at least one candidate face image, and the second facial key point detection result includes the facial key point coordinates of each candidate face image; when the third face detection result indicates that the preview image includes a face, perform face alignment on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one second aligned image; filter out the face image of the third object from the at least one second aligned image; and extract the fourth object feature vector of the third object from the face image of the third object.

[0280] In some embodiments of this application, when the electronic device 1300 is a cloud server: Network module 1302 is used to receive a first image and object information of a first object in the first image sent by an electronic device, wherein the object information includes the identification information of the first object; Processor 1310 is used to generate object description information of the first image based on the first image and object information; Network module 1302 is also used to send object description information to electronic devices.

[0281] In some embodiments of this application, the number of first objects is one, and the processor 1310 is specifically used to: generate object identity semantic information of the first object based on object information; input the object identity semantic information and the first image into a multimodal model to obtain object description information output by the multimodal model.

[0282] In some embodiments of this application, the number of first objects is multiple, and the identity information also includes the pixel coordinates of each first object in the first image; The processor 1310 is specifically used for: determining the image region of each first object in the first image based on the pixel coordinates of each first object; adding a marker to the image region of each first object in the first image to obtain an object index map; wherein the markers of different image regions have different marking patterns; generating object identity semantic information of each first object based on the identity recognition information of each first object and the corresponding marking pattern; and inputting the object identity semantic information and object index map of each first object into the multimodal model to obtain the object description information output by the multimodal model.

[0283] It should be understood that, in this embodiment, the input unit 1304 may include a graphics processing unit (GPU) 13041 and a microphone 13042. The GPU 13041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1306 may include a display panel 13061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0284] The memory 1309 can be used to store software programs and various data. The memory 1309 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1309 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1309 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0285] Processor 1310 may include one or more processing units; optionally, processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1310.

[0286] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement each process of the above-described image processing method embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0287] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0288] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0289] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0290] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement each process of the image processing method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0291] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0292] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of this application.

[0293] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, executed by an electronic device, characterized in that, The method includes: Obtain the identity information of the first object in the first image and the image description information of the first image; Based on the identity recognition information and the image description information, the object description information of the first image is output; the object description information is used to describe the identity and visual attribute information of the first object. The identity recognition information is obtained by the electronic device through identity recognition of the first object; the image description information is obtained by the cloud server through image recognition of the first image.

2. The method according to claim 1, characterized in that, Before outputting the object description information of the first image based on the identity recognition information and the image description information, the method further includes: The first object is identified to obtain its identity information. Send the first image to the cloud server; Receive image description information of the first image sent by the cloud server; Based on the identity recognition information and the image description information of the first image, object description information of the first image is generated.

3. The method according to claim 1, characterized in that, Before outputting the object description information of the first image based on the identity recognition information and the image description information, the method further includes: The first object is identified to obtain its identity information. Send the first image and the identity recognition information to the cloud server; Receive object description information of the first image sent by the cloud server.

4. The method according to claim 2 or 3, characterized in that, The step of identifying the first object to obtain its identification information includes: From the first image, extract the first object feature vector of the first object; Based on the first object feature vector, a second object feature vector with a similarity greater than a similarity threshold is determined from a pre-entered object identity information database; the object identity information database includes object feature vectors and object identity information associated with the object feature vectors. The object identity information associated with the feature vector of the second object is determined as the identity recognition information of the first object.

5. The method according to claim 4, characterized in that, The first object is a person. Before extracting the first object feature vector from the first image, the method further includes: If the first image includes at least one face, extract the face image corresponding to each face from the first image to obtain at least one face image; From the at least one face image, select face images that meet the first image selection criteria as face images to be identified; Extracting the first object feature vector from the first image includes: Extract facial feature information from the face image to be identified; The facial feature information is determined as the first object feature vector of the first object.

6. The method according to claim 5, characterized in that, The first image filtering criteria include at least one of the following: The resolution of the face image is greater than the first resolution threshold; The blurriness of the face image is less than the first blurriness threshold.

7. The method according to claim 4, characterized in that, The method further includes: Receive the user's first input when selecting a second image from the album interface; In response to the first input, a third object feature vector of the second object is extracted from the second image; Displays the interface for entering the object's identity information; Receive the object identity information of the second object entered by the user on the object identity information input interface; The third object feature vector and the object identity information of the second object are associated and stored in the object identity information database.

8. The method according to claim 7, characterized in that, The second object is a person, and the step of extracting the third object feature vector of the second object from the second image in response to the first input includes: The faces and facial landmarks in the second image are detected to obtain a second face detection result and a first facial landmark detection result; the second face detection result includes at least one candidate face image, and the first facial landmark detection result includes the facial landmark coordinates of each candidate face image; If the second face detection result indicates that the second image includes a face, face alignment is performed on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one first aligned image. From the at least one first aligned image, the face image of the second object is selected; from the face image of the second object, the third object feature vector of the second object is extracted.

9. The method according to claim 8, characterized in that, The step of filtering out the face image of the second object from the at least one first aligned image includes: when the number of first aligned images is one, using the first aligned image as the face image of the second object; If the number of first aligned images is at least two, and the area ratio of the first candidate image to the second candidate image in the at least two first aligned images is greater than or equal to a ratio threshold, the first candidate image is used as the face image of the second object. Wherein, the first candidate image is an image with an area greater than a first threshold among at least two first aligned images; the second candidate image face is an image with an area greater than a second threshold among at least two first aligned images; the first threshold is greater than the second threshold.

10. The method according to claim 8, characterized in that, Before extracting the third object feature vector of the second object from the face image of the second object, the method further includes: Determine whether the face image of the second object meets the second image filtering criteria; Extracting the third object feature vector from the face image of the second object includes: If the face image of the second object meets the second image filtering conditions, the third object feature vector of the second object is extracted from the face image of the second object.

11. The method according to claim 10, characterized in that, The second image filtering criteria include at least one of the following: The area of ​​the face image in the original image is greater than the first proportion threshold and less than the second proportion threshold; The pose information of the face in the face image satisfies the pose condition; The blurriness of the face image is less than the second blurriness threshold.

12. The method according to claim 4, characterized in that, The method further includes: Displays the shooting preview interface; Receive voice information input by the user; If the voice information indicates an intention to input object identity information, the object identity information of the third object is extracted from the voice information. Extract the fourth object feature vector of the third object from the preview image displayed on the shooting preview interface; The fourth object feature vector of the third object and the object identity information of the third object are associated and stored in the object identity information database.

13. The method according to claim 12, characterized in that, The third object is a person, and the step of extracting the fourth object feature vector of the third object from the preview image displayed on the shooting preview interface includes: The faces and facial landmarks in the preview image are detected to obtain a third face detection result and a second facial landmark detection result; the second face detection result includes at least one candidate face image, and the second facial landmark detection result includes the facial landmark coordinates of each candidate face image; If the third face detection result indicates that the preview image includes a face, face alignment is performed on each candidate face image based on the facial key point coordinates corresponding to each candidate face image to obtain at least one second aligned image. From the at least one second aligned image, the face image of the third object is selected; Extract the fourth object feature vector from the face image of the third object.

14. An image processing method, executed by a cloud server, characterized in that, The method includes: Receive a first image and object information of a first object in the first image sent by an electronic device, wherein the object information includes the identity information of the first object; Based on the first image and the object information, generate object description information for the first image; Send the object description information to the electronic device.

15. The method according to claim 14, characterized in that, The number of the first objects is one. The step of generating object description information for the first image based on the first image and the object information includes: Based on the object information, generate the object identity semantic information of the first object; The object identity semantic information and the first image are input into a multimodal model to obtain the object description information output by the multimodal model.

16. The method according to claim 14, characterized in that, The number of the first objects is multiple, and the object information also includes the pixel coordinates of each first object in the first image; The step of generating object description information for the first image based on the first image and the object information includes: Based on the pixel coordinates of each first object, determine the image region of each first object in the first image; In the first image, a label is added to the image region of each first object to obtain an object index map; wherein, the labeling pattern of the labels for different image regions is different; Based on the identity information and corresponding tagging pattern of each first object, generate the object identity semantic information of each first object; The object identity semantic information of each first object and the object index graph are input into the multimodal model to obtain the object description information output by the multimodal model.

17. An electronic device, characterized in that, include: The acquisition module is used to acquire the identity information of the first object in the first image and the image description information of the first image; The output module is configured to output object description information of the first image based on the identity recognition information and the image description information; the object description information is used to describe the identity and visual attribute information of the first object. The identity recognition information is obtained by the electronic device through identity recognition of the first object; the image description information is obtained by the cloud server through image recognition of the first image.

18. The electronic device according to claim 17, characterized in that, The electronic device also includes: The processing module is configured to perform identity recognition on the first object before outputting the object description information of the first image based on the identity recognition information and the image description information, so as to obtain the identity recognition information of the first object; A communication module is used to send the first image to the cloud server; The communication module is also used to receive image description information of the first image sent by the cloud server; The processing module is further configured to generate object description information of the first image based on the identity recognition information and the image description information of the first image.

19. The electronic device according to claim 17, characterized in that, The electronic device also includes: The processing module is configured to perform identity recognition on the first object before outputting the object description information of the first image based on the identity recognition information and the image description information, so as to obtain the identity recognition information of the first object; The communication module is used to send the first image and the identity recognition information to the cloud server; The communication module is also used to receive object description information of the first image sent by the cloud server.

20. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1-13 or 14-16.

Images