Method for determining target object, shooting method and device

Abstract

The embodiment of the present application discloses a method for determining a target object, a shooting method and a device, which are used to conveniently and accurately determine the target object and improve the convenience of shooting. The method for determining the target object includes: obtaining an image to be processed; determining the object information of each object in the image to be processed, the object information including region information and attribute information; extracting second index data of each attribute information from a pre-established data set, the data set including attribute keywords and attribute indexes; for each object, associating the second index data with the region information to obtain the association relationship between the index data and the region information; obtaining first voice information; based on the attribute keywords corresponding to the second index data in the data set, extracting the first index data of the first voice information from the data set; based on the association relationship and the second index data, determining the target region information corresponding to the first index data, the object corresponding to the target region information is the target object.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for determining a target object, a shooting method and a device. Background Art

[0002] With the continuous development of terminal technology and image processing technology, the shooting functions of terminal devices are becoming more and more powerful.

[0003] Currently, when a terminal device is shooting, a target object is manually selected and then the target object is shot to obtain a video or picture of the tracked object.

[0004] When the user manually selects a target object, the target object actually selected may be inconsistent with the target object the user wants to specify due to reasons such as wrong selection, that is, the target object cannot be accurately selected. In addition, when the user is holding an object, he needs to put down the object in his hand first and then manually select the target object, which is less convenient. Summary of the invention

[0005] The embodiments of the present application provide a method for determining a target object, a shooting method and a device, which can allow a user to specify a target object conveniently and accurately.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a target object, which is applied to a terminal device, by first acquiring an image to be processed, wherein the image to be processed includes at least one object; then determining object information of each object in the image to be processed, wherein the object information includes region information and attribute information, and the region information is used to describe the position of the object in the image to be processed; then, extracting second index data of each attribute information from a pre-established data set, wherein the data set includes attribute keywords and attribute indexes; for each object, associating the second index data with the region information to obtain an association relationship between the index data and the region information; when acquiring the first voice information, extracting the first index data of the first voice information from the data set based on the attribute keywords corresponding to the second index data in the data set; finally, determining the target region information corresponding to the first index data based on the association relationship and the second index data, wherein the object corresponding to the target region information is the target object.

[0007] In an embodiment of the present application, object information of each object in an image is pre-identified, and based on a pre-established data set, an association relationship between index data and area information of each object is established; then, when a voice command is collected, the object in the image corresponding to the voice command is determined based on the association relationship, so that the user can quickly and easily specify the target object.

[0008] In some possible implementations of the first aspect, the process of determining the target area information corresponding to the first index data based on the association relationship and the second index data may include: for each second index data, matching each index in the second index data with each index in the first index data, and determining the number of items of the target index, where the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, determining the area information corresponding to the second index data with the largest number of items as the target area information.

[0009] In some possible implementations of the first aspect, the above-mentioned process of extracting the second index data of each attribute information from a pre-established data set may include: for each attribute information, searching the data set for a first keyword term that matches the attribute information; and using the index data of the first keyword term in the data set as the second index data.

[0010] In some possible implementations of the first aspect, the process of extracting the first index data of the first voice information from the data set based on the attribute keyword corresponding to the second index data in the data set may include: performing voice recognition on the first voice information to obtain a first voice recognition result; determining, based on the first voice recognition result, that the first voice information is a voice command for specifying a target object; searching for a second keyword that matches the first voice recognition result from the attribute keyword corresponding to the second index data; and using the index data of the second keyword in the data set as the first index data.

[0011] In some possible implementations of the first aspect, the above-mentioned process of performing speech recognition on the first voice information to obtain the first speech recognition result may include: extracting the voiceprint features of the first voice information; determining the similarity between the voiceprint features and the pre-stored voiceprint features; when the similarity is greater than or equal to a preset threshold, inputting the voiceprint features and the first voice information into a semantic understanding model to obtain the first speech recognition result output by the semantic understanding model.

[0012] In some possible implementations of the first aspect, the process of determining the object information of each object in the image to be processed may include: inputting the image to be processed into an object recognition model; and obtaining the object information output by the object recognition model.

[0013] In some possible implementations of the first aspect, the first voice information is a voice command for initially specifying a target object, or a voice command for adding or reducing a target object, or a voice command for replacing a target object. In this implementation, the user can not only specify a target object initially by voice, but can also add or reduce a target object, and replace a target object by voice after specifying the target object, which further increases shooting convenience and improves user experience.

[0014] In some possible implementations of the first aspect, the image to be processed includes N original images with different field of view angles, where N is a positive integer greater than or equal to 1. After determining the target area information corresponding to the first index data, the method further includes: based on the original image, obtaining a first output image of the target object according to the target area information, and displaying the first output image in a viewfinder.

[0015] In some possible implementations of the first aspect, the process of obtaining a first output image of the target object based on the original image and according to the target area information may include: determining a first target image from N original images, the first target image being an image with the smallest field of view in an image set, the image set including at least one original image that can completely contain an external frame selection area of ​​the target object; cropping a first sub-image of the target object from the first target image according to the target area information; and generating a first output image based on the first sub-image.

[0016] In some possible implementations of the first aspect, when there are at least two target objects and each target object corresponds to a first sub-image, the process of obtaining the first output image based on the first sub-image may include: splicing the first sub-image corresponding to each target object to obtain the first output image.

[0017] In some possible implementations of the first aspect, after displaying the first output image in a viewfinder, the method further includes: acquiring second voice information; performing voice recognition on the second voice information to obtain a second voice recognition result; determining, based on the second voice recognition result, that the second voice information is a voice command for adjusting a display mode of a target object; determining a second target image from an original image; cropping a second sub-image of the target object from the second target image; generating a second output image based on the second sub-image, and displaying the second output image in a viewfinder.

[0018] In this implementation, the user can adjust the display mode of the target object through voice, which further improves the convenience of shooting.

[0019] In some possible implementations of the first aspect, the method may further include: in a third target image, determining the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge, the third target image being the image with the largest field of view in the image set; if the minimum value of multiple distances is less than or equal to a preset distance threshold, displaying a prompt message in the viewfinder, the prompt message being used to prompt adjustment of the camera posture.

[0020] In this implementation, when the target object is about to exceed the maximum capture range of the camera, the user experience can be improved by prompting the user to adjust the camera posture.

[0021] In some possible implementations of the first aspect, the attribute information includes at least one of the following: type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture.

[0022] In second aspect, an embodiment of the present application provides a method for determining a target object, which is applied to a terminal device. The method displays a picture captured by a camera in a viewfinder, where the picture includes at least one object. When acquiring a first voice information, the first voice information is used to specify the target object, and the object in the picture corresponding to the first voice information is determined as the target object.

[0023] In some possible implementations of the second aspect, the process of determining the object corresponding to the first voice information in the picture as the target object may include: extracting first index data of the first voice information from a pre-established data set, the data set including attribute keywords and attribute indexes; based on the association relationship between the objects, determining target area information corresponding to the first index data, the object corresponding to the target area information being the target object;

[0024] The association relationship is a mapping relationship between the second index data of the attribute information of the object and the region information, and the region information is used to describe the position of the object in the original image captured by the camera.

[0025] In some possible implementations of the second aspect, before determining the target area information corresponding to the first index data based on the association relationship between each object, the method also includes: determining the object information of each object in the original image, the object information including area information and attribute information; extracting second index data of the attribute information of each object from the data set; for each object, associating the second index data with the area information to obtain the association relationship between the index data and the area information.

[0026] In some possible implementations of the second aspect, the process of extracting the second index data of the attribute information of each object from the data set may include: for each attribute information, searching the data set for a first keyword term that matches the attribute information; and using the index data of the first keyword term in the data set as the second index data.

[0027] In some possible implementations of the second aspect, the process of determining the object information of each object in the original image may include: inputting the original image into a target recognition model; and obtaining the object information output by the target recognition model.

[0028] In some possible implementations of the second aspect, the process of determining the target area information corresponding to the first index data based on the association relationship between the objects may include: for each second index data, matching each index in the first index data with each index in the second index data, and determining the number of items of the target index, where the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, determining the area information corresponding to the second index data with the largest number of items as the target area information.

[0029] In some possible implementations of the second aspect, the above-mentioned process of extracting the first index data of the first voice information from a pre-established data set may include: performing voice recognition on the first voice information to obtain a first voice recognition result; determining, based on the first voice recognition result, that the first voice information is a voice command for specifying a target object; searching for a second keyword that matches the first voice recognition result from the attribute keyword words corresponding to the second index data; and using the index data of the second keyword word in the data set as the first index data.

[0030] In some possible implementations of the second aspect, the above-mentioned process of performing speech recognition on the first voice information to obtain the first speech recognition result may include: extracting the voiceprint features of the first voice information; determining the similarity between the voiceprint features and the pre-stored voiceprint features; when the similarity is greater than or equal to a preset threshold, inputting the voiceprint features and the first voice information into a semantic understanding model to obtain the first speech recognition result output by the semantic understanding model.

[0031] In some possible implementations of the second aspect, the camera includes N cameras with different fields of view, where N is a positive integer greater than or equal to 1.

[0032] After determining the target area information corresponding to the first index data based on the association relationship between the objects, the method also includes: based on the original images of N different field angles, obtaining a first output image of the target object according to the target area information, and displaying the first output image in the viewfinder.

[0033] In some possible implementations of the second aspect, the process of obtaining the first output image of the target object based on the target area information based on N original images with different fields of view may include: determining the first target image from the N original images, the first target image being the image with the smallest field of view in the image set, the image set including at least one original image that can completely contain the external frame selection area of ​​the target object; cropping a first sub-image of the target object from the first target image based on the target area information; and generating a first output image based on the first sub-image.

[0034] In some possible implementations of the second aspect, when there are at least two target objects and each target object corresponds to a first sub-image, the process of obtaining the first output image based on the first sub-image may include: splicing the first sub-image corresponding to each target object to obtain the first output image.

[0035] In some possible implementations of the second aspect, after displaying the first output image in the viewfinder, the method also includes: acquiring second voice information; performing voice recognition on the second voice information to obtain a second voice recognition result; determining, based on the second voice recognition result, that the second voice information is a voice command for adjusting the display mode of the target object; determining a second target image from the original image; cropping a second sub-image of the target object from the second target image; generating a second output image based on the second sub-image, and displaying the second output image in the viewfinder.

[0036] In some possible implementations of the second aspect, the method also includes: in a third target image, determining the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge, the third target image being the image with the largest field of view in the image set; if the minimum value of multiple distances is less than a preset distance threshold, displaying a prompt message in the viewfinder, the prompt message being used to prompt adjustment of the camera posture.

[0037] In a third aspect, an embodiment of the present application provides a shooting method, which is applied to a terminal device. The method displays a picture captured by a camera in a viewfinder, where the picture includes at least one object. When a first instruction is received, the first instruction is used to specify at least two target objects, and at least two objects in the picture corresponding to the first instruction are determined as at least two target objects. Then, the picture of each target object is displayed in different areas in the viewfinder, wherein one area displays the picture of one target object.

[0038] In some possible implementations of the third aspect, the first instruction is a voice instruction.

[0039] In some possible implementations of the third aspect, the process of determining the at least two objects corresponding to the first instruction in the screen as the at least two target objects may include: extracting first index data of the first instruction from a pre-established data set, the data set including attribute keywords and attribute indexes; determining target area information corresponding to the first index data based on an association relationship between the objects, the objects corresponding to the target area information being the target objects;

[0040] The association relationship is a mapping relationship between the second index data of the attribute information of the object and the region information. The region information is used to describe the position of the object in the original image captured by the camera.

[0041] In some possible implementations of the third aspect, before determining the target area information corresponding to the first index data based on the association relationship between each object, the method also includes: determining the object information of each object in the original image, the object information including area information and attribute information; extracting the second index data of the attribute information of each object from the data set; for each object, associating the second index data with the area information to obtain the association relationship between the index data and the area information.

[0042] In some possible implementations of the third aspect, the process of extracting the second index data of the attribute information of each object from the data set may include: for each attribute information, searching the data set for a first keyword term that matches the attribute information; and using the index data of the first keyword term in the data set as the second index data.

[0043] In some possible implementations of the third aspect, the process of determining the object information of each object in the original image may include: inputting the original image into a target recognition model; and obtaining the object information output by the target recognition model.

[0044] In some possible implementations of the third aspect, the process of determining the target area information corresponding to the first index data based on the association relationship between the objects may include: for each second index data, matching each index in the first index data with each index in the second index data, and determining the number of items of the target index, where the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, determining the area information corresponding to the second index data with the largest number of items as the target area information.

[0045] In some possible implementations of the third aspect, the above process of extracting the first index data of the first instruction from a pre-established data set may include: searching for a second keyword matching the first instruction from the attribute keyword corresponding to the second index data; and using the index data of the second keyword in the data set as the first index data.

[0046] In some possible implementations of the third aspect, the camera includes N cameras with different fields of view, where N is a positive integer greater than or equal to 1; the process of displaying the picture of each target object in a viewfinder in different areas may include: for each target object, determining a first target image from N original images, the first target image being an image with the smallest field of view in an image set, the image set including at least one original image that can completely include an external frame selection area of ​​the target object; for each target object, cropping a first sub-image of the target object from the first target image based on the target area information; splicing the first sub-images corresponding to each target object to obtain a first output image; and displaying the first output image in the viewfinder.

[0047] In some possible implementations of the third aspect, after displaying the image of each target object in a region within the viewfinder, the method further includes: acquiring second voice information; performing voice recognition on the second voice information to obtain a second voice recognition result; determining, based on the second voice recognition result, that the second voice information is a voice command for adjusting the display mode of the target object; determining a second target image from the original image; cropping a second sub-image of the target object from the second target image; generating a second output image based on the second sub-image, and displaying the second output image in the viewfinder.

[0048] In some possible implementations of the third aspect, the method also includes: in a third target image, determining the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge, the third target image being the image with the largest field of view in the image set; if the minimum value of multiple distances is less than a preset distance threshold, displaying a prompt message in the viewfinder, the prompt message being used to prompt adjustment of the camera posture.

[0049] In a fourth aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method of any one of the first aspect, the second aspect, or the third aspect described above is implemented.

[0050] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the first aspect, the second aspect, or the third aspect mentioned above.

[0051] In a sixth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method as described in any one of the first aspect, the second aspect, or the third aspect. The chip system can be a single chip, or a chip module composed of multiple chips.

[0052] In a seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device executes the method described in any one of the first aspect, the second aspect, or the third aspect.

[0053] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the hardware structure of the terminal device 100 provided in an embodiment of the present application;

[0055] Figure 2 A software structure block diagram of the terminal device 100 according to an embodiment of the present application;

[0056] Figure 3 A schematic flowchart of a photographing method provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of tracking shooting in a video recording scenario provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of stable tracking provided in an embodiment of the present application;

[0059] Figure 6 A tracking and shooting schematic diagram of multi-target tracking provided in an embodiment of the present application;

[0060] Fig. 7A A schematic diagram of an image processing process in a tracking shooting scenario provided in an embodiment of the present application;

[0061] Figure 7B A schematic diagram of an interface in a tracking shooting scenario provided by an embodiment of the present application;

[0062] Figure 8 A schematic diagram of replacing a target object provided in an embodiment of the present application;

[0063] Fig.9A A schematic diagram of another image processing process in a tracking shooting scenario provided by an embodiment of the present application;

[0064] Fig. 9B A schematic diagram of another interface in a tracking shooting scenario provided by an embodiment of the present application;

[0065] Fig. 9C A schematic diagram of an original image set provided in an embodiment of the present application;

[0066] Fig.9D A schematic diagram of a sub-image provided in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of tracking shooting in a photo shooting scenario provided in an embodiment of the present application;

[0068] Fig.11 A schematic block diagram of the shooting process provided in an embodiment of the present application;

[0069] Fig.12 A schematic diagram of the voice recording process provided in an embodiment of the present application;

[0070] Fig.13 A schematic flowchart of a method for determining a target object provided in an embodiment of the present application;

[0071] Fig.14 Another schematic block diagram of the process of the method for determining a target object provided in an embodiment of the present application;

[0072] Fig.15 Another schematic block diagram of the process of the shooting method provided in the embodiment of the present application;

[0073] Fig.16 A schematic block diagram of a device for determining a target object provided in an embodiment of the present application;

[0074] Fig.17 Another schematic block diagram of a device for determining a target object provided in an embodiment of the present application;

[0075] Fig.18 A schematic block diagram of a photographing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0077] The following is an illustrative introduction to terminal devices that may be involved in the embodiments of the present application.

[0078] See also Figure 1 , is a schematic block diagram of the hardware structure of the terminal device 100 provided in the embodiment of the present application. Figure 1 As shown, the terminal device 100 may include a processor 110, a memory 120, an audio module 130, a speaker 130A, a receiver 130B, a microphone 130C, a camera 140, a display screen 150, and a sensor module 160. The sensor module 160 may include but is not limited to a touch sensor 160A and the like.

[0079] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0080] For example, in some embodiments, the terminal device 100 may also include at least one of the following: a headphone jack, a button, a motor, an indicator, a subscriber identification module (SIM) card interface 1, an external memory interface, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, an antenna, a mobile communication module, and a wireless communication module, etc.

[0081] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0082] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. The processor 110 may also be provided with a memory for storing instructions and data.

[0083] In some embodiments, the processor 110 may include one or more interfaces. For example, the interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a mobile industry processor interface (MIPI), and a general-purpose input / output (GPIO) interface.

[0084] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 160A and the camera 140 through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 160A through the I2C interface, so that the processor 110 communicates with the touch sensor 160A through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0085] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 130 via the I2S bus to achieve communication between the processor 110 and the audio module 130.

[0086] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 150 and the camera 140. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 140 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 140 communicate via the DSI interface to implement the display function of the terminal device 100.

[0087] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 140, the display 150, the audio module 130, the sensor module 160, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a MIPI interface, etc.

[0088] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0089] The terminal device 100 implements the display function through a GPU, a display screen 150, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 150 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0090] The display screen 150 is used to display images, videos, etc. The display screen 150 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 150, where N is a positive integer greater than 1.

[0091] The terminal device 1000 can realize the shooting function through the ISP, the camera 140, the video codec, the GPU, the display screen 150 and the application processor.

[0092] The ISP is used to process the data fed back by the camera 140. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 140.

[0093] The camera 140 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 140, where N is a positive integer greater than 1.

[0094] When the terminal device 100 includes at least two cameras 140, the at least two cameras 140 may include cameras with different field of view angles. For example, the terminal device 100 includes a wide-angle camera, a main camera, and a telephoto camera, and the three cameras have different field of view angles. Through the cameras with different field of view angles, images with different field of view angles can be collected.

[0095] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0096] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0097] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0098] The memory 120 may include an internal memory and / or an external memory, and the internal memory may be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100, etc. In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0099] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 130, the speaker 130A, the receiver 130B, the microphone 130C, and the application processor.

[0100] The audio module 130 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 130 can also be used to encode and decode audio signals. In some embodiments, the audio module 130 can be arranged in the processor 110, or some functional modules of the audio module 130 can be arranged in the processor 110.

[0101] The speaker 130A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music, listen to a hands-free call, or send a voice prompt message through the speaker 130A.

[0102] The receiver 130B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 130B close to the human ear.

[0103] The microphone 130C, also called a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 130C with his mouth to input the sound signal into the microphone 130C. The terminal device 100 can be provided with at least one microphone 130C.

[0104] The touch sensor 160A is also called a "touch panel". The touch sensor 160A can be arranged on the display screen 150, and the touch sensor 160A and the display screen 150 form a touch screen, also called a "touch screen". The touch sensor 160A is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 150. In other embodiments, the touch sensor 160A can also be arranged on the surface of the terminal device 100, which is different from the position of the display screen 150.

[0105] After introducing the hardware architecture of the terminal device 100, Figure 2 The software architecture of the terminal device 100 is exemplarily introduced.

[0106] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0107] Figure 2 This is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0108] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0109] The application layer can include a series of application packages.

[0110] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0111] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0112] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0113] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0114] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0115] The phone manager is used to provide communication functions for the terminal device 100. For example, the management of call status (including connected, hung up, etc.). The resource manager is used to provide various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0116] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0117] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0118] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0119] The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing. The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0120] The following is an illustrative description of the software and hardware workflow of the terminal device 100 in conjunction with the capture and photo shooting scene.

[0121] When the touch sensor 160A receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is a control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a static image or video through the camera 140.

[0122] In specific applications, the terminal device 100 may be a mobile phone, a tablet computer, or other types of terminal devices. The embodiment of the present application does not limit the specific type of the terminal device 100.

[0123] After exemplarily introducing the hardware architecture and software architecture of the terminal device 100, the technical solution provided in the embodiment of the present application is described in detail below using the terminal device 100 as an example.

[0124] See also Figure 3 , Figure 3 A schematic flow chart of a shooting method provided in an embodiment of the present application, the method is applied to a terminal device 100, and the method may include the following steps:

[0125] Step S301: Start a shooting application, enter a video recording mode or a photo taking mode, and display the captured image in a viewfinder.

[0126] It can be understood that the above-mentioned shooting application refers to a shooting application, usually a camera application.

[0127] After detecting the operation for starting the shooting application, the terminal device 100 starts the shooting application in response to the operation, and calls the camera driver through the kernel layer to capture the image through the camera and display the image in the viewfinder.

[0128] For example, see Figure 4 A tracking shooting schematic diagram in a video recording scene is shown, such as Figure 4 As shown, the main interface 42 of the mobile phone 41 displays applications such as camera 43, smart life, settings, calendar, gallery and clock. At this time, the terminal device 100 is specifically represented by the mobile phone 41, and the shooting application is specifically represented by the camera 43.

[0129] After receiving the click operation on the camera 43, the mobile phone 41 responds to the click operation and starts the camera 43. After the mobile phone 41 enters the video recording mode, the video recording preview interface 44 is displayed, and the viewfinder of the video recording preview interface 44 displays the shooting subjects such as the object 46, the object 47, the object 48 and the object 49. In addition, the video recording preview interface 44 also includes an icon 45, which is used to turn on and off the tracking shooting mode, that is, the user can enter or exit the tracking shooting mode by clicking the icon 45. At this time, the icon 45 indicates that the mobile phone 41 has not entered the tracking shooting mode.

[0130] In the embodiment of the present application, tracking shooting can also be called following shooting or target tracking. When tracking the movement of a target, the tracking shooting can keep the tracking target in the center of the picture as much as possible without moving the terminal device or manually adjusting the focus, thus avoiding the picture jitter caused by the user moving the terminal device.

[0131] Step S302: a trigger operation is detected, and in response to the trigger operation, a voice shooting mode is entered.

[0132] It should be noted that the above trigger operation is used to trigger the voice shooting mode, and the trigger operation may include single click, double click or long press. For example, the trigger operation is a click operation. Figure 4 After the mobile phone 41 receives the user's click operation on the icon 45, it enters the tracking shooting mode. In the tracking shooting mode, the mobile phone 41 can call the microphone to collect voice information in real time to realize functions such as allowing the user to specify the target object through voice, that is, after the mobile phone 41 enters the tracking shooting mode, it can be considered that it has entered the voice shooting mode. Of course, the mobile phone 41 can also enter and exit the voice shooting mode through other icons or buttons.

[0133] The trigger operation may also be a voice command, that is, the user can turn on and off the tracking shooting mode through voice commands. For example, after the terminal device 100 receives the voice "turn on voice photography" or "turn on voice video", it performs voice recognition on the voice to obtain a voice recognition result. When the voice recognition result includes a preset keyword, it can be determined that the voice is a voice command for triggering the voice shooting mode; and then respond to the voice command to enter the voice shooting mode. Among them, the preset keyword can be pre-set, for example, the preset keyword can include "turn on", "enter", "photograph", "shoot" and "video" and the like.

[0134] After the terminal device 100 enters the voice shooting mode, it performs target recognition on the original images with different field of view angles, collects voice information in real time through the microphone, and recognizes the voice information.

[0135] In some other embodiments, after starting the shooting application, the terminal device 100 may also automatically enter the voice shooting mode without triggering the operation to enter the voice shooting mode. In other words, the above step S302 is an optional step.

[0136] Step S303: Perform target recognition on each of the N original images to obtain object information of each object in each original image.

[0137] The terminal device 100 includes N cameras with different field of view angles, and the images are collected by the N cameras with different field of view angles respectively to obtain N original images with different field of view angles, where N is a positive integer greater than or equal to 1. For example, when the terminal device 100 is a mobile phone, the mobile phone may include a wide-angle camera, a telephoto camera, and a main camera, and the main camera, the wide-angle camera, and the telephoto camera have different field of view angles. The mobile phone can collect three original images with different field of view angles through these three cameras with different field of view angles.

[0138] When the terminal device 100 includes at least two cameras with different fields of view, the terminal device 100 can call N cameras with different fields of view when starting a shooting application to acquire N original images with different fields of view; or it can call only one camera for image acquisition when starting a shooting application, and display the acquired image in the viewfinder of the preview interface, and then call N cameras again after entering the voice shooting mode to acquire N original images with different fields of view.

[0139] After collecting N original images, the terminal device 100 can perform target recognition on each original image to obtain object information of each object in each original image. Each original image may contain one or more objects, which may include people, animals, and text.

[0140] Object information includes attribute information and region information. Region information is used to characterize the region where the object is located in the original image, that is, the region information can be used to characterize the location and size of the object in the original image. Generally, the region information can be the pixel coordinate information of the circumscribed frame area of ​​the object, which is usually a rectangular area, but can also be an area of ​​other shapes. Of course, the region information can also be the coordinate information of the actual region where the object is located in the image.

[0141] Attribute information can be used to describe an object, including but not limited to type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture. Different objects may contain different types of attributes. For example, for a person, the types of attribute information include gender, age group, and clothing color. For an animal, the types of attribute information include category and color.

[0142] The type refers to the category of the object, which may include people, animals, and text. The activity in the attribute information is used to describe the current action of the object, which may include skateboarding, running, and hula hooping. That is, the activity information in the attribute information can be used to know whether the current action of the object is running or playing hula hooping. Posture can be used to describe the current posture of the object. For example, the posture can be used to know whether the person is standing or sitting.

[0143] In some embodiments, the terminal device 100 can input N original images into the target recognition model respectively to obtain the object information output by the target recognition model. The target recognition model can be pre-built and pre-trained. The target recognition model can include a target detection network model and a target classification network model, etc., which are used to recognize images to identify the attribute information of objects in the image, as well as the location of each object in the image and the size of the area occupied. Of course, the terminal device 100 can also perform target recognition in other ways. For example, it can combine image semantic segmentation and instance segmentation to recognize the object information of each object in the image.

[0144] by Figure 4 For example, after the mobile phone 41 captures the original image corresponding to the video preview interface 44, the original image can be input into the target recognition model to obtain the object information of each object output by the target recognition model. For example, the mobile phone 41 can recognize that the original image includes information about the object 47, and through the information about the object 47, it can be determined that the object is a female and is playing with a hula hoop, as well as its position and size in the original image.

[0145] The terminal device 100 may perform target recognition on each frame of the captured original image in real time, or may perform target recognition every preset number of frames.

[0146] Step S304: extracting second index data of attribute information of each object from the pre-established data set.

[0147] It should be noted that the above data set is a pre-established user common information database, which can be specifically expressed as a database. The data set is pre-set with different attribute indexes for different object types. For example, for the character type, it can include but is not limited to the following attribute types: gender, age group, hair length, hair color, clothing type, clothing color, posture and activity, etc. A corresponding attribute index is set under each attribute type. In addition to characters, object types can also include but are not limited to animals, and text, etc. For the convenience of description, the following description will be taken as an example.

[0148] As an example but not limitation, the attribute index of the character type in the database may be as shown in Table 1 below.

[0149] Table 1

[0150]

[0151]

[0152] The terminal device 100 extracts the attribute information of each object in each original image, and matches the attribute information of each object with the attribute keywords in the corresponding attribute type in the database, so as to find the first keyword matching the attribute information in the database. The first keyword may include a single word and / or a phrase. If the first keyword matching the attribute information is found in the database, the index data of the first keyword is used as the second index data. Figure 4 Taking the illustrated scene as an example, object 46 is a little boy playing with a skateboard. The object is a character type. In the above Table 1, the attribute category index corresponding to the character type includes A to N.

[0153] First, the mobile phone 41 obtains the attribute information and area information of the object 46 through target recognition. The attribute information may include: male, child, skateboard, short hair, black hair, white T-shirt, ..., and standing, etc.

[0154] The attribute information of object 46 is matched with the attribute keywords in the corresponding attribute type in the database. Specifically, for the gender attribute type, "male" in the attribute information is matched with the attribute keywords corresponding to 001A and the keywords corresponding to 002A in Table 1 above, and the attribute index matching "male" in the attribute information is determined to be 001A.

[0155] For the age attribute type, the "child" in the attribute information is matched with the attribute keyword corresponding to 001B, the attribute keyword corresponding to 002B, the attribute keyword corresponding to 003B, and the attribute keyword corresponding to 004B in Table 1 above, and it is determined that the attribute index matching "little boy" is 003B.

[0156] Similarly, the attribute types such as skateboard, short hair, black hair, white T-shirt, ..., and standing in the attribute information are matched with the attribute keywords of the corresponding attribute types in Table 1 above to obtain the attribute index corresponding to the attribute information of object 46, that is, to obtain the second index data of object 46.

[0157] As an example and not a limitation, the attribute index of object 46 is shown in Table 2 below.

[0158] Table 2

[0159]

[0160]

[0161] Similarly, the attribute index of object 47 and the attribute index of object 48 can be obtained.

[0162] Among them, the attribute index of object 47 is shown in Table 3 below.

[0163] Table 3

[0164]

[0165] The attribute index of object 48 is shown in Table 4 below.

[0166] Table 4

[0167]

[0168] Step S305: for each object, associate the second index data with the region information to obtain an association relationship of each object, where the association relationship is an association relationship between the index data and the region information.

[0169] For each object, after extracting the second index data of the attribute information of the object, the terminal device 100 associates the second index data of the object with the region information of the object in each original image, so as to establish an association relationship between the index data of each object and the region information of each original image. Based on the association relationship, the region information of the object can be found through the index data. Figure 4 For example, Figure 4The second index data of the object 46 in the image may be as shown in Table 2 above. The mobile phone 41 includes a wide-angle camera, a main camera, and a telephoto camera, and the field of view of the wide-angle camera is the largest, the field of view of the main camera is the second largest, and the field of view of the telephoto camera is the smallest. Assume that the original image captured by the wide-angle camera and the main camera includes the object 46, while the original image captured by the telephoto camera does not include the object 46. At this time, the area information of the object refers to the pixel coordinate information of the external frame selection area of ​​the object.

[0170] The second index data of object 46 in Table 2 above is associated with the coordinates of the external frame selection area of ​​object 46 in the original image captured by the wide-angle camera, and is associated with the coordinates of the external frame selection area of ​​object 46 in the original image captured by the main camera, so as to establish an association relationship between the index data of object 46 and the area information.

[0171] As an example but not a limitation, the association relationship of object 46 is shown in Table 5 below.

[0172] Table 5

[0173]

[0174] As can be seen from Table 5 above, the index data of object 46 includes 001A, 003B, 001C and 002N, etc. The area information of object 46 in the original image captured by the wide-angle camera and the area information of object 46 in the original image captured by the main camera can be found through the index data.

[0175] Exemplarily, the processor 110 of the terminal device 100 performs target recognition on the original image captured by the camera 140 to obtain object information of each object in the original image; the processor 110 reads the database in the memory 120, and extracts second index data of the attribute information of each object from the database; for each object, the processor 110 associates the second index data with the area information to obtain an association relationship between the index data and the area information, and stores the association relationship in the memory 120.

[0176] At a certain moment, after the processor 110 collects the first voice through the microphone 130C and the audio module 130, it performs voice recognition on the first voice to obtain a first voice recognition result; the processor 110 then reads the database from the memory 120, extracts the first index data of the first voice recognition result from the database, and then reads the pre-established association relationship, and determines the target area information corresponding to the first index data according to the first index data and the association relationship.

[0177] Step S306: Detect first voice information.

[0178] Specifically, after entering the voice shooting mode, the terminal device 100 can call the microphone to collect the first voice information in real time.

[0179] Step S307: Perform voice recognition on the first voice information to obtain a first voice recognition result.

[0180] After collecting the first voice information, the terminal device 100 can extract the first voiceprint feature of the first voice information, and then determine the similarity between the first voiceprint feature and the pre-stored voiceprint feature. When the similarity between the first voiceprint feature and the pre-stored voiceprint feature is greater than a preset threshold, it is considered that the first voice information is the voice of a specific user, and voice recognition is performed based on the first voiceprint feature and the first voice information to obtain a first voice recognition result.

[0181] When the similarity between the first voiceprint feature and the pre-stored voiceprint feature is less than or equal to a preset threshold, the terminal device 100 may prompt the user that he or she has no authority, or may store the first voiceprint feature for the next voice command recognition.

[0182] In a specific application, the terminal device 100 can input the first voiceprint feature and the first voice information into a semantic understanding model to obtain a first voice recognition result output by the semantic understanding model. The semantic understanding model can include a voice feature analysis module and a semantic analysis module. The voice feature analysis module can be used to filter background interference sounds, and the semantic analysis module can be used to extract user voice commands.

[0183] In other embodiments, the terminal device 100 may not extract the voiceprint feature of the first voice information, that is, it may not distinguish whether the first voice information is the voice of a specific user, but directly recognize the first voice information to obtain a first voice recognition result.

[0184] Step S308: When it is determined according to the first speech recognition result that the first speech information is speech for specifying a target object, first index data of the first speech recognition result is extracted from the data set based on attribute keywords corresponding to the second index data.

[0185] After obtaining the first voice recognition result, the terminal device 100 may further confirm whether the first voice information is a voice command. When the first voice recognition result includes a specific keyword, the first voice information may be considered to be a voice for specifying a target object, and the specific keyword is pre-set. For example, the specific keyword may include "shooting", "tracking", and "taking a photo".

[0186] The first voice information may be a voice command for initially specifying a target object, or a voice command for increasing or decreasing a target object, or a voice command for replacing a target object, all of which may be considered as voice commands for specifying a target object. Increasing or decreasing a target object, and replacing a target object are all relative to a specified target object.

[0187] For example, Figure 4 The voice "tracking the little boy playing skateboard" collected in the example is the voice command for specifying the target object for the first time. After the interface 412 is displayed, the target object can be replaced from object 46 to other objects by collecting the first voice information, or other objects can be added or reduced as the target object based on object 46.

[0188] When the first voice information is not a voice for specifying a target object, and the first voice information is not a specific voice command, the terminal device 100 may prompt the user to re-enter the voice, or may not prompt the user. The specific voice command may exemplarily include a command to adjust the display size and display area of ​​the tracked object, etc.

[0189] When the first voice information is not a voice for specifying a target object but a specific voice command, the terminal device 100 may perform an operation corresponding to the specific voice command.

[0190] When the first voice information is a voice used to specify a target object, the terminal device 100 can match the first voice recognition result with the attribute keyword corresponding to the second index data, find out the second keyword matching the first voice recognition result, and then use the attribute index corresponding to the second keyword matching the first voice recognition result as the index data of the first voice recognition result.

[0191] It should be noted that in some other embodiments, the first speech recognition result may also be matched with each keyword in the database. However, matching with the attribute keyword corresponding to the second index data narrows the matching scope and improves the matching accuracy and matching speed compared to matching with all attribute keywords in the database.

[0192] For example, see Figure 4, the mobile phone 41 displays the video preview interface 44 in response to the user operation, and enters the voice shooting mode. The mobile phone 41 performs target recognition on the N original images collected, obtains the object information of each object in each original image, and extracts the second index data of the attribute information of each object from the database, and establishes the association relationship between the index data and the area information. Then, at a certain moment, the mobile phone 41 collects the first voice information through the microphone, and the first voice information is "tracking the little boy playing skateboard". After determining that the first voice information is the voice of a specific user according to the pre-stored voiceprint feature, the mobile phone 41 performs voice recognition on the first voice information, and obtains the first voice recognition result of "tracking the boy playing skateboard". Since the first voice recognition result contains the keyword "tracking", it is determined that the voice is a voice command for specifying the target object, and then searches for the index data of "skateboard" and "boy" from the attribute keywords corresponding to the second index data to obtain the first index data of the first voice recognition result. As shown in Table 1 above, it can be obtained that the index data of “skateboard” is 001C, and the index data of “boy” are 001A and 003B, that is, the first index data includes 001C, 001A and 003B.

[0193] Step S309: Determine the target region information corresponding to the first index data according to the association relationship between the index data and the region information, and the object corresponding to the target region information is the target object.

[0194] Based on the association relationship, the terminal device 100 searches the index data set for an index that matches the first index data, determines the object with the largest number of matching items as the target object, and determines the area information corresponding to the object as the target area information.

[0195] If no index matching the first index data is found in the index data set, the terminal device 100 may prompt the user that no object matching the voice is found. The prompting method may include voice prompting and / or text prompting. For example, the terminal device may issue a voice message "No matching object found, please re-enter the voice", and also display text corresponding to the voice in the viewfinder.

[0196] The index data set refers to a set of second index data including each object. Figure 4 Taking the scenario as an example, the index data set may include the second index data of object 46, the second index data of object 47, the second index data of object 48, and the second index data of object 49.

[0197] If the first voice recognition result contains only one object, but matches at least two target objects, the terminal device 100 can prompt the user to make further selections. The prompt information can be text prompt information, voice prompt information, or other types of prompt information. For example, the first voice recognition result is "tracking a little boy playing skateboard", that is, the first voice recognition result contains only one object "little boy", but matches two target objects. At this time, the mobile phone can allow the user to perform a point-and-select operation, or allow the user to re-enter the voice to further determine the target object that the user wants to specify from the two target objects.

[0198] When the terminal device 100 searches for the target area information corresponding to the first index data, the area information with the most matching items is used as the target area information corresponding to the first index data.

[0199] For example, Figure 4 Take the scenario of tracking a boy playing skateboard as an example, at this time, the first speech recognition result is "tracking a boy playing skateboard", and the index data corresponding to "skateboard" is 001C, and the index data corresponding to "boy" are 001A and 003B. That is, the first index data includes 001C, 001A and 003B.

[0200] Based on the index data set, it can be known that the second index data corresponding to the area information of object 46 includes 001A, 003B, 001C, 002D, 005E, 008F and 002N, and the second index data corresponding to the area information of target object 48 includes 001A, 003B, 003C and 002D, 005E, 001F and 002N.

[0201] By comparison, it can be seen that in the second index data of object 46, the target indexes that match the indexes in the first all data include 001C, 001A and 003B, that is, the number of items matching the first index data is 3, that is, the number of items of the target index is 3; while in the second index data of object 48, the target indexes that match the indexes in the first all data only include 001A and 003B, and 001C is not included, that is, the number of items matching the first index data is 2.

[0202] Similarly, it can be seen that the number of matching items for object 47 and object 49 is both 0.

[0203] That is, the target index of object 46 has the largest number of items, so the region information of object 46 is used as the target region information, and object 46 is used as the target object. As shown in Table 5 above, the target region information includes the coordinate information of the external frame selection region of the original image captured by the wide-angle camera and the coordinate information of the external frame selection region of the original image captured by the main camera.

[0204] In some embodiments, after determining the target object specified by the user's voice, the terminal device 100 may use a corresponding mark to identify the target object so that the user can know the target object. The marking method may be, but is not limited to, a rectangular box. For example, see Figure 4 After collecting the voice "tracking the boy playing skateboard" and performing voice recognition on the voice and determining the target area information, the mobile phone 41 can determine that the target object specified by the user is object 46, and based on the target area information, the area where the object 46 is located in the original image can be determined, and the object 46 can be marked with a rectangular frame, that is, the rectangular frame 411 is used to select the object 46 in the video preview interface 410 to prompt the user that the target object specified by the voice is object 46.

[0205] Of course, in some other embodiments, after determining the target object and the target area information, the mobile phone 41 may not use a corresponding mark to identify the target object.

[0206] Step S310: determine a target image from the N original images, and then crop a sub-image from the target image according to the target area information. The sub-image includes the target object. The target image refers to the image with the smallest field of view in the image set.

[0207] The image set includes at least one original image that can completely contain the external frame selection area of ​​the target object, that is, images that can completely contain the external frame selection area of ​​the target object are screened out from N original images, and these images are combined into an image set.

[0208] The terminal device 100 may first select images that can completely contain the circumscribed framed area of ​​the target object from N original images to form an image set, and then select the original image with the smallest field of view as the target image from the image set in ascending order of field of view. The circumscribed framed area of ​​the target object is usually a circumscribed rectangular area.

[0209] For example, as shown in Table 5 above, the target object is object 46, and the original image that can completely contain the external frame selection area of ​​object 46 includes the original image captured by the wide-angle camera and the original image captured by the main camera, while the original image captured by the telephoto camera does not contain object 46. At this time, the image set includes the original image captured by the wide-angle camera and the original image captured by the main camera. Since the field of view of the wide-angle camera is greater than the field of view of the main camera, the original image captured by the main camera is determined as the target image. Then, according to the target area information of object 46 in the original image captured by the main camera, a sub-image containing object 46 is collected.

[0210] After the terminal device 100 determines the target image, it can expand the outer frame area of ​​the target object by a certain ratio (for example, 5%), and then cut out a sub-image from the target image according to the preset image width-height ratio and the target area information of the target image. The preset image width-height ratio can be set according to actual needs.

[0211] It should be noted that one sub-image includes one target object. When there are multiple target objects, multiple sub-images are obtained by cropping. For example, the voice collected by the terminal device 100 is "tracking a little boy on a skateboard and a little boy running". At this time, the target objects include "tracking a little boy on a skateboard" and "a little boy running". According to the target area information corresponding to "tracking a little boy on a skateboard", a sub-image including "tracking a little boy on a skateboard" is cropped, and according to the target area information corresponding to "a little boy running", a sub-image including "a little boy running" is cropped.

[0212] Step S311: Generate an output image based on the sub-image, and then display the output image in a viewfinder.

[0213] It should be noted that, when there is only one sub-image, the terminal device 100 can reduce or enlarge the single sub-image to a preset resolution to obtain an output image, and then display the output image in the viewfinder according to the expected display method. When there are multiple sub-images, the terminal device 100 can splice the multiple sub-images into one image, and then reduce or enlarge the spliced ​​image to obtain the output image, or it can also reduce or enlarge a single sub-image, and then splice it to obtain the output image, and then display the output image in the viewfinder according to the expected display method. For example, see Figure 4 After collecting the voice "tracking the little boy playing skateboard", the mobile phone 41 recognizes the voice information and finds the corresponding target area information according to the voice recognition result; then the target object is selected by the circumscribed rectangular area 411 to prompt the user that the target object specified by the voice is object 46. At this time, the user can track and shoot the object 46 by clicking the shooting button. After receiving the click operation on the shooting button, the mobile phone 41 can crop a sub-image from the target image, generate an output image based on the sub-image, and display the output image in the viewfinder, that is, the display interface 412.

[0214] Of course, the user does not need to manually click the shooting button to trigger tracking shooting. At this time, the mobile phone 41 recognizes that the voice contains the keyword "tracking" and determines that the target object is object 46. After displaying interface 410, interface 412 can be automatically displayed.

[0215] In other embodiments, after the mobile phone 41 collects the voice "tracking the little boy playing skateboard" and determines through voice recognition that the object 46 needs to be tracked and photographed, the interface 412 can be directly displayed. That is, the mobile phone 41 may not display the interface 410, but after identifying the target object and the shooting action, it may directly track and photograph the object 46.

[0216] In the tracking shooting mode, when the target object moves, the terminal device 100 performs stable tracking of the target object. For example, see Figure 5 , Figure 5 A schematic diagram of stable tracking provided in an embodiment of the present application, Figure 4 and Figure 5 The same scenario. Figure 5 As shown, the mobile phone 41 captures an image 51 through a wide-angle camera, a main camera or a telephoto camera, and the image 51 includes multiple objects; target recognition is performed on the image 51 to obtain object information of each object in the image 51; second index data of the attribute information of each object is extracted from the database, and an association relationship between the index data and the area information of each object is established.

[0217] When the voice 52 input by the user is detected, the mobile phone 41 performs voice recognition on the voice 52 in combination with the object information, that is, the mobile phone 41 performs voice recognition on the voice 42 and obtains the voice recognition result "tracking the boy playing skateboard", and then determines the target object corresponding to the voice recognition result and the target area information corresponding to the target object based on the association between the index data and the area information. At this time, the mobile phone 41 determines that the target object specified by the voice 52 is the object 53, and according to the area information of the object 53, a sub-image containing the object 53 is cropped, and an output image 54 is generated based on the sub-image.

[0218] When the object 53 moves, the mobile phone 41 captures an image 55. Compared with the image 51, the position of the target object 53 in the image 55 has changed. The mobile phone 41 performs target recognition on the image 55 to obtain the object information, and then cuts it according to the area information of the object 53 to obtain a sub-image containing the object 53, and generates an output image 56 based on the sub-image.

[0219] It should be noted that, in the sub-image obtained by cropping according to the regional information of the object 53, in addition to the object 53, other background information may also be included. Figure 4 , Figure 5In the subsequent figures, some background information may be omitted, and only information related to the target object is shown. In the embodiment of the present application, the terminal device first performs target recognition on the images of each field of view angle to obtain the object information of each object, then extracts the index data corresponding to the attribute information of the object in the database, associates the index data with the area information, and obtains the association relationship between each object; then after detecting the voice information, the voice information is recognized, and the target object specified by the voice information and the target area information of the target object are determined according to the association relationship, and finally the target object is photographed according to the target area information. In this way, the user does not need to manually select the object in the frame to specify the target object, but can conveniently and accurately specify the tracking target through voice, thereby improving the convenience of shooting.

[0220] The embodiments of the present application can realize single target tracking as well as multi-target tracking.

[0221] See also Figure 6 , Figure 6 A tracking and shooting schematic diagram of multi-target tracking provided in an embodiment of the present application. Figure 6 As shown, when the mobile phone 61 receives a click operation on the camera 62, the mobile phone 61 starts the camera application and displays the captured image in the viewfinder. After entering the video recording mode, the mobile phone 61 displays the video recording preview interface 63, which includes the shooting subjects such as the object 64, the object 65, the object 66 and the object 67.

[0222] After the mobile phone 61 enters the voice shooting mode or the tracking shooting mode, the original images of different viewing angles are collected through the cameras of different viewing angles, and the target is recognized for each original image to obtain the object information of each object. In addition, based on the pre-established database, the index data of the attribute information of each object is extracted, and the association relationship between the index data and the area information is established.

[0223] Mobile phone 61 collects the voice "Two screens track a boy playing skateboard and a girl playing hula hoop" through the microphone, recognizes the voice and obtains a voice recognition result; then extracts index data of the voice recognition result from the database, and based on the correlation between the index data and the area information, determines the target object corresponding to the voice recognition result and the target area information.

[0224] It should be noted that when the speech specifies at least two target objects, the first index data of the speech recognition result includes the indexes of the at least two target objects. At this time, for the index data of each target object, the target area information is determined based on the association between the index data and the area information.

[0225] For example, Figure 6For example, the voice collected by mobile phone 61 includes two target objects, namely "boy playing skateboard" and "girl playing hula hoop". The first index data corresponding to the voice recognition result includes the index data corresponding to "boy playing skateboard" and the index data of "girl playing hula hoop". Then, according to the index data corresponding to "boy playing skateboard", the target area information of "boy playing skateboard" is determined based on the association relationship, and according to the index data of "girl playing hula hoop", the target area information of "girl playing hula hoop" is determined based on the association relationship.

[0226] At this time, the mobile phone 61 determines that the target objects corresponding to the voice "Two screens tracking the boy playing skateboard and the girl playing hula hoop" are object 64 and object 66. According to the area information of object 64 and the area information corresponding to object 66, the mobile phone 61 uses the circumscribed rectangle 69 to frame object 66, and uses the circumscribed rectangle 610 to frame object 64. That is, after the mobile phone 61 collects the voice and determines the target object specified by the voice, it displays the video preview interface 69 and prompts the user in the pre-preview interface 68 that the target objects specified by the voice are object 64 and object 66 by means of the circumscribed dotted rectangle.

[0227] Through voice recognition, mobile phone 61 can determine that the shooting targets corresponding to the voice "two screens tracking a boy playing skateboard and a girl playing hula hoop" are objects 64 and 66, and can also determine that the shooting action is tracking shooting and the display mode is two-screen display based on the extracted keywords "two screens" and "tracking".

[0228] At this time, after determining the target object specified by the voice, mobile phone 61 determines a target image that can completely contain the framed area of ​​object 64 and the framed area of ​​object 66 and has the smallest field of view. At this time, the target images of object 64 and object 66 may be different or the same. Specifically, for object 64, the target image of object 64 is determined; for object 66, the target image of object 66 is determined; then, based on the target area information, the first sub-image corresponding to object 64 is cropped from the target image of object 64, and the second sub-image corresponding to object 66 is cropped from the target image of object 66. Finally, the first sub-image and the second sub-image are spliced ​​to obtain the output image, and the viewfinder is displayed.

[0229] like Figure 6 As shown, the mobile phone 61 displays a tracking and shooting interface 611. The left half of the tracking and shooting interface 611 tracks and displays an object 64, and the right half tracks and displays an object 66.

[0230] In other embodiments, after the mobile phone 61 collects the voice and determines the target object corresponding to the voice, it can directly display the tracking shooting interface 611 without displaying the video preview interface 68.

[0231] It should be noted that when at least two sub-images are included, the target objects may be displayed sequentially in the order in which they appear in the user's voice. The sequential display may be exemplified as follows: from left to right in the horizontal screen state of the terminal device; from top to bottom in the vertical screen state. Figure 6 As shown, in the user voice "Two screens tracking a boy playing skateboard and a girl playing hula hoop", object 64 appears before object 66, and the mobile phone 61 is in landscape mode, and object 64 is displayed in the left area and object 66 is displayed in the right area.

[0232] In addition, the user voice may not specify two-screen tracking. When the mobile phone 61 determines that there are two target objects, it automatically uses two screens to track the two target objects. For example, the collected voice is "tracking a boy playing skateboard and a girl playing hula hoop".

[0233] Understandably, Figure 6 In this paper, two targets are used as an example to illustrate the multi-target tracking scenario. When more than three target objects are involved, the process is similar to Figure 6 For example, if the collected voice is "tracking a boy playing skateboard, a girl playing hula hoop, and a boy running", when the mobile phone 61 determines that the target objects specified by the voice include three, it uses three screens to track and display the three target objects.

[0234] In some embodiments, when the terminal device 100 determines the target object specified by voice and tracks and shoots the target object, if it is detected that the target object is close to the edge of the original image with the maximum field of view, a prompt message can be displayed on the interface to prompt the user to adjust the position of the camera or terminal to improve the user experience.

[0235] Specifically, the terminal device 100 detects the distance between the edge of the circumscribed frame selection area of ​​the target object and the edge of the image with the maximum field of view in the image with the maximum field of view. When the distance corresponding to the edge with the smallest distance is less than a certain distance threshold, it can be considered that the target object is about to exceed the maximum capture range of the camera. At this time, in order to continue tracking and shooting the target object, a prompt message can be displayed in the viewfinder. The prompt message is used to prompt the user to adjust the camera posture. Further, it can also be used to prompt the direction of camera adjustment. For example, the prompt message can be text information and symbol information, that is, the text "Please adjust the camera direction" is displayed in the viewfinder, and an arrow is used to indicate the direction of camera adjustment.

[0236] In some embodiments, when the terminal device 100 determines the target object specified by voice and tracks and shoots the target object, if the second voice information is collected, the second voice information is voice recognized to obtain a second voice recognition result. When it is determined according to the second voice recognition result that the second voice information is a voice command for adjusting the display mode of the target object, the cropping position of the sub-image of the target object is adjusted according to the target area information of the target object. The display mode of the target object includes the display position and / or display size of the target object.

[0237] For example, when the second voice information is "a little to the right", the terminal device 100 adjusts the cropping position of the sub-image so that the target object in the output image deviates from the center of the picture and deviates to the right. Generally, when cropping the sub-image, the target object is placed in the center of the picture as much as possible.

[0238] Among them, when it is determined according to the second voice recognition result that the second voice information is a voice command for enlarging or reducing the target object, that is, when the second voice information is used to adjust the display size of the target object, a suitable field of view angle image can be determined from different field of view angle images as the target image, and then a sub-image is obtained by cropping from the target image. For example, when the second voice information is "zoom in to the upper body", and the terminal device 100 determines according to the second voice recognition result that the second voice information is a voice command for enlarging the target object, the image corresponding to the telephoto camera is selected as the target image, and the sub-image is obtained by cropping from the original image captured by the telephoto camera.

[0239] During the tracking shooting process, the user can adjust the display position and / or display size of the target object through voice, further improving the convenience of shooting.

[0240] Exemplarily, the terminal device 100 is a mobile phone, which includes a main camera, a wide-angle camera and a telephoto camera, and the field of view angles from large to small are: wide-angle camera, main camera and telephoto camera. Fig. 7A This is a schematic diagram of an image processing process in a tracking shooting scene. Figure 7B This is a schematic diagram of an interface in a tracking shooting scenario.

[0241] like Fig. 7AThe mobile phone acquires images with different field of view angles through three cameras, performs target recognition on the images of each field of view angle, and obtains the object information of each object; based on the attribute information and area information of each object, an association relationship between the index data and the area information is established. After the mobile phone detects the voice "track the boy playing skateboard", it performs voice command recognition in combination with the object information, and determines that the target object specified by the voice is object 72; and, assuming that the images acquired by the main camera and the wide-angle camera can completely include the framed area of ​​object 72 at this time, the image acquired by the main camera is used as the target image, and then according to the target area information of object 72, the image acquired by the main camera is cropped according to the default ratio and position to obtain a sub-image of object 72; finally, an output image 73 is generated based on the sub-image, and image 73 is displayed on the tracking shooting interface 78. For details, please refer to Figure 7B .

[0242] The mobile phone continues to track and shoot the object 72. At a certain moment, it collects the voice "move to the right". The mobile phone recognizes the voice and determines that the voice is a voice command for adjusting the display position of the target object. Then, the mobile phone adjusts the cropping position according to the target area information corresponding to the object 72, and crops a sub-image from the image collected by the main camera to make the object 72 appear on the right side of the screen. That is, by adjusting the cropping position, the sub-image completely includes the framed area of ​​the object 72, and makes the object 72 deviate from the center of the screen and deviate to the right in the output image. Fig. 7A The mobile phone adjusts the cropping position to crop a sub-image, obtains image 74 based on the sub-image, and displays image 74 on the Figure 7B The tracking shooting interface 79 in.

[0243] During the tracking and shooting process, when the target object moves to the edge of the original image captured by the main camera, the mobile phone determines that the original image captured by the main camera cannot completely include the framed area of ​​the object 72, then selects the wide-angle camera image that can completely include the framed area of ​​the object 72, that is, determines the image captured by the wide-angle camera as the target image; then, according to the target area information corresponding to the object 72, the sub-image of the object 72 is cropped from the image captured by the wide-angle camera, and then an image 75 is generated based on the sub-image, and the image 75 is displayed on the screen. Figure 7B The tracking shooting interface 710 in FIG.

[0244] The object 72 continues to move. When the mobile phone detects that the target object 72 is close to the edge of the wide-angle camera image, it determines that the tracking target is about to exceed the maximum capture range of the camera, and then generates a prompt message to prompt the user to adjust the camera posture. Fig. 7A and Figure 7BAs shown, the mobile phone generates an image 76 and displays the image 76 on the tracking shooting interface 711. The image 76 includes a text prompt message "Please adjust the camera direction" and an arrow prompt to prompt the user to move the mobile phone to the right.

[0245] At a certain moment, after the mobile phone detects the voice "zoom in to the upper body", it performs voice recognition on the voice and obtains the voice recognition result. Among them, the pre-established database includes keywords of body parts, for example, the pre-established database includes upper body, lower body, head, eyes, mouth and upper limbs, etc.; according to the voice recognition result, it is determined that the voice is a voice command for zooming in on the target object, then from the images with three different field of view angles, the image captured by the telephoto camera is determined as the target image, and a sub-image is obtained by cropping from the target image, and image 77 is generated based on the sub-image. Finally, image 77 is displayed on the screen. Figure 7B The tracking shooting interface 712 shown in FIG.

[0246] In some embodiments, when the terminal device 100 tracks and shoots a target object, it can change the target object, add a target object, or reduce a target object through voice commands.

[0247] For example, see Figure 8 A schematic diagram showing the replacement of a target object is shown. Figure 8 and Figure 4 The same scenario. Figure 4 and Figure 8 As shown, after the mobile phone 41 displays the tracking shooting interface 412, the mobile phone 41 continues to track and shoot the object 46. At a certain moment, the mobile phone 41 collects the voice "Change to the running boy", and then the mobile phone 41 performs voice recognition on the voice to determine that the user needs to change the tracking object from object 46 to object 48; after extracting the index data of the voice, based on the association between the index data and the area information, the target area information of object 48 is determined; according to the target area information corresponding to object 48, a sub-image of object 48 is cropped from the framed area that can completely contain object 48 and the target image with the smallest field of view, and an output image is generated based on the sub-image, and the output image is displayed, that is, the tracking shooting interface 413 is displayed. The tracking object displayed in the tracking shooting interface 413 is object 48, that is, the tracking object is changed from object 46 to object 48.

[0248] After changing the tracking object, mobile phone 41 continues to track and shoot object 48. At a certain moment, mobile phone 41 collects the voice "add a screen, track the girl playing with hula hoop", and then mobile phone 41 performs voice recognition on the voice to obtain the voice recognition result; when it is determined according to the voice recognition result that the voice is a voice command for adding a tracking object, after extracting the index data of the voice recognition result, based on the association between the index data and the area information, it is determined that the object specified by the voice is object 47 and the target area information of object 47; then, according to the target area information corresponding to object 47, the sub-image of object 47 is cropped; finally, the sub-image of object 48 and the sub-image of object 47 are spliced ​​to obtain the output image, and the output image is displayed in the tracking shooting interface 414. In the tracking shooting interface 414, the left area tracks object 48, and the right area tracks object 47.

[0249] After adding the tracking object, the mobile phone 41 can continuously track the object 48 and the object 47 through two screens. At a certain moment, the mobile phone 41 collects the voice "remove the running boy", and then the mobile phone 41 recognizes the voice and obtains the voice recognition result; when it is determined that the voice is a voice command for reducing the tracking object according to the voice recognition result, after extracting the index data of the voice recognition result, based on the association between the index data and the area information, it is determined that the object specified by the voice is object 48, that is, it is determined that the object to be removed is object 48, and then based on the target area information of object 47, a sub-image of object 47 is cropped, an output image is generated based on the sub-image, and the output image is displayed on the tracking shooting interface 415.

[0250] For example, Fig.9A This is a schematic diagram of the image processing process in the tracking shooting scene. Fig. 9B for Fig.9A The corresponding interface diagram. Fig.9A and Fig. 9B and Figure 6 The same scenario.

[0251] like Fig.9A As shown, the mobile phone acquires images of different field of view angles through cameras of different field of view angles, performs target recognition on the images of each field of view angle, and obtains the object information of each object; based on the attribute information and area information of each object, an association relationship between the index data and the area information is established.

[0252] After the mobile phone detects the voice "tracking a boy playing skateboard and a girl playing hula hoop on two screens", the voice command recognition is performed in combination with the object information, and it is determined that the tracking objects specified by the voice are objects 92 and 93; then, an external rectangular frame can be used to select objects 92 and 93, as shown in image 91, which also includes objects 94 and 95; then, a target image is determined that can completely contain the framed area of ​​objects 92 and 93 and has the smallest field of view; then, according to the target area information of object 92 and the target area information of object 93, a sub-image of object 92 and a sub-image of object 93 are cropped from the target image; finally, the two sub-images are spliced ​​to generate image 96, and image 96 is displayed on Fig. 9B The tracking shooting interface 99 shown in the figure shows that the left area of ​​the tracking shooting interface 99 tracks and displays the object 92, and the right area tracks and displays the object 93.

[0253] For example, a mobile phone includes three cameras: a main camera, a telephoto camera, and a wide-angle camera. The original image set collected by the mobile phone is as follows: Fig. 9C As shown. The original image set includes an image 91 captured by the main camera, an image 912 captured by the telephoto camera, and an image 913 captured by the wide-angle camera. By comparison, it can be seen that compared with the image 91 of the main camera, the image in the original image set 913 with a larger field of view shows more objects, while the image in the original image set 912 with a smaller field of view shows fewer objects.

[0254] based on Fig. 9C In the original image set shown, for object 92, the object is included in the images captured by the main camera, wide-angle camera, and telephoto camera, but only the images captured by the main camera and wide-angle camera can completely include the external frame selection area of ​​object 92, that is, the image set of object 92 includes image 91 and image 913; then, the image with the smallest field of view angle is determined from the image set as the target image. At this time, the image set of object 92 includes image 91 and image 913. Since the field of view angle of image 91 is smaller than that of image 913, it can be determined that image 91 is the target image.

[0255] like Fig.9D It can be seen from the schematic diagram of the sub-image that after the mobile phone determines that the target image of the object 92 is the image 91, the sub-image 914 of the object 92 is cropped from the image 91 according to the predetermined aspect ratio.

[0256] Similarly, the sub-image of the object 93 is obtained by cropping, and then the sub-image 914 of the object 92 and the sub-image of the object 93 are scaled and displayed, and then spliced ​​to obtain the output image 96 .

[0257] The mobile phone continuously tracks and shoots objects 92 and 93, and at a certain moment, it collects a voice message "remove the boy playing skateboard and replace it with a boy running". The mobile phone recognizes the voice message and determines that the voice message is a voice command for changing the tracking object. Then, the mobile phone determines the target area information of objects 93 and 94, and cuts sub-images from the target image according to the target area information of objects 93 and 94, splices the sub-images to obtain image 97, and displays image 97 on the mobile phone. Fig. 9B The tracking shooting interface 910 is shown. The left area of ​​the tracking shooting interface 910 tracks and displays the object 94, and the right area tracks and displays the object 93.

[0258] The mobile phone continuously tracks and shoots objects 93 and 94, and at a certain moment, a voice "add a screen, track the little girl in red clothes" is collected. The mobile phone recognizes the voice, determines that the voice is a voice command for changing the tracking object, and determines that the target object corresponding to the voice is object 95; then, the target area information corresponding to objects 93, 94, and 95 is determined, and sub-images are obtained from the target image according to the target area information corresponding to objects 93, 94, and 95. The three sub-images are spliced ​​to obtain image 98, and the image 98 is displayed on Fig. 9B The tracking shooting interface 911 is shown. The left area of ​​the tracking shooting interface 911 tracks and displays the object 94, the middle area tracks and displays the object 93, and the right area tracks and displays the object 95.

[0259] It should be noted that, in the process of tracking and shooting the tracked object, the terminal device 100 can perform target recognition on the images of each field of view angle collected in real time, obtain the object information of each object, and establish the association relationship between the object's area information and index data. Based on this, after detecting the user's voice, the voice is recognized to obtain the voice recognition result, and based on the association relationship between the area information and the index data, the information contained in the voice is determined, for example, the target object corresponding to the voice is determined, the action contained in the voice is determined, etc.; finally, the action corresponding to the voice command is executed.

[0260] In an embodiment of the present application, during the process of tracking and shooting the tracked object, the user can change the tracked object, add the tracked object, and reduce the tracked object through voice, and can also adjust the display size and display position of the tracked object through voice, further improving the convenience and accuracy of tracking and shooting.

[0261] The above embodiments all use video recording scenes to exemplify tracking shooting. In specific applications, the technical solutions provided in the embodiments of the present application can also be applied to photo shooting scenes.

[0262] For example, see Fig.10Schematic diagram of tracking shooting in a photo shooting scene is shown. Fig.10 As shown, the main interface 102 of the mobile phone 101 includes multiple applications such as camera 103, smart life, settings, calendar, clock, and gallery. When the mobile phone 101 receives a click operation on the camera 103, the mobile phone 101 responds to the click operation and displays a photo preview interface 104. The viewfinder of the photo preview interface 104 includes multiple shooting subjects such as object 105, object 106, object 107, and object 108.

[0263] After the mobile phone 101 enters the voice shooting mode, it performs target recognition on images with different viewing angles to obtain object information, extracts index data of the object's attribute information from the database, and establishes an association relationship between the index data and the area information.

[0264] After collecting the voice "tracking the running boy", the mobile phone 101 displays the photo preview interface 109. In this process, the mobile phone 101 performs voice recognition on the voice, obtains the voice recognition result, and extracts the index data of the voice recognition result; then, according to the association between the index data and the area information, the target object corresponding to the voice recognition result and the target area information of the target object are searched, and at this time, the target object corresponding to the voice is determined to be the object 107; then, from the images with different field of view angles, the framed area that can completely include the object 107 and the target image with the smallest field of view angle are determined, and then according to the target area information of the object 107, a sub-image is cropped from the target image, and the sub-image is scaled and displayed to obtain an output image, and the photo preview interface 109 is displayed.

[0265] During the movement of the object 107, the mobile phone 101 continuously tracks the object 107 and displays a photo preview interface 1010. During the tracking and shooting process, the user can click the shooting button 1011 to take a photo of the object 107. After receiving the click operation on the shooting button 1011, the mobile phone 101 captures a photo of the object 107 in response to the click operation. When the mobile phone 101 receives the click operation on the button 1012, the mobile phone 101 displays a photo preview interface 1013 in response to the click operation, and the photo preview interface 1013 displays the photo of the object 107.

[0266] It should be noted that in the photo-taking scenario, the user can change, add, or reduce the tracking object through voice commands. The user can also use voice commands to adjust the display position of the tracking object, zoom in or out the tracking object, and prompt the user to adjust the camera posture when the tracking object reaches the edge of the image with the maximum field of view.

[0267] In order to better introduce the solution provided by the embodiment of the present application, the following will be described in conjunction with specific examples. At this time, the terminal device 100 is a mobile phone, which includes a wide-angle camera, a main camera and a telephoto camera, and the three cameras have different field of view angles.

[0268] See also Fig.11 The shooting process is shown in the schematic block diagram, as shown in Fig.11 As shown, photos or videos of the tracking target can be taken in two ways, one of which is a traditional way, and the other is a way provided in an embodiment of the present application.

[0269] In the traditional way, after the mobile phone enters the camera photo preview or video recording mode, it compares the target direction; the user manually moves the mobile phone to find the target object; after finding the target object, it manually zooms in and out to adjust the display ratio and focal length of the target object in the image; then, when the target object moves, it manually tracks the moving target, that is, manually moves the mobile phone to make the target object as close to the center of the picture as possible; finally, a photo or video of the target object is generated.

[0270] In the traditional method, the user needs to manually determine the target object, manually adjust the display ratio of the target object, and manually move the mobile phone when the target object moves.

[0271] In the method provided in the embodiment of the present application, when the mobile phone enters the camera to preview or record a photo, after comparing the target direction, the wide-angle camera, the main camera and the telephoto camera are used to collect wide-angle images, main camera images and telephoto images respectively, and target recognition is performed on the images of each field of view angle to obtain the object information of each object, and index data of the attribute information of the object is extracted from the database to establish a correlation relationship between the index data and the area information.

[0272] The user can send a tracking command by voice. The mobile phone can accurately identify the voice command based on the correlation between the index data and the area information and the pre-stored voiceprint features, and determine the target object corresponding to the voice command and the target area information of the target object; then, according to the target area information of the target object and the expected display mode, a sub-image is cropped from the field of view image, that is, a single target area or multiple target areas are generated. The expected display mode may include the image aspect ratio. Finally, according to the size of the target object, the focal length is adjusted, that is, the cropped image is scaled and displayed to achieve the target display ratio, and a photo or video of the tracked object is generated.

[0273] The method provided in the embodiments of the present application can also be found in Fig.12 The voice shooting process diagram is shown in FIG. Fig.12As shown, after the mobile phone enters the camera photo preview or video recording scene, the user can manually adjust the camera posture to align with the target direction. At this time, the mobile phone can collect wide-angle images, main camera images and telephoto images in real time, and perform target recognition on each image to obtain object information. At the same time, the mobile phone will also collect the user's voice commands in real time through the microphone, accurately recognize the user's voice commands based on the object information and pre-stored voiceprint features, determine the target object corresponding to the voice command, and the information contained in the voice command; then, according to the target area information of the target object and the expected display mode, a sub-image is cropped from the field of view angle image, that is, a single target area or multiple target areas are generated. The expected display mode may include the image aspect ratio. Finally, according to the size of the target object, adjust the focal length to achieve the target display ratio and generate photos or videos of the tracked object.

[0274] When the target object is detected to be deviated, that is, when the target object is detected to be close to the edge of the image with the maximum field of view angle, the user may be prompted to adjust the posture in the preview interface.

[0275] In the method provided in the embodiment of the present application, the user can specify the tracking object by voice, and the tracking object is continuously tracked without manual operation by the user. In comparison, the method provided in the embodiment of the present application allows the user to conveniently and accurately specify the tracking object, thereby improving the convenience of tracking shooting.

[0276] The embodiment of the present application also provides a method for determining a target object, so that a user can specify a target object by voice, thereby improving the convenience and accuracy of determining the target object.

[0277] See also Fig.13 , is a schematic flow chart of a method for determining a target object provided in an embodiment of the present application. The method can be applied to a terminal device 100. The method may include the following steps:

[0278] Step S1301: Acquire an image to be processed, where the image to be processed includes at least one object.

[0279] The above-mentioned image to be processed may include at least two original images with different field of view angles. For example, the image to be processed includes an original image captured by a telephoto camera and an original image captured by a wide-angle camera; it may also include an original image with one field of view angle.

[0280] Step S1302: Determine the object information of each object in the image to be processed, where the object information includes region information and attribute information, and the region information is used to describe the position of the object in the image to be processed.

[0281] In some embodiments, target recognition may be performed on the images to be processed to determine object information of each image in the images to be processed.

[0282] Step S1303: extract second index data of each attribute information from a pre-established data set, where the data set includes attribute keywords and attribute indexes.

[0283] Step S1304: for each object, associate the second index data with the region information to obtain an association relationship between the index data and the region information.

[0284] Step S1305: Acquire first voice information.

[0285] Step S1306: extracting first index data of the first voice information from the data set based on the attribute keyword corresponding to the second index data in the data set.

[0286] Step S1307: Determine the target area information corresponding to the first index data based on the association relationship and the second index data, wherein the object corresponding to the target area information is the target object.

[0287] in, Fig.13 The same points as those in the above embodiment can be found in the corresponding contents of the above embodiment, which will not be repeated here.

[0288] It should be noted that the method for determining the target object provided in the embodiment of the present application can be applied to the tracking video recording scene and the photo taking scene mentioned above, but is not limited to the scenes mentioned above. Fig.13 The same points as those in the above embodiment can be found in the corresponding contents of the above embodiment, which will not be repeated here.

[0289] In an embodiment of the present application, object information of each object in an image is pre-identified, and based on a pre-established data set, an association relationship between index data and area information of each object is established; when a voice command is collected, the object in the image corresponding to the voice command is determined based on the association relationship, so that the user can quickly and easily specify the target object.

[0290] See also Fig.14 , is another schematic flow chart of a method for determining a target object provided in an embodiment of the present application. The method can be applied to a terminal device 100. The method may include the following steps:

[0291] Step S1401: Display a picture captured by a camera in a viewfinder, where the picture includes at least one object.

[0292] The terminal device 100 may include one or at least two cameras with different viewing angles. The terminal device 100 may display an output image in a viewfinder based on an original image captured by one of the cameras.

[0293] Step S1402: Acquire first voice information, where the first voice information is used to specify a target object.

[0294] Step S1403: determine the object in the picture corresponding to the first voice information as the target object.

[0295] Among them, after collecting the original image, the terminal device 100 can perform target recognition on the original image to obtain object information of each object in the original image; then, based on the object information and a pre-established data set, establish an association relationship between the index data and the area information; after collecting the first voice information, the first voice information can be recognized, and based on the voice recognition result and the association relationship, the object corresponding to the first voice information can be determined, and the object is determined as the target object.

[0296] The process of establishing the association relationship and the process of determining the object corresponding to the first voice information can be referred to the above embodiment and will not be repeated here.

[0297] In the tracking shooting scenario, after the object corresponding to the first voice information in the picture is determined as the target object, the output image of the target object can be displayed, for example, Figure 4 In the scenario, after determining that the tracked object is object 46, interface 412 is displayed.

[0298] During the tracking shooting process, the user can adjust the display method of the target object, increase or decrease the target object, change the target object, etc. through voice. For detailed introduction, please refer to the above embodiment and will not be repeated here.

[0299] In the embodiment of the present application, the user can quickly and conveniently specify the target object through voice.

[0300] See also Fig.15 , is another schematic flow chart of a shooting method provided in an embodiment of the present application. The method can be applied to a terminal device 100. The method may include the following steps:

[0301] Step S1501: Display a picture captured by a camera in a viewfinder, where the picture includes at least one object.

[0302] Step S1502: Receive a first instruction, where the first instruction is used to specify at least two target objects.

[0303] The first instruction may be a voice instruction or a non-voice instruction. For example, the user may input the first instruction by finger clicking or touching.

[0304] Step S1503: determine at least two objects in the screen corresponding to the first instruction as at least two target objects.

[0305] When the first instruction is a voice instruction, the terminal device can first establish an association relationship between the index data and the region information of each object based on the object information of each object and a pre-established data set, and then determine the object corresponding to the first instruction in the image based on the association relationship and the voice recognition result. The specific process can be referred to in the above embodiment, which will not be repeated here.

[0306] When the first instruction is a non-voice instruction, the terminal device can determine the target object specified by the user according to the user's touch position. Figure 6 The first instruction is input by selecting objects 64 and 66 in the image, and the terminal device determines that the object corresponding to the position selected by the user is the target object according to the display position of each object in the image.

[0307] Step S1504: Displaying the image of each target object in the viewfinder in different regions, wherein one region displays the image of one target object.

[0308] When the first instruction is a voice instruction, the terminal device can crop the sub-images separately based on the target area information of the target object, and then splice the sub-images to display them in the expected display mode.

[0309] When the first instruction is a non-voice instruction, after determining the display area of ​​the target object, the terminal device may also crop the sub-images separately and splice the sub-images before displaying them.

[0310] For example, see Figure 6 In the scenario shown, the first instruction is a voice instruction. The terminal device specifies two target objects in the collected voice "Two screens tracking a boy playing skateboard and a girl playing hula hoop". After recognizing the voice and determining the target objects based on the association relationship, the interface 611 is displayed. The left area of ​​the interface 611 displays the screen of object 64, and the right area displays the screen of object 66. When the first instruction is used to specify three target objects, the display result of the terminal device can be seen in Fig. 9B Interface 911 in.

[0311] In the embodiment of the present application, when the user specifies at least two target objects, the terminal device displays the at least two target objects in different areas, thereby improving the convenience of tracking shooting.

[0312] Corresponding to the method for determining the target object in the above embodiment, Fig.16 A schematic block diagram of an apparatus for determining a target object provided in an embodiment of the present application is shown. For ease of explanation, only the portion related to the embodiment of the present application is shown.

[0313] See also Fig.16 , the device may include:

[0314] An image acquisition module 161 is used to acquire an image to be processed, where the image to be processed includes at least one object;

[0315] A first object information determination module 162, used to determine object information of each object in the image to be processed, where the object information includes region information and attribute information, and the region information is used to describe the position of the object in the image to be processed;

[0316] A first extraction module 163, used to extract second index data of each attribute information from a pre-established data set, the data set including attribute keywords and attribute indexes;

[0317] A first establishing module 164 is used to associate the second index data with the region information for each object to obtain an association relationship between the index data and the region information;

[0318] A first voice information acquisition module 165, used to acquire first voice information;

[0319] A second extraction module 166, configured to extract first index data of the first voice information from the data set based on the attribute keyword corresponding to the second index data in the data set;

[0320] The first target object determination module 167 is used to determine the target area information corresponding to the first index data based on the association relationship and the second index data, wherein the object corresponding to the target area information is the target object.

[0321] In some embodiments, the above-mentioned first target object determination module is specifically used to: for each second index data, match each index in the second index data with each index in the first index data, and determine the number of items of the target index, where the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, determine the area information corresponding to the second index data with the largest number of items as the target area information.

[0322] In some embodiments, the first extraction module is specifically used to: for each piece of attribute information, search the data set for a first keyword word that matches the attribute information; and use the index data of the first keyword word in the data set as the second index data.

[0323] In some embodiments, the above-mentioned second extraction module is specifically used to: perform voice recognition on the first voice information to obtain a first voice recognition result; determine, based on the first voice recognition result, that the first voice information is a voice command for specifying a target object; search for a second keyword that matches the first voice recognition result from the attribute keyword corresponding to the second index data; and use the index data of the second keyword in the data set as the first index data.

[0324] In some embodiments, the above-mentioned second extraction module is specifically used to: extract the voiceprint features of the first voice information; determine the similarity between the voiceprint features and the pre-stored voiceprint features; when the similarity is greater than or equal to a preset threshold, input the voiceprint features and the first voice information into the semantic understanding model to obtain the first voice recognition result output by the semantic understanding model.

[0325] In some embodiments, the first object information determination module is specifically used to: input the image to be processed into the target recognition model; and obtain the object information output by the target recognition model.

[0326] In some embodiments, the first voice information is a voice command for initially specifying a target object, or a voice command for increasing or decreasing a target object, or a voice command for replacing a target object.

[0327] In some embodiments, the image to be processed includes N original images with different field of view angles, where N is a positive integer greater than or equal to 1. The above apparatus may also include:

[0328] The first display module is used to obtain a first output image of the target object based on the original image and the target area information, and display the first output image in a viewfinder.

[0329] In some embodiments, the above-mentioned first display module is specifically used to: determine a first target image from N original images, the first target image is an image with the smallest field of view in an image set, and the image set includes at least one original image that can completely contain an external frame selection area of ​​the target object; according to the target area information, crop a first sub-image of the target object from the first target image; and generate a first output image based on the first sub-image.

[0330] In some embodiments, when there are at least two target objects and each target object corresponds to a first sub-image, the first display module is specifically used to: splice the first sub-images corresponding to each target object to obtain a first output image.

[0331] In some embodiments, the above apparatus further comprises:

[0332] A second voice information acquisition module, used to acquire second voice information;

[0333] A first recognition module is used to perform voice recognition on the second voice information to obtain a second voice recognition result;

[0334] A first determination module, configured to determine, based on the second voice recognition result, that the second voice information is a voice command for adjusting a display mode of a target object;

[0335] A second determination module, used to determine a second target image from the original image;

[0336] A first cropping module, used for cropping a second sub-image of a target object from a second target image;

[0337] The second display module is used to generate a second output image based on the second sub-image, and display the second output image in the viewfinder.

[0338] In some embodiments, the above apparatus may further include:

[0339] A first detection module is used to determine the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge in a third target image, where the third target image is an image with the largest field of view angle in the image set;

[0340] The first prompt module is used to display a prompt message in the viewfinder if the minimum value of the multiple distances is less than or equal to a preset distance threshold, and the prompt message is used to prompt the adjustment of the camera posture.

[0341] In some embodiments, the attribute information includes at least one of the following: type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture.

[0342] The above-mentioned device for determining the target object has the function of implementing the above-mentioned method for determining the target object. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function, and the module can be software and / or hardware.

[0343] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0344] Corresponding to the method for determining the target object in the above embodiment, Fig.17 A schematic block diagram of an apparatus for determining a target object provided in an embodiment of the present application is shown. For ease of explanation, only the portion related to the embodiment of the present application is shown.

[0345] See also Fig.17 , the device may include:

[0346] A first picture display module 171, used to display a picture captured by a camera in a viewfinder, the picture including at least one object;

[0347] A first acquisition module 172, used to acquire first voice information, where the first voice information is used to specify a target object;

[0348] The second target object determination module 173 is used to determine the object in the picture corresponding to the first voice information as the target object.

[0349] In some embodiments, the second target object determination module is specifically used to: extract first index data of the first voice information from a pre-established data set, the data set including attribute keywords and attribute indexes; based on the association relationship between various objects, determine the target area information corresponding to the first index data, the object corresponding to the target area information is the target object;

[0350] The association relationship is a mapping relationship between the second index data of the attribute information of the object and the region information, and the region information is used to describe the position of the object in the original image captured by the camera.

[0351] In some embodiments, the above apparatus may further include:

[0352] A second object information determination module is used to determine the object information of each object in the original image, where the object information includes region information and attribute information;

[0353] A third extraction module, used to extract second index data of attribute information of each object from the data set;

[0354] The second establishing module is used to associate the second index data with the region information for each object to obtain an association relationship between the index data and the region information.

[0355] In some embodiments, the third extraction module is specifically used to: for each piece of attribute information, search the data set for a first keyword word that matches the attribute information; and use the index data of the first keyword word in the data set as the second index data.

[0356] In some embodiments, the second object information determination module is specifically used to: input the original image into the target recognition model; and obtain the object information output by the target recognition model.

[0357] In some embodiments, the second target object determination module is specifically used to: for each second index data, match each index in the first index data with each index in the second index data, and determine the number of items of the target index, where the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, determine the area information corresponding to the second index data with the largest number of items as the target area information.

[0358] In some embodiments, the second target object determination module is specifically used to: perform voice recognition on the first voice information to obtain a first voice recognition result; determine, based on the first voice recognition result, that the first voice information is a voice command for specifying a target object; search for a second keyword that matches the first voice recognition result from the attribute keyword corresponding to the second index data; and use the index data of the second keyword in the data set as the first index data.

[0359] In some embodiments, the second target object determination module is specifically used to: extract the voiceprint features of the first voice information; determine the similarity between the voiceprint features and the pre-stored voiceprint features; when the similarity is greater than or equal to a preset threshold, input the voiceprint features and the first voice information into the semantic understanding model to obtain the first voice recognition result output by the semantic understanding model.

[0360] In some embodiments, the camera includes N cameras with different field of view angles, where N is a positive integer greater than or equal to 1. The above device also includes:

[0361] The third display module is used to obtain a first output image of the target object based on the original images with N different viewing angles according to the target area information, and display the first output image in the viewfinder.

[0362] In some embodiments, the third display module is used to: determine a first target image from N original images, where the first target image is an image with the smallest field of view in a set of images, and the set of images includes at least one original image that can completely contain an external framed area of ​​the target object; based on the target area information, crop a first sub-image of the target object from the first target image; and generate a first output image based on the first sub-image.

[0363] In some embodiments, when there are at least two target objects and each target object corresponds to a first sub-image, the third display module is specifically used to: splice the first sub-image corresponding to each target object to obtain a first output image.

[0364] In some embodiments, the above apparatus further comprises:

[0365] A second acquisition module, used to acquire second voice information;

[0366] A second recognition module, used to perform voice recognition on the second voice information to obtain a second voice recognition result;

[0367] A third determination module, configured to determine, based on the second voice recognition result, that the second voice information is a voice command for adjusting a display mode of a target object;

[0368] A fourth determination module, used to determine a second target image from the original image;

[0369] A second cropping module, used for cropping a second sub-image of the target object from the second target image;

[0370] The fourth display module is used to generate a second output image based on the second sub-image, and display the second output image in the viewfinder.

[0371] In some embodiments, the above apparatus may further include:

[0372] A second detection module is used to determine the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge in a third target image, where the third target image is an image with the largest field of view angle in the image set;

[0373] The second prompt module is used to display a prompt message in the viewfinder if the minimum value of the multiple distances is less than a preset distance threshold, and the prompt message is used to prompt the adjustment of the camera posture.

[0374] The above-mentioned device for determining the target object has the function of implementing the above-mentioned method for determining the target object. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function, and the module can be software and / or hardware.

[0375] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0376] Corresponding to the shooting method of the above embodiment, Fig.18 A schematic block diagram of a photographing device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0377] See also Fig.18 , the device may include:

[0378] The second picture display module 181 is used to display the picture captured by the camera in the viewfinder, and the picture includes at least one object;

[0379] A receiving module 182, configured to receive a first instruction, where the first instruction is used to specify at least two target objects;

[0380] A third target object determination module 183, configured to determine at least two objects in the picture corresponding to the first instruction as at least two target objects;

[0381] The regional display module 184 is used to display the image of each target object in the viewfinder in regional ways, wherein one region displays the image of one target object.

[0382] In some embodiments, the first instruction is a voice instruction.

[0383] In some embodiments, the third target object determination module is specifically used to: extract first index data of the first instruction from a pre-established data set, the data set including attribute keywords and attribute indexes; determine target area information corresponding to the first index data based on the association relationship between various objects, the object corresponding to the target area information is the target object;

[0384] The association relationship is a mapping relationship between the second index data of the attribute information of the object and the region information. The region information is used to describe the position of the object in the original image captured by the camera.

[0385] In some embodiments, the above apparatus may further include:

[0386] A third object information determination module is used to determine the object information of each object in the original image, where the object information includes region information and attribute information;

[0387] A fourth extraction module, used for extracting second index data of attribute information of each object from the data set;

[0388] The third establishing module is used to associate the second index data with the region information for each object to obtain an association relationship between the index data and the region information.

[0389] In some embodiments, the fourth extraction module is specifically used to: for each piece of attribute information, search the data set for a first keyword word that matches the attribute information; and use the index data of the first keyword word in the data set as the second index data.

[0390] In some embodiments, the third object information determination module is specifically used to: input the original image into the target recognition model; and obtain the object information output by the target recognition model.

[0391] In some embodiments, the third target object determination module is specifically used to: for each second index data, match each index in the first index data with each index in the second index data, and determine the number of items of the target index, where the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, determine the area information corresponding to the second index data with the largest number of items as the target area information.

[0392] In some embodiments, the third target object determination module is specifically used to: search for a second keyword matching the first instruction from the attribute keyword corresponding to the second index data; and use the index data of the second keyword in the data set as the first index data.

[0393] In some embodiments, the camera includes N cameras with different fields of view, where N is a positive integer greater than or equal to 1; the above-mentioned regional display module is specifically used to: for each target object, determine a first target image from N original images, the first target image is an image with the smallest field of view in an image set, and the image set includes at least one original image that can completely contain an external frame selection area of ​​the target object; for each target object, according to the target area information, crop a first sub-image of the target object from the first target image; splice the first sub-images corresponding to each target object to obtain a first output image; and display the first output image in the viewfinder.

[0394] In some embodiments, the above apparatus further comprises:

[0395] A third acquisition module, used to acquire second voice information;

[0396] A third recognition module is used to perform voice recognition on the second voice information to obtain a second voice recognition result;

[0397] a fifth determination module, configured to determine, based on the second voice recognition result, that the second voice information is a voice command for adjusting a display mode of a target object;

[0398] A sixth determination module, used to determine a second target image from the original image;

[0399] A third cropping module, used for cropping a second sub-image of the target object from the second target image;

[0400] The fifth display module is used to generate a second output image based on the second sub-image, and display the second output image in the viewfinder.

[0401] In some embodiments, the above apparatus may further include:

[0402] A third detection module is used to determine the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge in a third target image, where the third target image is an image with the largest field of view angle in the image set;

[0403] The third prompt module is used to display a prompt message in the viewfinder if the minimum value of the multiple distances is less than a preset distance threshold, and the prompt message is used to prompt the adjustment of the camera posture.

[0404] The above-mentioned shooting device has the function of implementing the above-mentioned shooting method, which can be implemented through hardware or through hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and / or hardware.

[0405] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0406] The terminal device provided in the embodiment of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method as described in any one of the above method embodiments is implemented.

[0407] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0408] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0409] The present application also provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0410] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above-mentioned embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. The reference "one embodiment" or "some embodiments" described in the present application specification means that one or more embodiments of the present application include specific features, structures or characteristics described in combination with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0411] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining a target object, characterized in that: The method comprises: Acquire an image to be processed, where the image to be processed includes at least one object; Determine object information of each of the objects in the image to be processed, the object information including region information and attribute information, the region information being used to describe the position of the object in the image to be processed; Extracting second index data of each of the attribute information from a pre-established data set, wherein the data set includes attribute keywords and attribute indexes; For each of the objects, associating the second index data with the region information to obtain an association relationship between the index data and the region information; Acquiring first voice information; extracting first index data of the first voice information from the data set based on the attribute keyword corresponding to the second index data in the data set; Based on the association relationship and the second index data, target area information corresponding to the first index data is determined, wherein the object corresponding to the target area information is a target object.

2. The method according to claim 1, characterized in that Determining target area information corresponding to the first index data based on the association relationship and the second index data includes: For each of the second index data, matching each index in the second index data with each index in the first index data respectively, and determining the number of items of a target index, where the target index is an index in the second index data that matches an index in the first index data; Based on the association relationship, the region information corresponding to the second index data with the largest number of items is determined as the target region information.

3. The method according to claim 1, characterized in that Extracting second index data of each of the attribute information from a pre-established data set includes: For each piece of attribute information, searching the data set for a first keyword matching the attribute information; The index data of the first keyword in the data set is used as the second index data.

4. The method according to claim 1, characterized in that: Extracting first index data of the first voice information from the data set based on the attribute keyword corresponding to the second index data in the data set includes: Performing voice recognition on the first voice information to obtain a first voice recognition result; Determining, according to the first voice recognition result, that the first voice information is a voice command for specifying a target object; Searching for a second keyword matching the first speech recognition result from the attribute keywords corresponding to the second index data; The index data of the second keyword in the data set is used as the first index data.

5. The method according to claim 4, characterized in that Performing speech recognition on the first speech information to obtain a first speech recognition result includes: Extracting voiceprint features of the first voice information; Determining the similarity between the voiceprint feature and a pre-stored voiceprint feature; When the similarity is greater than or equal to a preset threshold, the voiceprint feature and the first speech information are input into a semantic understanding model to obtain the first speech recognition result output by the semantic understanding model.

6. The method according to claim 1, characterized in that Determining object information of each of the objects in the image to be processed includes: Inputting the image to be processed into a target recognition model; The object information output by the target recognition model is obtained.

7. The method according to any one of claims 1 to 6, characterized in that: The first voice information is a voice command for initially specifying a target object, or a voice command for increasing or decreasing a target object, or a voice command for replacing a target object.

8. The method according to any one of claims 1 to 6, characterized in that: The image to be processed includes N original images with different field of view angles, where N is a positive integer greater than or equal to 1; After determining the target area information corresponding to the first index data, the method further includes: Based on the original image, a first output image of the target object is obtained according to the target area information, and the first output image is displayed in a viewfinder.

9. The method according to claim 8, characterized in that Based on the original image, obtaining a first output image of the target object according to the target area information includes: Determine a first target image from the N original images, where the first target image is an image with the smallest field of view in an image set, and the image set includes at least one original image that can completely contain an external frame selection area of ​​the target object; According to the target area information, cropping a first sub-image of the target object from the first target image; The first output image is generated based on the first sub-image.

10. The method according to claim 9, characterized in that When there are at least two target objects, and each target object corresponds to one first sub-image, obtaining the first output image based on the first sub-image includes: The first sub-images corresponding to each of the target objects are spliced ​​to obtain the first output image.

11. The method according to claim 9, characterized in that After displaying the first output image in a viewfinder, the method further includes: Acquiring second voice information; Performing voice recognition on the second voice information to obtain a second voice recognition result; According to the second voice recognition result, determining that the second voice information is a voice command for adjusting a display mode of the target object; Determine a second target image from the original image; cropping a second sub-image of the target object from the second target image; A second output image is generated based on the second sub-image, and the second output image is displayed in the viewfinder.

12. The method according to claim 9, characterized in that The method further comprises: In a third target image, determining the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge, the third target image being the image with the largest field of view angle in the image set; If the minimum value among the multiple distances is less than or equal to the preset distance threshold, a prompt message is displayed in the viewfinder, and the prompt message is used to prompt the adjustment of the camera posture.

13. The method according to claim 1, characterized in that The attribute information includes at least one of the following: type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture.

14. A method for determining a target object, characterized in that: Applied to a terminal device, the method comprises: Displaying a picture captured by a camera in a viewfinder, wherein the picture includes at least one object; Acquire first voice information, where the first voice information is used to specify a target object; Determine the object in the picture corresponding to the first voice information as a target object; The step of determining the object in the picture corresponding to the first voice information as the target object includes: Extracting first index data of the first voice information from a pre-established data set, the data set comprising attribute keywords and attribute indexes; Based on the association relationship between the objects, determine the target area information corresponding to the first index data, the object corresponding to the target area information is the target object; The association relationship is a mapping relationship between the second index data of the attribute information of the object and the region information, and the region information is used to describe the position of the object in the original image captured by the camera.

15. The method according to claim 14, characterized in that Before determining the target area information corresponding to the first index data based on the association relationship between the objects, the method further includes: Determine object information of each of the objects in the original image, the object information including the region information and the attribute information; Extracting second index data of the attribute information of each object from the data set; For each of the objects, the second index data is associated with the region information to obtain an association relationship between the index data and the region information.

16. The method according to claim 15, characterized in that Extracting second index data of the attribute information of each object from the data set includes: For each piece of attribute information, searching the data set for a first keyword matching the attribute information; The index data of the first keyword in the data set is used as the second index data.

17. The method according to claim 15, characterized in that Determining object information of each object in the original image includes: Inputting the original image into a target recognition model; The object information output by the target recognition model is obtained.

18. The method according to claim 14, characterized in that Determining target area information corresponding to the first index data based on the association relationship between the objects includes: For each of the second index data, matching each index in the first index data with each index in the second index data respectively, and determining the number of items of a target index, where the target index is an index in the second index data that matches an index in the first index data; Based on the association relationship, the region information corresponding to the second index data with the largest number of items is determined as the target region information.

19. The method according to claim 14, characterized in that Extracting first index data of the first voice information from a pre-established data set includes: Performing voice recognition on the first voice information to obtain a first voice recognition result; Determining, according to the first voice recognition result, that the first voice information is a voice command for specifying a target object; Searching for a second keyword matching the first speech recognition result from the attribute keywords corresponding to the second index data; The index data of the second keyword in the data set is used as the first index data.

20. The method according to claim 14, characterized in that Performing speech recognition on the first speech information to obtain a first speech recognition result includes: Extracting voiceprint features of the first voice information; Determining the similarity between the voiceprint feature and a pre-stored voiceprint feature; When the similarity is greater than or equal to a preset threshold, the voiceprint feature and the first speech information are input into a semantic understanding model to obtain the first speech recognition result output by the semantic understanding model.

21. The method according to any one of claims 14 to 20, characterized in that The camera includes N cameras with different field of view angles, where N is a positive integer greater than or equal to 1; After determining the target area information corresponding to the first index data based on the association relationship between the objects, the method further includes: Based on the original images of N different viewing angles, a first output image of the target object is obtained according to the target area information, and the first output image is displayed in a viewfinder.

22. The method according to claim 21, characterized in that Based on the original images of N different viewing angles, obtaining a first output image of the target object according to the target area information includes: Determine a first target image from the N original images, where the first target image is an image with the smallest field of view in an image set, and the image set includes at least one original image that can completely contain an external frame selection area of ​​the target object; According to the target area information, cropping a first sub-image of the target object from the first target image; The first output image is generated based on the first sub-image.

23. The method according to claim 22, characterized in that When there are at least two target objects, and each target object corresponds to one first sub-image, obtaining the first output image based on the first sub-image includes: The first sub-images corresponding to each of the target objects are spliced ​​to obtain the first output image.

24. The method according to claim 21, characterized in that After displaying the first output image in a viewfinder, the method further includes: Acquiring second voice information; Performing voice recognition on the second voice information to obtain a second voice recognition result; According to the second voice recognition result, determining that the second voice information is a voice command for adjusting a display mode of the target object; Determine a second target image from the original image; cropping a second sub-image of the target object from the second target image; A second output image is generated based on the second sub-image, and the second output image is displayed in the viewfinder.

25. The method according to claim 22, characterized in that The method further comprises: In a third target image, determining the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge, the third target image being the image with the largest field of view angle in the image set; If the minimum value among the multiple distances is less than the preset distance threshold, a prompt message is displayed in the viewfinder, and the prompt message is used to prompt the adjustment of the camera posture.

26. A shooting method, characterized in that: Applied to a terminal device, the method comprises: Displaying a picture captured by a camera in a viewfinder, wherein the picture includes at least one object; receiving a first instruction, wherein the first instruction is used to specify at least two target objects; determining at least two objects in the picture corresponding to the first instruction as the at least two target objects; Displaying the picture of each target object in the viewfinder in different areas, wherein one area displays the picture of one target object; Determining at least two objects in the picture corresponding to the first instruction as the at least two target objects includes: Extracting first index data of the first instruction from a pre-established data set, the data set including attribute keywords and attribute indexes; Determine object information of each object in the original image, wherein the object information includes region information and attribute information; Extracting second index data of the attribute information of each object from the data set; For each of the objects, associating the second index data with the region information to obtain an association relationship between the index data and the region information; Based on the association relationship between the objects, determine the target area information corresponding to the first index data, the object corresponding to the target area information is the target object; The association relationship is a mapping relationship between the second index data of the attribute information of the object and the region information, and the region information is used to describe the position of the object in the original image captured by the camera.

27. The method according to claim 26, characterized in that The first instruction is a voice instruction.

28. The method according to claim 26, characterized in that Extracting second index data of the attribute information of each object from the data set includes: For each piece of attribute information, searching the data set for a first keyword matching the attribute information; The index data of the first keyword in the data set is used as the second index data.

29. The method according to claim 26, characterized in that Determining object information of each object in the original image includes: Inputting the original image into a target recognition model; The object information output by the target recognition model is obtained.

30. The method according to claim 26, characterized in that Determining target area information corresponding to the first index data based on the association relationship between the objects includes: For each of the second index data, matching each index in the first index data with each index in the second index data respectively, and determining the number of items of a target index, where the target index is an index in the second index data that matches an index in the first index data; Based on the association relationship, the region information corresponding to the second index data with the largest number of items is determined as the target region information.

31. The method according to claim 26, characterized in that Extracting first index data of the first instruction from a pre-established data set includes: Searching for a second keyword matching the first instruction from the attribute keywords corresponding to the second index data; The index data of the second keyword in the data set is used as the first index data.

32. The method according to claim 26, characterized in that The camera includes N cameras with different field of view angles, where N is a positive integer greater than or equal to 1; Displaying each target object in the viewfinder in different regions includes: For each of the target objects, determine a first target image from the N original images, where the first target image is an image with the smallest field of view in an image set, and the image set includes at least one original image that can completely contain an external frame selection area of ​​the target object; For each of the target objects, cropping a first sub-image of the target object from the first target image according to the target area information; splicing the first sub-images corresponding to each of the target objects to obtain a first output image; The first output image is displayed in a viewfinder.

33. The method according to any one of claims 26 to 32, characterized in that After displaying the image of each target object in the viewfinder in different regions, the method further includes: Acquiring second voice information; Performing voice recognition on the second voice information to obtain a second voice recognition result; According to the second voice recognition result, determining that the second voice information is a voice command for adjusting a display mode of the target object; Determine a second target image from the original image; cropping a second sub-image of the target object from the second target image; A second output image is generated based on the second sub-image, and the second output image is displayed in the viewfinder.

34. The method according to claim 32, characterized in that The method further comprises: In a third target image, determining the distance between each edge of the circumscribed frame selection area of ​​the target object and the corresponding image edge, the third target image being the image with the largest field of view angle in the image set; If the minimum value among the multiple distances is less than the preset distance threshold, a prompt message is displayed in the viewfinder, and the prompt message is used to prompt the adjustment of the camera posture.

35. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 13, 14 to 25, or 26 to 34 is implemented.

36. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13, 14 to 25, or 26 to 34 is implemented.

Images