A moving target detection method, an electronic device and a storage medium

Abstract

The application provides a moving target detection method, an electronic device and a storage medium, and relates to the technical field of terminals. The method comprises the following steps: firstly, in response to a shooting operation triggered by a user on a terminal device, a terminal device acquires a first target image and a second target image, and performs key point registration processing on the two target images to obtain two target images after registration. Then, after performing binaryzation processing on a target optical flow map calculated according to the two target images after registration, a primary selected connected domain in which a moving target is located in the obtained optical flow binaryzation image is determined. Then, the two target images after registration are subjected to interframe difference and binaryzation processing to obtain a difference binaryzation image, and the primary selected connected domain is projected onto the difference binaryzation image, so as to filter out a filtered connected domain by using a preset threshold, and the filtered connected domain is used as a position of the moving target. Finally, the filtered connected domain is projected onto the target optical flow map, and the speed of the moving target in the filtered connected domain can be calculated more accurately.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a moving target detection method, electronic device, and storage medium. Background Technology

[0002] With the continuous development of technology, terminal devices, represented by mobile phones and tablets, are increasingly used in people's lives and work, bringing great convenience. For example, people can use these devices to take photos, make video calls, or record videos.

[0003] However, in scenarios where users use terminal devices to record videos or take photos of moving targets (such as vehicles driving on the road), the main subject being focused on is constantly moving, which easily leads to the problem of shooting moving targets out of focus. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a moving target detection method, electronic device, and storage medium, with the aim of more accurately detecting the position and speed of a moving target, enabling precise focus tracking based on the detected position and speed, thereby resolving the problem of capturing moving targets as out of focus.

[0005] Firstly, this application provides a moving target detection method, which includes: a terminal device first responding to a user-triggered shooting operation by acquiring consecutive first target images and second target images (e.g., two consecutive frames containing images of a moving car), and performing keypoint registration processing on the first and second target images to obtain registered first target images and registered second target images. Then, based on the registered first and second target images, a target optical flow map is calculated. After binarizing the target optical flow map, a preliminary connected component containing the moving target in the obtained optical flow binarized image is determined. Next, inter-frame difference and binarization processing is performed on the registered first and second target images to obtain a difference binarized image, and the previously obtained preliminary connected component is projected onto the difference binarized image to filter out the filtered connected component using a preset threshold as the location region of the moving target. Furthermore, the filtered connected component can be projected onto the target optical flow map to more accurately calculate the velocity of the moving target within the filtered connected component.

[0006] As can be seen, in the above-mentioned moving target detection method, when a user holds a mobile phone or other terminal device to take a picture of a moving target (such as a moving car) (or the mobile phone or other terminal device is under a tripod to take a picture of a moving target), the terminal device in this embodiment first performs precise registration of key points on two consecutive frames of target images, then calculates the optical flow connected components and inter-frame differences (auxiliary information) of the two precisely registered target images, and then, through comprehensive processing of the two, more accurately determines the detection result of the moving target (including position and speed). Thus, the focus tracking algorithm can be used to achieve precise focus tracking of the moving target based on the detection result of the moving target (including position and speed), optimize the focus tracking result, and thus solve the problem of shooting moving targets (such as moving cars) out of focus, thereby improving the user's shooting experience.

[0007] In one possible implementation, keypoint registration processing is performed on the first target image and the second target image to obtain registered first target image and registered second target image. This can include: denoising the first target image and the second target image using a mean filtering algorithm to obtain denoised first target image and denoised second target image; detecting keypoints in the denoised first target image and denoised second target image using a preset keypoint detection algorithm to obtain keypoints contained in each of the denoised first target image and denoised second target image; then registering the denoised first target image and denoised second target image using the keypoints contained in each of the denoised first target image and denoised second target image, and calculating the bounding rectangle based on the keypoints as a common region; and cropping the denoised first target image and denoised second target image based on the common region to obtain cropped and registered first target image and cropped and registered second target image, thereby removing the influence of noise and improving the efficiency and accuracy of subsequent image processing.

[0008] In one possible implementation, the preset keypoint detection algorithm can be the ORB keypoint detection algorithm, so as to more accurately detect the keypoints contained in the denoised first target image and the denoised second target image respectively.

[0009] In one possible implementation, the target optical flow map is calculated based on the registered first target image and the registered second target image. This can include: calculating the dense optical flow of the registered first target image and the registered second target image using the FarneBack algorithm to obtain the target optical flow map. This can improve the accuracy of the calculated target optical flow map.

[0010] In one possible implementation, after calculating the target optical flow map based on the registered first target image and the registered second target image, the method further includes: performing grayscale processing on the target optical flow map to obtain a corresponding optical flow grayscale map; and performing binarization processing on the optical flow grayscale map using a preset image binarization algorithm to obtain a corresponding optical flow binarized image. This facilitates a more accurate determination of the initially selected connected component where the moving target is located.

[0011] In one possible implementation, the preset image binarization algorithm can be the Otsu's maximum inter-class variance algorithm to obtain a more accurate optical flow binarized image.

[0012] In one possible implementation, after binarizing the target optical flow map, determining the initial connected component where the moving target is located in the obtained optical flow binarized image can include: performing morphological operations such as dilation and / or erosion on the optical flow binarized image to filter noise, obtaining a smoother optical flow binarized image after morphological operations, and determining the initial connected component where the moving target is located based on the coordinates of the pixels in the morphologically operated optical flow binarized image, so as to improve the accuracy of the initial location result of the moving target.

[0013] In one possible implementation, inter-frame differencing and binarization are performed on the registered first target image and the registered second target image to obtain a differential binarized image. This may include: calculating the inter-frame difference between the registered first target image and the registered second target image, and statistically obtaining the gray-level histogram of the inter-frame difference; binarizing the gray-level histogram using a preset dynamic threshold to obtain a binarized inter-frame difference image; and performing dilation and / or erosion morphological operations on the binarized inter-frame difference image to filter noise, resulting in a smoother inter-frame difference image after morphological operations, which serves as the differential binarized image.

[0014] In one possible implementation, the initially selected connected components are projected onto the differential binarized image to filter out the filtered connected components using a preset threshold, which can be used as the location region of the moving target. This can include: projecting the initially selected connected components onto the differential binarized image, calculating the density of difference points falling within the initially selected connected components, filtering out the initially selected connected components with a density less than a preset threshold, and selecting the filtered connected components as the location region of the moving target, thereby improving the accuracy of detecting the location of the moving target.

[0015] In one possible implementation, the preset threshold can be 0.5.

[0016] In one possible implementation, projecting the filtered connected components onto the target optical flow map and calculating the velocity of the moving target within the filtered connected components can include: projecting the filtered connected components onto the target optical flow map and calculating the average optical flow of all points falling within the filtered connected components as the velocity of the moving target within the filtered connected components, thereby improving the accuracy of detecting the motion velocity of the moving target.

[0017] Secondly, this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor is used to call and execute the computer program to implement the moving target detection method described in any one of the first aspects above.

[0018] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when run by a processor of an electronic device, is used to implement the moving target detection method described in any one of the first aspects above.

[0019] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the moving target detection method as described in any one of the first aspects. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0021] Figure 2 A schematic diagram of an electronic device provided in an embodiment of this application;

[0022] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0023] Figure 4 A flowchart of the moving target detection method provided in the embodiments of this application;

[0024] Figure 5 Example diagrams of key point registration processing performed on a first target image and a second target image to obtain a registered first target image and a registered second target image provided in the embodiments of this application;

[0025] Figure 6 An example diagram illustrating the process of determining the initially selected connected component where the moving target is located in the optical flow binarized image provided in this application embodiment;

[0026] Figure 7 An example diagram illustrating the process of determining the location region of a moving target as provided in this application embodiment;

[0027] Figure 8An example diagram illustrating the process of determining the velocity of a moving target as provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0031] To enable those skilled in the art to better understand the solution of this application, the application scenario of the technical solution of this application will be described first.

[0032] See Figure 1 The illustration shows a scenario diagram provided by an embodiment of this application.

[0033] In this example scenario, a user is holding a mobile phone to take a picture. It's worth noting that the phone's camera display will feature various function buttons to enhance the shooting effect, allowing users to click and meet their individual shooting needs, such as adjusting the image ratio, turning on the flash, etc. Figure 1As shown, when a user holds a mobile phone to take a picture of a moving car, since the main subject of the focus is the constantly moving car, in order to improve the image sharpness, the user can click the "Motion Focus" button on the phone's camera display to use multiple focus frames (such as...) in a series of consecutive frames. Figure 1 Focus on the moving car (the eight small squares above the car in the upper right corner).

[0034] However, currently, when users use terminal devices to photograph moving targets, such as Figure 1 When a user takes a picture of a moving car with their mobile phone, they can use multiple focus-tracking frames (such as...) in a timely manner. Figure 1 The eight small squares above the car in the upper right corner are focused on the moving car, but it's still easy to capture moving targets as out of focus, such as... Figure 1 The issue of blurry photos of cars in the images reduces the user's shooting experience.

[0035] To overcome the above technical problems, this application provides a moving target detection method, electronic device, and storage medium. These methods can more accurately detect the position and speed of moving targets, enabling precise focus tracking based on the detected position and speed. This solves the problem of out-of-focus shots of moving targets, thereby improving the user's shooting experience.

[0036] The moving target detection method provided in this application embodiment can be applied to electronic devices (i.e., terminal devices) such as mobile phones, tablets, personal digital assistants (PDAs), desktop, laptop, and notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices.

[0037] To enable those skilled in the art to better understand the frame delivery and display time anchoring method provided in this application, the hardware architecture and software system architecture of the electronic device will be described in detail below.

[0038] See Figure 2 It shows a schematic diagram of the electronic device provided in the embodiments of this application.

[0039] like Figure 2 As shown, the electronic device 200 may include a processor 210, a mobile communication module 220, a wireless communication module 230, a display screen 240, an internal memory 241, a camera 242, an audio module 243, a speaker 243A, a receiver 243B, a microphone 243C, a headphone jack 243D, an antenna group 1, and an antenna group 2.

[0040] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0041] Processor 210 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. For example, it can determine camera parameter acquisition instructions based on a received user shooting request to detect the position and speed of the moving target being photographed.

[0042] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0043] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0044] Internal memory 241 can be used to store computer executable program code, including instructions. Internal memory 241 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound capture, image capture, etc.), etc. The data storage area may store data created during the use of electronic device 200 (such as audio data, image data, etc.). Furthermore, internal memory 241 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of electronic device 200 by running instructions stored in internal memory 241 and / or instructions stored in memory located within the processor.

[0045] In some embodiments, the internal memory 241 stores instructions for executing the moving target detection method. The processor 210 can execute the instructions stored in the internal memory 241 to respond to a user-triggered shooting operation on the terminal device by continuously capturing two frames of images (defined as a first target image and a second target image). Then, keypoint registration processing is performed on the first and second target images to obtain registered first and second target images. Next, a target optical flow map is calculated based on the registered first and second target images. After binarizing the target optical flow map, a preliminary connected component containing the moving target in the obtained optical flow binarized image is determined. Then, inter-frame difference and binarization processing is performed on the registered first and second target images to obtain a differential binarized image. The preliminary connected component is projected onto the differential binarized image to filter out the filtered connected components using a preset threshold, which are then used as the location region of the moving target. The filtered connected components can then be projected onto the target optical flow graph, and the velocity of the moving target within the filtered connected components can be calculated to obtain the detection results of the moving target (including position and velocity).

[0046] Display screen 240 is used to display images, videos, etc., such as images or videos captured containing moving targets, for example. Figure 1 The user's mobile phone display screen can show images or videos captured by the device, including those of a moving car. The display screen 540 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 500 may include one or N displays 240, where N is a positive integer greater than 1.

[0047] Camera 242 is used to capture still images or videos. For example, after a user holds the electronic device 200 in their hand, they can use the camera 242 of the electronic device 200 to capture images containing moving targets, etc. Figure 1When a user holds a mobile phone, they can use the phone's camera to capture images or videos of a moving car. In some embodiments, the electronic device 200 may include one or N cameras 242, where N is a positive integer greater than 1.

[0048] Electronic device 200 implements display functions through a GPU, display screen 240, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0049] Electronic device 200 can implement audio functions through audio module 243, speaker 243A, receiver 243B, microphone 243C, headphone jack 243D, and application processor, such as music playback and recording for voice input and output.

[0050] The audio module 243 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 243 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 243 may be located in the processor 210, or some functional modules of the audio module 243 may be located in the processor 210.

[0051] The speaker 243A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or make hands-free calls through the speaker 243A.

[0052] The receiver 243B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a telephone call or voice message, the receiver 243B can be brought close to the ear to hear the voice.

[0053] Microphone 243C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 243C, inputting the sound signal into microphone 243C. Electronic device 200 may have at least one microphone 243C. In some embodiments, electronic device 200 may have two microphones 243C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 200 may also have three, four, or more microphones 243C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0054] The headphone jack 243D is used to connect wired headphones and does not restrict the standard attributes of the jack.

[0055] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200.

[0056] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 220, wireless communication module 230, modem processor, and baseband processor.

[0057] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0058] The mobile communication module 220 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 200. The mobile communication module 220 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 220 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 220 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 220 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 220 and at least some modules of the processor 210 may be housed in the same device.

[0059] The wireless communication module 230 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 230 can be one or more devices integrating at least one communication processing module. The wireless communication module 230 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 230 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0060] In addition, the electronic device 200 runs an operating system on top of the aforementioned components. Examples include iOS, Android, and Windows operating systems. Applications can be installed and run on this operating system.

[0061] See Figure 3 It shows a schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0062] The software system of electronic device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 200.

[0063] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer (APK), the application framework layer (Framework), the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0064] The application layer can include a series of application packages (APPs). For example... Figure 3 As shown, the application package may include applications for camera, calling, navigation, WLAN, Bluetooth, gallery, etc. When the user's handheld electronic device is used with 200 pairs of moving targets (such as...) Figure 1When taking pictures of a moving car (as shown), the camera application can communicate with the camera-related devices in the camera access interface of the frame layer to request camera functions and obtain image data, etc.

[0065] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example... Figure 3 As shown, the application framework layer can include a window manager, notification manager, resource manager, and camera access interface (including but not limited to camera management, camera devices, etc.). The application framework layer can be used to implement interaction between camera services and camera APIs. That is, it provides a unified interface, enabling different camera hardware to interact with different camera applications. The framework layer also handles many common aspects of camera functionality, such as autofocus, motion focus, and exposure control.

[0066] The window manager is used to manage window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots, among other things.

[0067] The notification manager allows applications to display notifications in the status bar. These can be used to deliver announcements or notifications, and can disappear automatically after a short pause without user interaction. For example, the notification manager can be used to notify users of download completion or message alerts.

[0068] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0069] In this embodiment, the Hardware Abstraction Layer (HAL) provides a set of standard interfaces, enabling the framework layer to communicate with camera hardware from various manufacturers without needing to understand the underlying hardware details. The HAL stores the hardware abstraction layer itself and camera algorithm libraries, etc. It should be noted that the moving target detection algorithm provided in this application is stored in the camera algorithm library of the HAL layer, such as... Figure 3 As shown.

[0070] Among them, the moving target detection algorithm is used to detect moving targets in two consecutive frames of images acquired by the camera (such as...). Figure 1 Images captured by users' mobile phone cameras, including those of moving cars, are processed to more accurately detect moving targets (such as...). Figure 1The system detects the position and speed of a moving car. Specifically, in response to a user-triggered camera capture, after acquiring two consecutive frames (represented by a first target image and a second target image), keypoint registration is first performed on the first and second target images to obtain registered first and second target images. Then, based on the registered first and second target images, a target optical flow map is calculated. After binarizing the target optical flow map, a preliminary connected component containing the moving target is determined in the obtained optical flow binarized image. Next, inter-frame differencing and binarization are performed on the registered first and second target images to obtain a difference binarized image. The preliminary connected component is projected onto the difference binarized image, and a preset threshold is used to filter out the connected components, which are then used as the location region of the moving target. The filtered connected components are then projected onto the target optical flow map, and the speed of the moving target within the filtered connected components is calculated to obtain the detection result of the moving target (including position and speed).

[0071] The hardware layer may include the hardware components of the aforementioned electronic device. For example, Figure 3 The exhibition showcased sensors, image signal processors, digital signal processors, and graphics processors.

[0072] Among them, the sensor is used for image exposure processing, etc.

[0073] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 200 selects a frequency point, the DSP is used to perform Fourier transforms on the frequency energy.

[0074] In this way, through the application layer, framework layer, HAL layer, driver layer and hardware layer, interoperability between different camera applications and hardware devices on the Android platform can be achieved, so as to more accurately detect the position and speed of moving targets.

[0075] The technical solutions involved in the following embodiments can all be implemented in electronic devices with the above-described hardware and software architectures.

[0076] The following section will detail the implementation process of the moving target detection method provided in this application:

[0077] like Figure 4 As shown, the specific implementation process of this moving target detection method may include the following steps S401-S405:

[0078] S401: In response to a user-triggered shooting operation on the terminal device, acquire a first target image and a second target image; the first target image and the second target image are two consecutive frames captured by the terminal device.

[0079] In this embodiment, when a user uses a mobile phone or other terminal device to move a target (such as...) Figure 1 When the car shown is driving, the terminal device will immediately launch the camera APP and start the camera in the background (such as the camera pre-loaded on the terminal device) in response to the user's shooting operation triggered by the terminal device, such as the user's click operation on the "camera" icon on the mobile phone display screen. The camera will capture images containing moving targets and any two consecutive frames captured by the terminal device will be used as target images to be detected, and defined as the first target image and the second target image respectively, in order to execute the subsequent step S402.

[0080] It should be noted that this embodiment does not limit the types of the first target image and the second target image. For example, the target image can be a color image composed of the three primary colors of red (R), green (G), and blue (B), or it can be a grayscale image, etc. It should also be noted that subsequent embodiments will describe the first target image and the second target image as RGB images. Furthermore, this application does not limit the method of acquiring the first target image and the second target image. For example, two consecutive frames can be acquired from a camera preview stream in a tripod or handheld state as the first target image and the second target image; or, any two consecutive frames from a pre-recorded video stream can be used as the first target image and the second target image, etc.

[0081] S402: Perform key point registration processing on the first target image and the second target image to obtain the registered first target image and the registered second target image.

[0082] After the terminal device acquires the first target image and the second target image to be detected by the camera in step S401, it can further use existing or future key point detection algorithms to detect the key points of the first target image and the second target image. Then, it uses the obtained key points to register the first target image and the second target image to eliminate the influence of camera shake and ensure that the key points of the two target images are aligned one by one, so as to obtain the registered first target image and the registered second target image, which are used to execute the subsequent step S403.

[0083] It should be noted that, in order to improve the accuracy of key point registration between the first target image and the second target image, an optional implementation method is that the implementation process of this step S402 may include the following steps S4021-S4024:

[0084] S4021: Use the mean filtering algorithm to denoise the first target image and the second target image to obtain the denoised first target image and the denoised second target image.

[0085] It should be noted that this application improves the accuracy of image processing and eliminates the influence of noise. After obtaining the first target image and the second target image, existing or future mean filtering algorithms can be used to denoise the first target image and the second target image, making the image smoother, and obtaining the denoised first target image and the denoised second target image, which are then used to perform the subsequent step S4022.

[0086] For example: Figure 5 As shown, assume the first target image and the second target image are as follows: Figure 5 In the two images shown in (a), where the moving target to be detected is a hand, the mean filtering algorithm can be used to denoise both images separately to obtain the following results: Figure 5 The two images shown in (b) are shown in the middle.

[0087] S4022: Use a preset key point detection algorithm to perform key point detection on the denoised first target image and the denoised second target image to obtain the key points contained in each of the denoised first target image and the denoised second target image.

[0088] After obtaining the denoised first target image and the denoised second target image through step S4021, the terminal device can further use a preset key point detection algorithm to perform key point detection on the first target image and the second target image to obtain the key points contained in the denoised first target image and the denoised second target image (referring to special points that represent key information of the denoised first target image and the denoised second target image), which are then used to execute the subsequent step S4023.

[0089] The specific content of the preset key point detection algorithm is not limited and can be set according to the actual situation and experience. For example, one possible implementation is to set the preset key point detection algorithm to the ORB key point detection algorithm. That is, the ORB key point detection algorithm can be used to detect key points in the denoised first target image and the denoised second target image to obtain the image key points contained in each of them.

[0090] For example: Figure 5 As shown, based on the above example, it is still assumed that the first target image and the second target image are as follows: Figure 5 The two images shown in (a) contain a hand as the moving target. By applying a mean filtering algorithm to denoise both images, the following results can be obtained: Figure 5After denoising the two images shown in (b), keypoint detection algorithms such as ORB can be used to detect keypoints in both images separately, resulting in images as shown in (b). Figure 5 The two images shown in (c) contain key points of the image.

[0091] S4023: Using the key points contained in the denoised first target image and the denoised second target image respectively, perform registration processing on the denoised first target image and the denoised second target image, and calculate the bounding rectangle based on the key points as the common area.

[0092] After obtaining the denoised first target image and the denoised second target image in step S4022, the terminal device further performs registration processing on the key points contained in each of the denoised first target image and the denoised second target image to ensure that the key points of the two images are aligned one-to-one and that the positions of the moving targets are as close as possible to each other. This ensures that the moving target to be detected is located as centrally as possible in the image before proceeding to the subsequent step S4024.

[0093] The specific content of the registration algorithm is not limited and can be set according to the actual situation and experience.

[0094] For example: Figure 5 As shown, based on the above example, it is still assumed that the first target image and the second target image are as follows: Figure 5 The two images shown in (a) have a moving target to be detected, a hand. The mean filtering algorithm is used to denoise both images, resulting in the following image: Figure 5 The two denoised images shown in (b) are compared with the keypoint detection results obtained by performing keypoint detection on the two images using the ORB keypoint detection algorithm. Figure 5 After the two images containing key points shown in (c), we can use... Figure 5 In section (c), key points are used to register the two images, and the bounding rectangle is calculated based on the key points as the common region, resulting in the following: Figure 5 The two images shown in (d) and their common area indicated by the box.

[0095] S4024: Based on the common area, crop the first target image and the second target image after denoising to obtain the first target image and the second target image after cropping and registration.

[0096] After obtaining the common area of ​​the denoised first target image and the denoised second target image through step S4023, the terminal device can use the common area to perform cropping processing on the denoised first target image and the denoised second target image to obtain the cropped and registered first target image and the cropped and registered second target image, which are used as the final registered first target image and registered second target image to perform subsequent step S403 to further remove the influence of noise and improve the efficiency and accuracy of subsequent image processing.

[0097] For example: Figure 5 As shown, based on the above example, it is still assumed that the first target image and the second target image are as follows: Figure 5 The two images shown in (a) have a moving target to be detected, a hand. The mean filtering algorithm is used to denoise both images, resulting in the following image: Figure 5 The two denoised images shown in (b) are compared with the keypoint detection results obtained by performing keypoint detection on the two images using the ORB keypoint detection algorithm. Figure 5 The two images containing image keypoints shown in (c) and the data using... Figure 5 The key points of the two images in (c) are used to register the two images, and the bounding rectangle is calculated based on the key points as the common area, resulting in the following: Figure 5 After examining the two images shown in (d) and their common area represented by a box, the common area can be used for cropping to obtain the image shown in (d). Figure 5 The two images shown in (e) are the first and second target images after registration.

[0098] S403: Based on the registered first target image and the registered second target image, calculate the target optical flow map; and after binarizing the target optical flow map, determine the initial connected component where the moving target is located in the obtained optical flow binarized image.

[0099] After obtaining the registered first target image and the registered second target image in step S402, the terminal device can further utilize existing or future optical flow algorithms (the specific content is not limited) to perform optical flow calculations on the registered first target image and the registered second target image, obtaining their corresponding color optical flow maps, which are defined as the target optical flow map. This target optical flow map is a color RGB image, where the brightness represents the speed of the moving target. Then, after binarizing the target optical flow map, the initially selected connected components containing the moving target in the obtained binary optical flow image can be determined for subsequent step S404.

[0100] Specifically, one possible implementation is to use the FarneBack algorithm to calculate the dense optical flow of the registered first target image and the registered second target image in order to improve the accuracy of the calculated target optical flow map, thereby obtaining the target optical flow map.

[0101] The FarneBack algorithm refers to a motion estimation algorithm based on all pixels in two consecutive frames of images. It achieves optical flow tracking by using the displacement vectors of all pixels in the two frames. It has good tracking performance for moving targets. Specifically, the main implementation process of the FarneBack algorithm is as follows: The coordinate position of each pixel is expanded into a polynomial using the neighborhood information (the weights are determined by the pixel values ​​and positions of neighboring pixels). This results in a polynomial with the original coordinates (x0, y0) as the independent variable and the new coordinates (x, y) as the dependent variable. Substituting the coordinate data, the displacement (dx, dy) of the pixel in the x and y directions is calculated. This yields the displacement vector of each pixel in the two frames, including amplitude and phase. If the amplitude and phase information of each pixel's displacement vector are converted into H, S, and V three-channel information, the motion of moving objects can be observed intuitively in consecutive frames, thus achieving dense optical flow tracking of moving objects.

[0102] Furthermore, after obtaining the target optical flow map, it can first be converted to grayscale to obtain the corresponding optical flow grayscale map. The specific calculation formula is as follows:

[0103]

[0104]

[0105]

[0106] Where g represents the gray level of the optical flow grayscale image; ω represents the current angular velocity; V y V represents the velocity of the current point along the y-axis. x V represents the velocity of the current point along the x-axis. max and V min These represent the maximum and minimum speeds of the current frame, respectively.

[0107] Then, a preset image binarization algorithm can be used to binarize the optical flow grayscale image to obtain the corresponding optical flow binarized image. The specific content of the preset image binarization algorithm is not limited and can be set according to actual conditions and empirical values. For example, one optional implementation is to set the preset image binarization algorithm to the Otsu's algorithm (Maximum Inter-Class Variance Method), that is, the OTSU algorithm can be used to binarize the optical flow grayscale image to obtain the corresponding optical flow binarized image.

[0108] For example: Figure 6 As shown, based on the above examples, assume Figure 6 The two images shown in (a) are Figure 5 The registered first target image and the registered second target image shown in (e) can be used to calculate their corresponding target optical flow maps using the Farneback algorithm, as follows: Figure 6 The image shown in (b) can be further processed by grayscale conversion of the optical flow map of the target to obtain... Figure 6 The optical flow grayscale image shown in (c) is then binarized using the OTSU algorithm to obtain... Figure 6 The optical flow binarized image shown in (d) is shown in the middle.

[0109] Based on this, an optional implementation is that, after binarizing the target optical flow map to obtain an optical flow binarized image, in order to improve the accuracy of the image processing results, morphological operations such as dilation and / or erosion can be further performed on the obtained optical flow binarized image (the specific operation process and operation method are not limited) to obtain a morphologically operated optical flow binarized image. Based on the coordinates of the pixels in the morphologically operated optical flow binarized image, the initial connected component where the moving target is located can be determined. For example, the coordinate set of white pixels in the morphologically operated optical flow binarized image can be used to construct the initial connected component where the moving target is located.

[0110] For example: Figure 6 As shown, based on the above examples, we still assume... Figure 6 The two images shown in (a) are Figure 5 The first and second registered target images are shown in (e), and their corresponding target optical flow maps are calculated using the Farneback algorithm, as shown below. Figure 6 The image shown in (b) and the grayscale processing of the optical flow map of the target are obtained. Figure 6 The optical flow grayscale image shown in (c) is then binarized using the OTSU algorithm to obtain... Figure 6 After binarizing the optical flow image shown in (d), morphological operations such as dilation and / or erosion can be performed on the binarized image to obtain... Figure 6 The optical flow binarized image after morphological operations shown in (e) is then used to define the location of the initially selected connected component containing the coordinate set of white pixels in the morphologically operated optical flow binarized image. Figure 6 As shown in (f).

[0111] S404: Perform inter-frame difference and binarization processing on the registered first target image and the registered second target image to obtain a differential binarized image; and project the initially selected connected components onto the differential binarized image to filter out the filtered connected components using a preset threshold as the location region of the moving target.

[0112] After obtaining the registered first target image and the registered second target image in step S402, the terminal device not only needs to process them for optical flow, grayscale, binarization, etc., to determine the initially selected connected component where the moving target is located, but also needs to perform inter-frame difference (the specific algorithm and calculation method are not limited) and binarization processing on the registered first target image and the registered second target image to obtain a difference binarized image; and then project the initially selected connected component onto this difference binarized image, so as to use a preset threshold to filter out the filtered connected component as the location region of the moving target, for subsequent step S405. The specific content of the preset dynamic threshold is not limited and can be set according to actual conditions and empirical values.

[0113] Specifically, one possible implementation is that, in order to improve the recognition accuracy of the location area of ​​the moving target, the implementation process of "performing inter-frame difference and binarization processing on the registered first target image and the registered second target image to obtain a difference binarized image" in step S404 above may include: firstly calculating the inter-frame difference between the registered first target image and the registered second target image, and statistically obtaining the grayscale histogram of the inter-frame difference.

[0114] Inter-frame differencing refers to detecting moving objects in an image by comparing the pixel value differences between adjacent frames. Specifically, the main process of inter-frame differencing is as follows: First, acquire two adjacent frames (such as a registered first target image and a registered second target image) and convert them into grayscale images. Then, perform a difference operation on the two grayscale images to obtain an inter-frame differencing grayscale histogram. In this differencing image, a larger pixel value indicates a greater change in that pixel between the two frames, which may be due to the presence or movement of a moving object.

[0115] Then, the grayscale histogram is binarized using a preset dynamic threshold to obtain the binarized inter-frame difference image.

[0116] It's important to note that to detect moving objects, the grayscale histogram of the inter-frame difference needs to be thresholded. Thresholding compares pixel values ​​in the difference image with a preset threshold; if a pixel value is greater than the threshold, it's considered a moving object. This separates the moving object from the background, achieving moving object detection. Specifically, after obtaining the grayscale histogram of the inter-frame difference, it can be sorted by frequency, and the top two values ​​are used as preset dynamic thresholds. The inter-frame difference image (i.e., the grayscale histogram) is then binarized to obtain the binarized inter-frame difference image.

[0117] Next, morphological operations such as dilation and / or erosion can be performed on the binarized inter-frame difference image to filter noise and obtain the morphologically operated inter-frame difference image, which can be used as the difference binarized image.

[0118] Furthermore, after obtaining the differential binarized image, the initially selected connected component containing the moving target obtained in step S403 can be projected onto the differential binarized image, and the density of difference points falling within the initially selected connected component can be calculated. The specific calculation formula is as follows:

[0119]

[0120] Where ρ represents the density of difference points falling within the initially selected connected domain; d i represents the value of the differential binarized image falling within the connected component; S represents the area of ​​the initially selected connected component.

[0121] Then, initially selected connected components with a density ρ less than a preset threshold can be filtered out, and the filtered connected components can be used as the location region of the moving target. The specific content of the preset threshold is not limited and can be set according to actual conditions and experience. For example, the preset threshold can be set to 0.5, meaning that initially selected connected components with a density ρ less than 0.5 will be filtered out, and initially selected connected components with a density ρ greater than 0.5 will be retained, thus selecting the filtered connected components as the location region of the moving target.

[0122] For example: Figure 7 As shown, based on the above examples, assume Figure 7 The two images shown in (a) are Figure 5 The first and second registered target images shown in (e) can be used to calculate their inter-frame difference, and the gray-level histogram of the inter-frame difference can be obtained statistically, as shown below. Figure 7 As shown in (b), the grayscale histogram is then binarized using a preset dynamic threshold to obtain the binarized inter-frame difference image, as shown below. Figure 7As shown in (c), the binarized inter-frame difference image is then subjected to dilation and / or erosion morphological operations to obtain a morphologically operated inter-frame difference image, which serves as the difference binarized image, as shown. Figure 7 As shown in (d). Further, the initially selected connected components can be projected onto this difference binarized image to filter out the filtered connected components using a preset threshold, which can then be used as the location region of the moving target (i.e., the hand), as shown in (d). Figure 7 As shown in (e).

[0123] S405: Project the filtered connected components onto the target optical flow graph and calculate the velocity of the moving target within the filtered connected components.

[0124] After the terminal device filters out the connected components in step S404 as the location area of ​​the moving target, it can further project the filtered connected components onto the target optical flow map calculated in step S403, and calculate the average optical flow of all points falling within the filtered connected components as the motion velocity of the moving target within the filtered connected components. Thus, it can combine the determined location of the moving target to form the detection result of the moving target (i.e., including the position and velocity of the moving target).

[0125] It should be noted that one possible implementation is that multiple moving targets may exist simultaneously in the first target image and the second target image, such as... Figure 1 There may be multiple cars driving in the scene, or there may be pedestrians walking, etc. Since the moving targets tracked by the same terminal device at the same time are often the fastest moving targets in the captured image, after calculating the motion speed of each moving target in the first target image and the second target image through the above step S405, as follows... Figure 8 As shown, the speeds of various moving targets can be sorted in descending order, and the moving target with the highest speed can be selected as the subject of focus. By using a tracking focus algorithm, the moving target can be accurately tracked based on its detected position and speed, thereby solving the problem of shooting the moving target out of focus.

[0126] In this way, when the user holds a mobile phone or other terminal device to observe a moving target (such as...) Figure 1 When taking pictures of a moving car (or when a mobile phone or other terminal device is positioned under a tripod to photograph a moving target), the terminal device can more accurately detect the moving target (such as a moving car) by executing the steps S401-S405 described above. Figure 1 The position and speed of the car (which is currently moving) are used to determine the location and speed of the moving target (such as...). Figure 1 The position and speed of a moving car (such as a car in motion) are used to track moving targets (such as cars in motion). Figure 1Precise focus tracking of moving targets (such as cars in motion) and optimized focus results, thereby solving the problem of accurately tracking moving targets (such as cars in motion). Figure 1 This addresses the issue of out-of-focus shots of cars in motion, thereby improving the user's shooting experience.

[0127] Furthermore, this application also provides an electronic device (i.e., a terminal device). For details regarding the hardware structure and software framework of the electronic device, please refer to [link to relevant documentation]. Figure 2 and Figure 3 The corresponding explanation is as follows: The electronic device includes a memory and a processor. The memory stores a computer program, and the processor calls and executes the computer program to implement the moving target detection method provided in the above description.

[0128] This application also provides a computer-readable storage medium storing a computer program thereon, which, when run by the processor of a terminal device, is used to implement the moving target detection method provided above.

[0129] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A moving object detection method characterized by, Applied to a terminal device, the method includes: In response to a user's shooting operation triggered by the terminal device, a first target image and a second target image are acquired; the first target image and the second target image are two consecutive frames captured by the terminal device. Key point registration processing is performed on the first target image and the second target image to obtain the registered first target image and the registered second target image. Based on the registered first target image and the registered second target image, a target optical flow map is calculated; after binarizing the target optical flow map to obtain an optical flow binarized image, morphological operations of dilation and / or erosion are performed on the optical flow binarized image to obtain a morphologically operated optical flow binarized image; and then, based on the coordinates of the pixels in the morphologically operated optical flow binarized image, the initially selected connected component where the moving target is located is determined. The registered first target image and the registered second target image are subjected to inter-frame difference and binarization processing to obtain a difference binarized image; the initially selected connected components are projected onto the difference binarized image, the density of difference points falling within the initially selected connected components is calculated, and the initially selected connected components with a density less than a preset threshold are filtered out to select the filtered connected components as the location region of the moving target; The filtered connected components are projected onto the target optical flow graph, and the velocity of the moving target within the filtered connected components is calculated.

2. The method of claim 1, wherein, The step of performing key point registration processing on the first target image and the second target image to obtain the registered first target image and the registered second target image includes: The first target image and the second target image are denoised using the mean filtering algorithm to obtain the denoised first target image and the denoised second target image. A preset key point detection algorithm is used to detect key points in the denoised first target image and the denoised second target image to obtain the key points contained in each of the denoised first target image and the denoised second target image. Using the key points contained in the denoised first target image and the denoised second target image respectively, the denoised first target image and the denoised second target image are registered, and the bounding rectangle is calculated based on the key points as a common area; The first and second target images after denoising are cropped based on the common area to obtain the first and second target images after cropping and registration.

3. The method of claim 2, wherein, The preset key point detection algorithm is the ORB key point detection algorithm.

4. The method of claim 1, wherein, The step of calculating the target optical flow map based on the registered first target image and the registered second target image includes: The dense optical flow of the registered first target image and the registered second target image is calculated using the FarneBack algorithm to obtain the target optical flow map.

5. The method of claim 1, wherein, After calculating the target optical flow map based on the registered first target image and the registered second target image, the method further includes: The target optical flow map is converted to grayscale to obtain the optical flow grayscale map corresponding to the target optical flow map; The optical flow grayscale image is binarized using a preset image binarization algorithm to obtain the optical flow binarized image corresponding to the optical flow grayscale image.

6. The method of claim 5, wherein, The preset image binarization algorithm is the Otsu algorithm, which is the maximum inter-class variance method.

7. The method of claim 1, wherein, The step of performing inter-frame differencing and binarization processing on the registered first target image and the registered second target image to obtain a differencing binarized image includes: Calculate the inter-frame difference between the registered first target image and the registered second target image, and obtain the gray-level histogram of the inter-frame difference. The grayscale histogram is binarized using a preset dynamic threshold to obtain a binarized inter-frame difference image; The binarized inter-frame difference image is subjected to dilation and / or erosion morphological operations to obtain a morphologically operated inter-frame difference image, which is used as the difference binarized image.

8. The method of claim 1, wherein, The preset threshold is 0.

5.

9. The method of claim 1, wherein, The step of projecting the filtered connected component onto the target optical flow map and calculating the velocity of the moving target within the filtered connected component includes: The filtered connected component is projected onto the target optical flow map, and the average optical flow of all points falling within the filtered connected component is calculated as the velocity of the moving target within the filtered connected component.

10. An electronic device, comprising: The electronic device includes a memory and a processor, the memory storing a computer program, and the processor being used to invoke and execute the computer program to implement the method of any one of claims 1-9.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by an electronic device, implements the method of any one of claims 1-9.

Images