Method and related device for detecting moving objects

Abstract

Embodiments of the present application provide a kind of moving object detection method and related equipment, the method comprises: using shooting device obtains two adjacent image frames;Calculate the initial optical flow map between the two image frames;Initial optical flow map is carried out denoising processing, obtain target optical flow map;Target optical flow map is divided into multiple sub-regions, and the first speed corresponding to each sub-region is determined according to the optical flow value of each pixel point in each sub-region;According to the first speed, determine multiple target regions in target optical flow map, and determine the second speed corresponding to each target region;According to target region, determine the region where the first image frame in the two image frames exists moving object, and according to the second speed, determine the motion speed of moving object.The embodiments of the present application can accurately determine the region where target optical flow map exists moving object, avoid mistaking noise as moving object, improve the calculation accuracy of the motion speed of moving object.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of terminals, and in particular to a moving object detection method and related equipment. BACKGROUND

[0002] In the field of computer vision, moving object detection technology can be used to identify and track objects in motion from video sequences or images. When applied to real-time shooting scenes, through moving object detection, it can be perceived whether there is a moving object in the scene being shot and related information of the moving object (such as the image region where the moving object is located, the motion speed of the moving object, etc.), so that the exposure parameters can be determined according to the related information to adjust the exposure strategy of the shooting device in real time, and the image clarity of the shot image can be improved.

[0003] In related technologies, the image processing function of the chip platform of the terminal device itself is usually used to determine the optical flow map between two adjacent image frames shot by the shooting device, so as to determine whether there is a moving object in the current scene and the related information of the moving object according to the optical flow map. However, since the optical flow map obtained in related technologies has a large amount of noise, it leads to problems such as mis-detection of noise as a moving object, low accuracy of the calculated speed of the moving object, and low quality of the shot image due to the above problems. SUMMARY

[0004] In view of the above, it is necessary to provide a moving object detection method and related equipment, which can solve the problems of mis-detection of moving objects caused by a large amount of noise in the optical flow map, low accuracy of the calculated speed of the moving object, and low quality of the shot image due to the above problems.

[0005] In a first aspect, the present application provides a moving object detection method applied to a terminal device, the method comprising: in response to a shooting operation of a user, acquiring two adjacent image frames by using a shooting device of the terminal device; calculating an initial optical flow map between the two image frames by using a preset optical flow estimation algorithm; performing denoising processing on the initial optical flow map to obtain a target optical flow map; dividing the target optical flow map into a plurality of sub-regions, and determining a first speed corresponding to each sub-region according to the optical flow value of each pixel point in each sub-region; determining a plurality of target regions in the target optical flow map according to the first speed, and determining a second speed corresponding to each target region, wherein each target region includes at least one sub-region; determining a region in which a moving object exists in a first image frame of the two image frames according to the target region, and determining a motion speed of the moving object according to the second speed.

[0006] By the technical solution, the optical flow estimation algorithm of the chip of the terminal device can be called when an image is captured to obtain an initial optical flow map between two adjacent image frames; the initial optical flow map is denoised to gradually remove noise and abnormal optical flow in the initial optical flow map; by dividing the target optical flow map into a plurality of sub-regions, the connected regions in which a moving object may exist in the target optical flow map can be determined according to the optical flow values in the target optical flow map; by screening the connected regions, the regions in which a moving object exists in the target optical flow map can be accurately determined, and the static noise regions are avoided from being mistakenly detected as moving objects, and the accuracy of calculating the moving speed of the moving object is improved.

[0007] In a possible implementation, each pixel point in the initial optical flow map includes at least a first channel and a second channel, where the first channel includes a first displacement value in a first direction, and the second channel includes a second displacement value in a second direction; and the optical flow value of each pixel point in the initial optical flow map includes the first displacement value, the second displacement value, and a third displacement value corresponding to a displacement vector determined according to the first displacement value and the second displacement value.

[0008] By the technical solution, the displacement information of each pixel point in the two adjacent image frames can be determined according to the numerical values of each channel in the initial optical flow map, where the displacement information includes displacement values in two directions, and a displacement value of a displacement vector determined according to the displacement values in the two directions, which facilitates subsequent processes to determine the moving regions in the optical flow map according to the displacement values.

[0009] In a possible implementation, the denoising processing of the initial optical flow map to obtain a target optical flow map includes: based on the optical flow value of each pixel point in the initial optical flow map and a preset optical flow value range, the initial optical flow map is denoised to obtain an updated optical flow map; based on the first image frame and the second image frame in the two image frames, a denoised difference map is obtained by using an improved difference algorithm; and the updated optical flow map is denoised by using the denoised difference map to obtain the target optical flow map.

[0010] By the technical solution, the noise in the initial optical flow map can be preliminarily removed according to the preset optical flow value range to obtain an updated optical flow map; a denoised difference map of the two image frames is obtained according to the improved difference algorithm, and the updated optical flow map is further denoised by using the denoised difference map, which can highlight the difference between the moving and static pixel points in the updated optical flow map, and the optical flow noise that interferes with the distinction between the moving and static regions can be effectively removed by resetting the optical flow values in the updated optical flow map.

[0011] In a possible implementation, the optical flow value of each pixel point in the initial optical flow map includes a third displacement value, and the denoising processing of the initial optical flow map based on the optical flow value of each pixel point in the initial optical flow map and a preset optical flow value range includes: if the third displacement value of any pixel point in the initial optical flow map is not within the optical flow value range, updating the optical flow value of the any pixel point to a preset first value; or if the third displacement value of the any pixel point is within the optical flow value range, maintaining the optical flow value of the any pixel point.

[0012] According to the technical solution, the preset optical flow value range can be used to remove slight noise caused by shaking of a terminal device when capturing an image and abnormal large noise caused by an optical flow estimation algorithm.

[0013] In a possible implementation, the improved difference algorithm used for obtaining the denoised difference map based on the first image frame and the second image frame of the two image frames includes: determining a first gray value of each pixel point in the first image frame and a second gray value of the corresponding pixel point in the second image frame; performing difference calculation on the first image frame and the second image frame based on the first gray value and the second gray value of each pixel point to obtain an initial difference image between the first image frame and the second image frame; performing binarization processing on the initial difference image to obtain a binarization image; and performing post-processing operation on the binarization image to obtain the denoised difference map.

[0014] According to the technical solution, the improved difference algorithm can be used to perform difference calculation on the two image frames according to the gray values of each corresponding pixel point in the two image frames, thereby highlighting the difference between the moving pixel points and the stationary pixel points in the updated optical flow map to obtain an initial difference image of the two image frames. Then, the moving area and the stationary area in the initial difference image can be distinguished through binarization processing. Further, white noise in the binarization image can be removed, and the edges and contours of the moving area and the stationary area in the binarization image can be enhanced through post-processing operation such as image erosion processing and image dilation processing, so that the edges of different areas are smoother and more continuous.

[0015] In a possible implementation, the formula used by the improved difference algorithm includes:

[0016]

[0017] wherein, D n (x,y) represents the gray value of the pixel point with coordinates (x,y) in the initial difference image, f n (x,y) represents the gray value of the pixel point with coordinates (x,y) in the first image frame f na first gray value of a pixel point with a coordinate of (x, y) in the first image frame f n-1 (x, y) represents a second gray value of a pixel point with a coordinate of (x, y) in the second image frame f n-1 (x, y) represents a second gray value of a pixel point with a coordinate of (x, y) in the second image frame f

[0018] By the above technical solution, compared with the method of subtracting the absolute value of the gray values of the corresponding pixel points in the two image frames in the traditional difference algorithm, the formula used in the above difference algorithm can focus on distinguishing the moving and static pixel points, and therefore, compared with the difference graph with a large amount of noise obtained by the traditional difference algorithm, the initial difference image obtained by using the above difference algorithm can more accurately distinguish the moving object and the static background.

[0019] In a possible implementation, the binarization processing of the initial difference image includes: if a gray value of any pixel point in the initial difference image is less than a preset gray threshold, updating the gray value of the any pixel point to a preset first value; or, if the gray value of the any pixel point is greater than or equal to the gray threshold, updating the gray value of the any pixel point to a preset second value.

[0020] By the above technical solution, the gray value of the pixel point in the initial difference image can be updated to a preset value, so that the moving pixel point and the static pixel point can be accurately identified according to the obtained binarization image.

[0021] In a possible implementation, the denoising processing of the updated optical flow graph by using the denoised difference graph includes: performing a pixel-by-pixel AND operation on the denoised difference graph and the updated optical flow graph, and updating the optical flow value of each pixel point in the updated optical flow graph according to a result of the AND operation.

[0022] In a possible implementation, the updating of the optical flow value of each pixel point in the updated optical flow graph according to the result of the AND operation includes: if a gray value of any pixel point in the denoised difference graph is equal to a preset first value, updating the optical flow value of the pixel point corresponding to the any pixel point in the updated optical flow graph to the first value; or, if the gray value of the any pixel point is equal to a preset second value, maintaining the optical flow value of the pixel point corresponding to the any pixel point in the updated optical flow graph.

[0023] By the above technical solution, the corresponding optical flow value in the updated optical flow graph can be reset to 0 according to the static pixel point indicated by the denoised difference graph, and the optical flow value of the corresponding pixel point in the updated optical flow graph is maintained unchanged according to the moving pixel point indicated by the denoised difference graph, so as to further eliminate the noise caused by the abnormal optical flow value in the updated optical flow graph.

[0024] In a possible implementation, the optical flow value of each pixel point in each sub-region includes a first displacement value and a second displacement value, and the method of dividing the target optical flow map into a plurality of sub-regions and determining a first speed corresponding to each sub-region according to the optical flow value of each pixel point in each sub-region includes: determining a first average value of the first displacement value of all pixel points in each sub-region and a second average value of the second displacement value of all pixel points; and determining the first speed based on the first average value and the second average value.

[0025] By the technical solution, the first speed corresponding to each sub-region can be preliminarily determined according to the average value of the first displacement value of all pixel points in each sub-region in the first direction and the average value of the second displacement value of all pixel points in the second direction, so that the region where the moving object is located can be screened out from the target optical flow map in a subsequent process.

[0026] In a possible implementation, the method of determining a plurality of target regions in the target optical flow map according to the first speed includes: regarding any sub-region as a target sub-region if the first speed of the any sub-region is greater than a preset speed threshold; and determining a connected region where each target sub-region is located based on a connected graph algorithm, and regarding the connected region as the target region, where each connected region includes at least one target sub-region.

[0027] By the technical solution, the target sub-region whose first speed exceeds the preset speed threshold can be determined according to the first speed of each sub-region, and the connected region where the moving object exists can be determined as the target region by connecting the target sub-regions.

[0028] In a possible implementation, before determining the connected region where each target sub-region belongs to based on the connected graph algorithm, the method further includes performing noise filtering on the target optical flow map, including: updating the optical flow value of an isolated target sub-region at a non-boundary position in the target optical flow map to a preset first value, where the isolated target sub-region includes a target sub-region that is not adjacent to other target sub-regions.

[0029] By the technical solution, the isolated target sub-region at the non-boundary position in the target optical flow map can be updated to a non-moving region, so that further noise filtering on the target optical flow map is implemented.

[0030] In a possible implementation, the method further includes: cleaning an abnormal value in the first speed of all sub-regions in each target region, determining a third speed of each target region according to the average value of the cleaned first speed; and determining a plurality of sliding window average values corresponding to the first speed of all sub-regions in each target region based on a sliding window algorithm, and determining a fourth speed of each target region according to the maximum value in the plurality of sliding window average values.

[0031] By the technical solution, the smaller value and the larger value in the first speed can be removed based on the principle of normal distribution, and the average value of the first speed after filtering out the larger value and the smaller value is obtained as the third speed of the target region; and the average value corresponding to the larger value in the first speed can be obtained as the fourth speed of the target region by using the sliding window algorithm.

[0032] In a possible implementation, the method further includes: determining the shooting state of the shooting device according to the target optical flow map, including: determining a ratio of a number of all target sub-regions in the target optical flow map to a total number of all sub-regions in the target optical flow map, wherein the target sub-region represents a sub-region with a first speed greater than a preset speed threshold; if the ratio is greater than a preset proportion threshold, determining that the shooting state of the shooting device is a motion state; or, if the ratio is less than or equal to the proportion threshold, determining that the shooting state of the shooting device is an approximately static state.

[0033] By the technical solution, the shooting state of the shooting device of the two image frames corresponding to the target optical flow map can be determined according to the proportion of the motion sub-region in the target optical flow map, for example, if the proportion of the motion sub-region in the target optical flow map is large, it can be considered that the existence of more motion sub-regions in the target optical flow map is caused by the motion of the shooting device, and thus it can be considered that the shooting device is in a motion state; or, if the proportion of the motion sub-region in the target optical flow map is small, it can be considered that the existence of less motion sub-region in the target optical flow map is caused by the motion object in the shooting scene, and thus it can be considered that the shooting device is in an approximately static state.

[0034] In a possible implementation, the method further includes: determining the second speed based on the shooting state of the shooting device, including: if the shooting state is the approximately static state, and the fourth speed of each target region is less than a product of a preset third numerical value and the third speed of each target region, determining the second speed according to the fourth speed; or, if the shooting state is the approximately static state, and the fourth speed of each target region is greater than or equal to the product of the third numerical value and the third speed of each target region, determining the second speed according to the third speed; or, if the shooting state is the motion state, and the fourth speed of each target region is less than a product of a preset fourth numerical value and the third speed of each target region, determining the second speed according to the fourth speed; or, if the shooting state is the motion state, and the fourth speed of each target region is greater than or equal to the product of the fourth numerical value and the third speed of each target region, determining the second speed according to the third speed.

[0035] By the technical solution, the second speed of the target region can be determined according to the shooting state of the shooting device, the third speed and the fourth speed of the target region, so as to avoid misjudgment of the movement speed of the target region caused by the shooting state of the shooting device, and improve the calculation accuracy of the movement speed of the target region.

[0036] In a possible implementation, the method further includes: performing noise removal processing on the target regions, including: sorting all target regions in descending order of area of the target regions, and selecting a preset number of target regions in front from the sorted sequence, and determining a noise region from the selected target regions.

[0037] In a possible implementation, determining the noise region in the target region includes: determining a corresponding region of each target region in the first image frame or the second image frame, obtaining a traditional difference map between the two image frames of the corresponding region, each pixel point in the traditional difference map corresponding to a difference value; determining an average difference value of the difference values of all pixel points in the traditional difference map corresponding to each target region; determining a difference threshold corresponding to each target region according to the number of sub-regions in each target region; and determining a target region corresponding to an average difference value less than or equal to the corresponding difference threshold as a noise region.

[0038] In a possible implementation, the difference threshold corresponding to each target region is determined according to the following formula:

[0039]

[0040] Wherein, threshold represents the difference threshold, num represents the number of sub-regions in each target region, and a, l, b, g, and k represent preset parameters.

[0041] By the technical solution, the difference threshold corresponding to each target region can be determined according to the number of sub-regions of each target region, so that the noise region in the target region can be determined according to the average difference value of the target difference region in the traditional difference map of the two image frames corresponding to each target region and the difference threshold, and the recognition accuracy of the movement region is improved.

[0042] In a second aspect, the application provides a terminal device, which includes a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the terminal device executes the motion object detection method described above.

[0043] In a third aspect, the present application provides a chip coupled with a memory in a terminal device, and the chip is configured to control a processor of the terminal device to perform the motion object detection method.

[0044] In a fourth aspect, the present application provides a computer storage medium storing program instructions, and when the program instructions are executed on a terminal device, the processor of the terminal device is caused to perform the motion object detection method.

[0045] In addition, the technical effects brought by the second aspect to the fourth aspect can be referred to the description of the method in the method part, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a schematic diagram of a visualized image of an optical flow provided by an embodiment of the present application.

[0047] Figure 2 is a schematic diagram of an initial optical flow map provided by an embodiment of the present application.

[0048] Figure 3 is a software architecture diagram of a terminal device provided by an embodiment of the present application.

[0049] Figure 4 is a flowchart of a motion object detection method provided by an embodiment of the present application.

[0050] Figure 5 is an example diagram of optical flow map denoising processing provided by an embodiment of the present application.

[0051] Figure 6 is an example diagram of a traditional difference map provided by an embodiment of the present application.

[0052] Figure 7 is an example diagram of a denoised difference map provided by an embodiment of the present application.

[0053] Figure 8 is a flowchart of a method of denoising processing on an initial optical flow map provided by an embodiment of the present application.

[0054] Figure 9 is a flowchart of a detailed flow of S302 provided by an embodiment of the present application.

[0055] Figure 10 is a schematic diagram of dividing a target optical flow map into a plurality of sub-regions provided by an embodiment of the present application.

[0056] Figure 11 is a schematic diagram of a motion state matrix corresponding to a target optical flow map provided by an embodiment of the present application.

[0057] Figure 12is a schematic diagram of noise filtering on a target optical flow map provided by an embodiment of the present application.

[0058] Figure 13 is a schematic diagram of a connected region provided by an embodiment of the present application.

[0059] Figure 14 is a flow chart of a method for determining a shooting state of a shooting device provided by an embodiment of the present application.

[0060] Figure 15 is a flow chart of a method for determining a third speed and a fourth speed of a target region provided by an embodiment of the present application.

[0061] Figure 16 is a flow chart of a method for determining a noise region in a target region provided by an embodiment of the present application.

[0062] Figure 17 is an example diagram of a pixel point of a target region provided by an embodiment of the present application.

[0063] Figure 18 is a flow chart of a method for differentially denoising an optical flow provided by an embodiment of the present application.

[0064] Figure 19 is a flow chart of a motion detection method provided by an embodiment of the present application.

[0065] Figure 20 is a flow chart of a method for noise checking on a connected graph provided by an embodiment of the present application.

[0066] Figure 21 is a hardware architecture diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] In the description of the embodiments of the present application, the terms "first", "second", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. Rather, the words "exemplary" or "for example" are used in the sense of presenting a specific example or instance.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0069] In the field of computer vision, moving object detection technology can be used to identify and track moving objects from video sequences or images. When applied to real-time shooting scenarios, moving object detection can detect the presence of moving objects in the scene and related information about them (such as the image region where the moving object is located, the speed of the moving object, etc.). Based on this information, exposure reduction parameters can be determined to adjust the exposure strategy of the shooting device in real time, thereby improving the image sharpness of the captured image.

[0070] In related technologies, optical flow estimation algorithms using the image processing capabilities of the terminal device's own chip platform are typically used to determine the optical flow between two adjacent (representing consecutive frames in a time series) image frames captured by the imaging device. Based on the optical flow, the presence of moving objects (such as vehicles, people, animals, etc.) and related information about the moving objects in the current scene can be determined.

[0071] For details, please refer to Figure 1 The diagram shows a visualization of optical flow (hereinafter referred to as an optical flow graph). Optical flow represents the movement of target pixels in an image due to the movement of moving objects and / or the movement of the imaging device between two adjacent image frames. Optical flow can be used to capture the displacement information of corresponding pixels between two adjacent image frames, such as the speed and direction of movement of each pixel. Since the time interval between two adjacent image frames is very small, for example, 1 / 60 of a second, the optical flow estimation algorithm can calculate the optical flow between two adjacent image frames based on the following physical assumptions: the pixel intensity of the scene remains essentially unchanged between two adjacent image frames, and adjacent pixels have similar motion.

[0072] The optical flow map of two adjacent image frames obtained by the chip platform of the terminal device itself (hereinafter referred to as an initial optical flow map) can be an image with a size of widthxheightxchannel, where width represents the image width of the initial optical flow map (equal to the image width of each image frame), height represents the image height of the initial optical flow map (equal to the image height of each image frame), and channel represents the number of channels included by each pixel point in the initial optical flow map. The unit length of the size of the image can be the size of one pixel point.

[0073] For example, the number of channels can be at least 2, indicating that each pixel point in the initial optical flow map includes at least two channels, one of which can be a displacement vector in the horizontal X-axis direction as shown in FIG. 1A. Figure 1 The other channel can be a displacement vector in the vertical Y-axis direction as shown in FIG. 1B. Figure 1 The displacement vector in FIG. 1A can be determined according to the sum of the displacement vector in FIG. 1A and the displacement vector in FIG. 1B. The displacement vector in FIG. 1A can be determined according to the sum of the displacement vector in FIG. 1A and the displacement vector in FIG. 1B. Figure 1 The displacement value can be equal to the square root of the sum of the squares of the displacement value and the displacement value. The displacement value can be equal to the square root of the sum of the squares of the displacement value and the displacement value. The larger the displacement value is, the greater the motion speed of the object at the corresponding pixel point is. The unit length of the displacement value can be the size of one pixel point.

[0074] For ease of understanding, the displacement vector in FIG. 1A can be understood as: a certain feature point changes from the position of pixel point A in image frame f n-1 to the position of pixel point B in image frame f n, and the obtained vector is the displacement vector in FIG. 1A. n-1 n n n-1 n n

[0075] In actual applications, because the optical flow map obtained in the related art has a large amount of noise, the noise region is mis-detected as a moving object, the accuracy of the calculated motion speed of the moving object is low, and the like, thereby causing the exposure parameter of the shooting device to be inaccurate, and the image quality of the shot image to be low (for example, there are obvious blurred regions).​​​​​​​​​​​​

[0076] Referring to Figure 2 Fig. 1 shows a schematic diagram of an initial optical flow map provided by an embodiment of the present application. As shown in Fig. 1, the white area in the initial optical flow map represents an area with an optical flow value of 0, and the darkness of the gray scale of the pixel points is used to indicate the size of the optical flow value at the corresponding position, with a darker gray scale indicating a larger optical flow value. As can be seen, the lighter gray areas in the initial optical flow map have more small optical flow value noise, and the darker areas in the lower right corner of the initial optical flow map also have areas with different color depths, with the darkest black dot area representing a large optical flow value noise area. Figure 2

[0077] To solve the above problems, an embodiment of the present application provides a moving object detection method, which can call the optical flow estimation algorithm of the chip of the terminal device when an image is captured to obtain an initial optical flow map between two adjacent image frames; the initial optical flow map is denoised to gradually remove the noise and abnormal optical flow in the initial optical flow map; by dividing the target optical flow map into a plurality of sub-regions, the connected regions in the target optical flow map that may contain moving objects can be determined according to the optical flow values in the target optical flow map; by screening the connected regions, the regions in the target optical flow map that contain moving objects can be accurately determined, avoiding the mis-detection of noise regions in a stationary state as moving objects, and improving the accuracy of calculating the moving speed of the moving objects. The moving object detection method is applied in various terminal devices, such as mobile phones, tablet computers, wearable devices, camera devices, computers, etc. The terminal device includes an application processor, which is used to run an operating system, and the software structure of the terminal device is described below. Figure 3 An exemplary description of the software structure of the terminal device is given.

[0078] Referring to Figure 3 Fig. 2 shows a software architecture diagram of a terminal device provided by an embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. For example, the Android system has an application layer 101, a framework layer 102, an Android runtime and a system library 103, a hardware abstraction layer 104, a kernel layer 105, and a hardware layer 106 from top to bottom.

[0079] The application layer 101 can include a series of application packages. For example, the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, device control service, etc.

[0080] ​The framework layer 102 provides an application programming interface (API) and a programming framework for applications of the application layer. The application framework layer includes some pre-defined functions. For example, the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0081] The window manager is used to manage window programs. The window manager can acquire a display screen size, determine whether there is a status bar, lock a screen, and intercept a screen, etc. The content provider is used to store and acquire data, and make the data accessible by an application. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, a phone book, etc. The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide a communication function of a terminal device. For example, management of a call state (including connection, hang-up, etc.). The resource manager provides various resources for an application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager makes an application display notification information in a status bar. The notification manager can be used to convey a message of an informing type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform a download completion, a message reminder, etc. The notification manager can also be a notification appearing in a system top status bar in a form of a chart or a scrolling bar text, such as a notification of an application running in the background, or a notification appearing on a screen in a form of a dialog window. For example, a text information is prompted in a status bar, a prompt sound is emitted, a terminal device vibrates, an indicator light blinks, etc.

[0082] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library includes two parts: one part is a function function called by a java language, and the other part is an Android core library.

[0083] The application layer 101 and the framework layer 102 run in a virtual machine. The virtual machine executes java files of the application layer and the framework layer as binary files. The virtual machine is used to perform functions of management of an object life cycle, stack management, thread management, security and exception management, and garbage collection, etc.

[0084] The system library 103 can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (for example, OpenGL ES), a 2D graphics engine (for example, SGL), and the like.

[0085] The surface manager is used to manage a display subsystem and provides fusion of 2D and 3D layers for a plurality of application programs. The media libraries support playback and recording of a plurality of commonly used audio, video formats, and static image files. The media libraries can support a plurality of audio and video encoding formats, for example, MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0086] The hardware abstraction layer 104 runs in a user space, encapsulates kernel layer drivers, and provides a calling interface to an upper layer.

[0087] The kernel layer 105 is a layer between hardware and software. The kernel layer 105 at least includes display drivers, touch control drivers, audio drivers, and sensor drivers.

[0088] The kernel layer 105 is a core of an operating system of a terminal device, is a first layer of software expansion based on hardware, provides most basic functions of an operating system, is a basis for operating system work, is responsible for managing processes, memory, device drivers, files, and network systems of a system, and determines performance and stability of the system. For example, the kernel layer can determine an operation time of an application program on a certain part of hardware.

[0089] The kernel layer 105 includes programs closely related to hardware, for example, interrupt handling programs, device drivers, and the like, further includes basic, common, and high-frequency running modules, for example, clock management modules, process scheduling modules, and the like, and further includes critical data structures. The kernel layer can be disposed in a processor or be fixed in an internal memory.

[0090] The hardware layer 106 includes hardware of a terminal device, for example, a display screen, a key, a camera, and the like.

[0091] Referring to Figure 4 FIG. 1 shows a flowchart of a motion object detection method provided by an embodiment of the present application. The motion object detection method is applied in a terminal device and includes the following flow.

[0092] S201, in response to a shooting operation of a user, acquiring two adjacent image frames by using a shooting device of the terminal device.

[0093] In an embodiment of the present application, the photographing operation of the user can be an operation of starting the camera application, or an operation of clicking the photographing control after the camera application is started. The terminal device can acquire a series of image frames in response to the photographing operation of the user, and select two adjacent image frames from the acquired image frames. The two adjacent image frames can be two adjacent preview image frames. For example, the user starts the camera application and aims the camera at an object to take a photo or record a video, and the terminal device obtains two adjacent image frames of the object in response to the operation of the user. In another embodiment, the photographing operation of the user can also be to invoke the camera application to acquire two adjacent image frames by using a photographing function provided by an application (APP) when using the application, for example, to invoke the camera application to take a photo of an arbitrary object by using an instant messaging application.

[0094] In an embodiment of the present application, the two adjacent image frames can include an nth image frame f n (referred to as a first image frame), and an (n-1)th image frame f n (referred to as a second image frame), where n represents an integer greater than or equal to 2. n-1

[0095] The first time node corresponding to the first image frame is later than the second time node corresponding to the second image frame, for example, the first time node is 1 / 60 second later than the second time node, and the second image frame has the same size as the first image frame. The first image frame and the second image frame can be multi-channel images, for example, RGB images in which each pixel point includes three color channels of red (R), green (G), and blue (B). The first image frame and the second image frame can also be grayscale images or images in other formats, which are not limited in the present application.

[0096] In S202, an initial optical flow map between the two image frames is calculated by using a preset optical flow estimation algorithm.

[0097] In an embodiment of the present application, the optical flow estimation algorithm of the chip platform of the terminal device can be invoked to calculate the optical flow map between the two image frames as the initial optical flow map. The description of the initial optical flow map can be referred to the description of the initial optical flow map in the foregoing embodiments. Figure 1 Figure 2

[0098] In an embodiment of the present application, each pixel point in the initial optical flow map includes at least a first channel and a second channel, where the first channel includes a first displacement value in a first direction (for example, a displacement value in the X-axis direction in the horizontal direction in FIG. 1), and the second channel includes a second displacement value in a second direction (for example, a displacement value in the Y-axis direction in the vertical direction in FIG. 1). Figure 1 Figure 1 ​​​​​​ The optical flow value of each pixel point in the initial optical flow map includes a first displacement value, a second displacement value, and a displacement vector determined according to the first displacement value and the second displacement value (for example Figure 1 The displacement vector in the initial optical flow map corresponds to a third displacement value (for example Figure 1 The displacement value in the initial optical flow map The optical flow value (for example, the third displacement value) in the initial optical flow map represents a value greater than or equal to 0, and when the optical flow value is equal to 0, it indicates that the corresponding pixel point is in a static state, and the greater the optical flow value, the greater the motion speed of the corresponding pixel point.

[0099] S203, performing denoising processing on the initial optical flow map to obtain a target optical flow map.

[0100] In an embodiment of the present application, as shown in Figure 2 Due to the shaking of the shooting device, the limitations of the mathematical model and physical assumptions of the optical flow estimation algorithm (for example, the pixel intensity of the shooting scene is basically unchanged between two adjacent image frames, and adjacent pixels have similar motion), the complexity and variability of the shooting scene, and other reasons, the initial optical flow map may contain a large amount of noise, and the initial optical flow map needs to be denoised to improve the accuracy of the subsequent motion object detection process.

[0101] In an embodiment of the present application, for different noises in the initial optical flow map caused by different reasons, different denoising processing methods can be used to perform multiple-stage and multiple-time denoising processing on the initial optical flow map, so as to remove different noises in the initial optical flow map in each stage and improve the denoising effect of the initial optical flow map.

[0102] In an embodiment of the present application, the denoising processing method (hereinafter referred to as the first denoising processing method) for the abnormally small optical flow value noise (for example, the small optical flow value noise caused by the small shaking of the shooting device) and the abnormally large optical flow value noise (for example, the large optical flow value noise caused by the optical flow estimation algorithm) in the initial optical flow map can include: by setting an optical flow value range, resetting the optical flow value of the pixel point in the initial optical flow map corresponding to the optical flow value not belonging to the optical flow value range, for example, resetting to 0, thereby removing the influence of the small optical flow value noise and the large optical flow value noise, and obtaining an updated optical flow map.

[0103] Referring to Figure 5 As shown in the example diagram of the optical flow map denoising processing provided by an embodiment of the present application. Figure 5 Each diagram in the figure represents an optical flow map, and the darkness of the gray scale of the pixel points in the optical flow map is used to indicate the size of the optical flow value at the corresponding position, and the darker the gray scale, the larger the optical flow value, and white represents an optical flow value of 0. Figure 5 In the initial optical flow map, there are more small optical flow value noises in the lighter gray area, and there are large optical flow value noises in the black dot area.Figure 5 The updated optical flow map represents the optical flow map obtained after the first denoising processing of the initial optical flow map. As can be seen, after the first denoising processing, the light gray small optical flow value noise area in the initial optical flow map is processed as white, achieving the removal of small optical flow value noise; in addition, the black dot large optical flow value noise area in the initial optical flow map is processed as white, achieving the removal of large optical flow value noise.

[0104] In an embodiment of the present application, further denoising processing (hereinafter referred to as second denoising processing) can be performed on the updated optical flow map through the difference image between the first image frame and the second image frame, and the principle includes: the optical flow vector in the updated optical flow map represents the displacement of each pixel point between two image frames, and the noise therein is usually random and has little effect on the pixel distribution of the entire image; while the difference image between two image frames can highlight the dynamic changes of pixel points between two image frames, and the noise therein usually does not cause large changes, so the dynamic change information in the difference image between two image frames can be used to filter the optical flow noise in the updated optical flow map, further denoising the updated optical flow map, and thus more accurately distinguishing the foreground moving object from the background static area in the optical flow map.

[0105] In an embodiment of the present application, the conventional difference method is usually used in the related art to obtain a conventional difference image between two image frames, wherein the formula used by the conventional difference method includes: n (x,y)=|f n (x,y)-f n-1 (x,y)|, wherein f n represents the nth image frame (for example, the first image frame), f n (x,y) represents the first gray value of the pixel point with the coordinate (x, y) in f n (x,y) represents the second gray value of the pixel point with the coordinate (x, y) in f n-1 represents the (n-1)th image frame (for example, the second image frame) adjacent to the nth image frame, and f n-1 (x,y) represents the second gray value of the pixel point with the coordinate (x, y) in f n-1 (x,y) represents the second gray value of the pixel point with the coordinate (x, y) in f Figure 1 As shown in FIG. 1, the upper left corner point of the image frame (for example, the first image frame) is taken as the coordinate origin O to establish a rectangular coordinate system OXY, and the horizontal direction is taken as the direction of the X axis and the vertical direction is taken as the direction of the Y axis, so as to determine the coordinate of each pixel point in each image frame.

[0106] In an embodiment of the present application, the traditional difference image usually contains a large amount of noise, which leads to the problems that the complete edge of the object in the image cannot be obtained according to the traditional difference image, and the foreground moving object in the image cannot be distinguished from the background static region. Even if the traditional difference image is subjected to binarization processing and conventional noise removal, since there is too much noise in the traditional difference image, there will still be a lot of noise in the traditional difference image after noise removal (for example Figure 6 illustrated), which cannot be directly used for subsequent noise removal of the updated optical flow image.

[0107] An improved difference algorithm is provided in the embodiments of the present application, which can use the following formula:

[0108]

[0109] wherein D n (x, y) represents the gray value of the pixel point with coordinates (x, y) in the initial difference image, f n (x, y) represents the first gray value of the pixel point with coordinates (x, y) in the first image frame f n (x, y) represents the second gray value of the pixel point with coordinates (x, y) in the second image frame f n-1 (x, y) represents the second gray value of the pixel point with coordinates (x, y) in the second image frame f n-1 (x, y) represents the second gray value of the pixel point with coordinates (x, y) in the second image frame f

[0110] In the traditional difference algorithm, the absolute value of the difference between the gray values of the corresponding pixel points in the two image frames is taken. Compared with the traditional difference algorithm, the formula used in the improved difference algorithm provided in the embodiments of the present application can focus on distinguishing the moving and static pixel points. Therefore, compared with the traditional difference algorithm which will obtain a difference image with a large amount of noise, the noise in the initial difference image obtained by using the improved difference algorithm provided in the embodiments of the present application is very small, and the moving object and the static background can be more accurately distinguished (for example Figure 7 illustrated).

[0111] In an embodiment of the present application, further post-processing operations can be performed on the initial difference image, and the initial difference image after the post-processing is taken as a noise removal difference image for second noise removal processing of the updated optical flow image. For example, the post-processing operations can include, but are not limited to, one or a combination of the following operations: binarization processing, image erosion processing and image expansion processing, etc., so as to achieve the following effects: further distinguishing the moving region and the static region in the initial difference image, removing the white noise in the binarized image, enhancing the edges and contours of the moving region and the static region in the binarized image, and making the edges of different regions smoother and more continuous.

[0112] In an embodiment of the present application, when the updated optical flow map is denoised using the denoised difference map, the denoised difference map and the updated optical flow map can be pixel-by-pixel ANDed, and the optical flow value of each pixel in the updated optical flow map can be updated according to the result of the AND operation. Specifically, the optical flow value of the pixel corresponding to the pixel in the denoised difference map indicating the motion state can be maintained unchanged in the updated optical flow map, and the optical flow value of the pixel corresponding to the pixel in the denoised difference map indicating the static state can be updated to 0 in the updated optical flow map, thereby further removing the noise in the updated optical flow map.

[0113] For example Figure 5 As shown in the figure, Figure 5 The target optical flow map obtained by performing the second denoising on the updated optical flow map can be seen. After the second denoising, the noise area in the updated optical flow map is further removed, and the target optical flow map obtained basically removes the influence of noise.

[0114] In an embodiment of the present application, the method for denoising the initial optical flow map can further refer to the description of the embodiments shown in the figure. Figure 8

[0115] S204, dividing the target optical flow map into a plurality of sub-regions, and determining the first speed corresponding to each sub-region according to the optical flow value of each pixel in each sub-region.

[0116] In an embodiment of the present application, the target optical flow map can be divided into MxN sub-regions, each sub-region including mxm pixels, wherein M, N and m can be positive integers greater than 1 according to actual needs. For example, if the target optical flow map contains 640x480 pixels, the value of M can be 128, the value of N can be 96, and the value of m can be 5. For example Figure 10 As shown in the figure, the figure on the right side of the figure shows a schematic diagram of dividing the target optical flow map into a plurality of sub-regions according to an embodiment of the present application, wherein each grid in the image on the right side represents a sub-region.

[0117] In an embodiment of the present application, the optical flow value of each pixel in each sub-region includes a first displacement value and a second displacement value. Based on the first displacement value of all pixels in each sub-region, a first average value can be determined; based on the second displacement value of all pixels, a second average value can be determined; based on the first average value and the second average value, a first speed can be determined. For example, the following formula can be used:

[0118]

[0119] wherein V velicity represents the first speed of any sub-region, and V X ​a first average value of first displacement values of all pixel points in the sub-region, V Y a second average value of second displacement values of all pixel points in the sub-region.

[0120] Based on the above embodiments, the first speed corresponding to each sub-region can be preliminarily determined according to the average value of the first displacement values of all pixel points in each sub-region in the first direction and the average value of the second displacement values of all pixel points in the second direction, so as to facilitate the selection of the region where the moving object is located from the target optical flow map in the subsequent process.

[0121] S205, determining a plurality of target regions in the target optical flow map according to the first speed, and determining a second speed corresponding to each target region.

[0122] In an embodiment of the present application, each target region includes at least one sub-region, and the determination of the plurality of target regions in the target optical flow map according to the first speed can include the following processes (1)-(2):

[0123] (1) If the first speed of any sub-region is greater than a preset speed threshold, the any sub-region is taken as a target sub-region.

[0124] In an embodiment of the present application, the preset speed threshold can be set according to actual needs, for example, the speed threshold can be 1, and the present application does not make specific limitation thereto. In an embodiment of the present application, if the first speed of any sub-region is greater than the speed threshold, the sub-region can be taken as a target sub-region in the motion state. Referring to Figure 11 As shown, the target optical flow map can be abstracted into a motion state matrix containing only two elements "0" and "1" to determine whether there is a motion state in each sub-region, wherein each element in the motion state matrix represents the motion state of a sub-region at the corresponding position. As shown in Figure 11 As shown, the element "1" in the motion state matrix corresponds to a grid representing a target sub-region in the motion state, and the element "0" in the motion state matrix corresponds to a grid representing a region of a non-target sub-region, i.e., a sub-region in the static state.

[0125] Through the above embodiments, the sub-region in the motion state in the target optical flow map can be preliminarily screened, and the position of the target sub-region in the motion state in the target optical flow map can be determined. In addition, by abstracting each sub-region in the target optical flow map into an element in a motion state matrix, data dimensionality reduction can be achieved, and the calculation efficiency of the algorithm can be improved.

[0126] (2) determining a connected region where each target sub-region is located based on a connected graph algorithm, and taking the connected region as a target region.

[0127] In an embodiment of the present application, before determining the connected region to which each target sub-region belongs based on the connected graph algorithm, the target optical flow map can be first subjected to noise filtering, including: updating the optical flow value (e.g., the third optical flow value) of the isolated target sub-region of the non-boundary position in the target optical flow map to a preset first value (e.g., 0), the isolated target sub-region including a target sub-region not adjacent to other target sub-regions.

[0128] For example Figure 12 As shown in FIG. 1, a schematic diagram of noise filtering of a target optical flow map provided by an embodiment of the present application is shown. In this figure, Figure 12 The elements "1" in the rectangular frame in the left motion state matrix are surrounded by "0", which can be determined as isolated data. The elements "1" in the rectangular frame can be updated to elements "0", thereby achieving filtering of the isolated data. Correspondingly, the target sub-region corresponding to the elements "1" in the rectangular frame is an isolated target sub-region of a non-boundary position, and the third optical flow value of the isolated target sub-region of the non-boundary position can be updated to 0, thereby updating the isolated target sub-region of the non-boundary position to a non-motion region, and achieving further denoising of the target optical flow map.

[0129] In an embodiment of the present application, the connected graph algorithm is an algorithm for detecting and traversing a connected region, which can include but is not limited to a depth-first search algorithm, a breadth-first search algorithm, etc. Taking the depth-first search algorithm as an example, determining the connected region in which each target sub-region is located can include: regarding each sub-region in the target optical flow map as a node, wherein each target sub-region represents a connectable node, and each non-target sub-region represents a non-connectable node; starting from a starting node in the target optical flow map, recursively visiting its adjacent nodes, if the adjacent nodes are connectable nodes, determining that the node and the adjacent nodes are connected to each other, and marking the visited nodes; by continuously traversing the unvisited adjacent nodes until the non-connectable nodes cannot be further expanded, a connected region can be obtained.

[0130] In an embodiment of the present application, each connected region includes at least one target sub-region, representing a larger motion region; and each target optical flow map can include at least one connected region. For example Figure 13 As shown in FIG. 2, a schematic diagram of a connected region provided by an embodiment of the present application is shown. In this figure, Figure 13 In the left motion state matrix in FIG. 2, the data points adjacent to the elements "1" can be connected to obtain a connected region, which can be as shown in Figure 13 As shown in the right motion state matrix in FIG. 2, a unique label (e.g., a number, a word, English, etc.) is assigned to each connected region to distinguish different connected regions, for example Figure 13 The natural numbers 2 to 7 in the right motion state matrix in FIG. 2 respectively represent different connected regions.

[0131] Through the above embodiments, the connected regions in which the target sub-regions are connected can be determined as the target regions in which the moving objects are possibly present, so that the regions in which different moving objects are present can be divided.

[0132] In an embodiment of the present application, since the first speed of at least one sub-region included in each target region is known, the average value of the first speed of all sub-regions included in each target region is generally used as the speed of the corresponding target region in the related art. However, the accuracy of this method is low, because the related art does not consider the abnormal values possibly existing in the first speed due to noise and the like, and does not consider the influence of the state of the photographing device on the first speed.

[0133] The state of the photographing device can include an approximately stationary state (for example, the photographing device is in a hand-held or a tripod mode) and a moving state (for example, the photographing device is in a moving state), and the photographing state of the photographing device corresponding to the two image frames of the target optical flow map can be determined according to the proportion of the moving sub-region in the target optical flow map. For example, if the proportion of the moving sub-region in the target optical flow map is large, it can be considered that the relatively large number of moving sub-regions in the target optical flow map is caused by the movement or shaking of the photographing device, and thus it can be considered that the photographing device is in a moving state. Alternatively, if the proportion of the moving sub-region in the target optical flow map is small, it can be considered that the relatively small number of moving sub-regions in the target optical flow map is the region corresponding to the moving object in the photographed scene, and the photographing device is approximately stationary, and thus it can be considered that the photographing device is in an approximately stationary state. The method of determining the photographing state of the photographing device can also refer to the description of the embodiments shown in the following. Figure 14

[0134] According to the above description, it can be known that the moving state of the photographing device can seriously affect the numerical distribution of the first speed of all sub-regions included in each target region. For example, the photographing device in a moving state can cause very few non-zero values in the first speed, and in addition, the moving speed of the photographing device in a moving state can cause many abnormally large values in the first speed. In order to solve the above problems, two different methods can be used to determine the speed of the corresponding target region according to the first speed of all sub-regions included in each target region.

[0135] ​The first method can include: cleaning the abnormal values possibly existing in the first speed of each target region, for example, cleaning the smaller and larger values in the first speed of each target region, and then determining the average value of the first speed of each target region after cleaning as the speed of the corresponding target region (hereinafter referred to as the third speed). This method can regard the smaller and larger values in the first speed as abnormal values and remove the abnormal values, and obtain the average value of the first speed after filtering out the abnormal values as the third speed of the target region, thereby avoiding the influence of the abnormal values possibly existing in the first speed on the calculation accuracy of the speed of the target region.

[0136] The second method can include: sorting the first speed of each target region in descending order, selecting a plurality of first speeds from the front in the sequence obtained by sorting, and determining the average value of the selected plurality of first speeds as the speed of the corresponding target region (hereinafter referred to as the fourth speed). The fourth speed of each target region is generally greater than or equal to the third speed. This method can determine the average value corresponding to the larger values in the first speed as the fourth speed of the target region in view of the fact that the shooting device in the motion state can cause more abnormal large values in the first speed, and avoid the influence of the state of the shooting device on the calculation accuracy of the speed of the target region by judging whether the fourth speed is the average value of the abnormal large values caused by the shooting device in the motion state in the subsequent process.

[0137] In an embodiment of the present application, the method for determining the third speed and the fourth speed of the target region can also refer to the description of the embodiment shown in Figure 15

[0138] In an embodiment of the present application, the determination of the second speed based on the shooting state of the shooting device can include: if the shooting state is the approximate static state and the fourth speed of each target region is less than the product of a preset third value and the third speed of each target region, determining the second speed according to the fourth speed; or if the shooting state is the approximate static state and the fourth speed of each target region is greater than or equal to the product of the third value and the third speed of each target region, determining the second speed according to the third speed. The third value can be set according to prior data, for example, the third value can be 5, which is not limited in the present application.

[0139] ​Specifically, if the shooting state is the approximate static state, it can be determined that there are not many abnormal large values in the first speed caused by the shooting device in the motion state, i.e., the fourth speed is not likely to be the average of abnormal large values. On this basis, if the fourth speed is less than the product of the third value and the third speed of each target region, it can be considered that there are not abnormal values in the large values removed in the first method, and thus the accuracy of the third speed obtained by using the first method as the speed of the target region is low, and the fourth speed can be used as the speed of the target region (e.g., the second speed); or, if the fourth speed is greater than or equal to the product of the third value and the third speed of each target region, it can be considered that there are abnormal values in the large values removed in the first method, and thus the accuracy of the third speed used as the speed of the target region is high, and thus the third speed can be used as the speed of the target region (e.g., the second speed).

[0140] In an embodiment of the present application, the second speed is determined based on the shooting state of the shooting device, and the method can further include: if the shooting state is the motion state and the fourth speed of each target region is less than the product of the preset fourth value and the third speed of each target region, determining the second speed according to the fourth speed; or, if the shooting state is the motion state and the fourth speed of each target region is greater than or equal to the product of the fourth value and the third speed of each target region, determining the second speed according to the third speed. The fourth value can be set according to prior data, for example, the fourth value can be 1.2, and the present application does not make a specific limitation in this regard.

[0141] Specifically, if the shooting state is the motion state, it can be determined that there are many abnormal large values in the first speed caused by the shooting device in the motion state, i.e., the fourth speed is likely to be the average of abnormal large values. On this basis, if the fourth speed is less than the product of the fourth value and the third speed of each target region, it can be considered that the fourth speed is not the average of abnormal large values, and there are not abnormal values in the large values removed in the first method, and thus the accuracy of the third speed obtained by using the first method as the speed of the target region is low, and the fourth speed can be used as the speed of the target region (e.g., the second speed); or, if the fourth speed is greater than or equal to the product of the fourth value and the third speed of each target region, it can be considered that the fourth speed is the average of abnormal large values, and there are abnormal values in the large values removed in the first method, and thus the accuracy of the third speed used as the speed of the target region is high, and thus the third speed can be used as the speed of the target region (e.g., the second speed).

[0142] Through the above embodiment, the second speed of the target region can be determined according to the shooting state of the shooting device, the third speed and the fourth speed of the target region, so as to avoid misjudgment of the movement speed of the target region due to the shooting state of the shooting device, and improve the calculation accuracy of the movement speed of the target region.

[0143] In S206, the region in which the moving object exists in the first image frame of the two image frames is determined according to the target region, and the movement speed of the moving object is determined according to the second speed.

[0144] In an embodiment of the present application, the accuracy of directly taking the region corresponding to the target region in the first image frame as the region in which the moving object exists can be relatively low, because there can be a case that the entire target region is a noise region. Therefore, the target region can be denoised to eliminate the noise region in the target region and improve the accuracy of the moving region detection.

[0145] Since the target optical flow map in the above embodiment is obtained by denoising the initial optical flow map multiple times, and the target region is obtained by filtering the isolated noise (for example, isolated target sub-regions) in the target optical flow map, the noise region in the target region can be a relatively large noise region. The noise region in the target region can be determined by noise detection on a preset number of target regions with the largest area, so as to realize denoising of the target region.

[0146] In an embodiment, the denoising of the target region can include: sorting all target regions in descending order of the area of the target region, and selecting a preset number of target regions in front from the sorted sequence, and determining the noise region from the selected target regions. The preset number represents a positive integer less than or equal to the total number of target regions, which can be set according to actual needs, for example, the preset number can be 7, which is not limited in the present application.

[0147] In an embodiment of the present application, the area of any target region is proportional to the number of target sub-regions of the target region, so when all target regions are sorted in descending order of the area of the target region, all target regions can be sorted in descending order of the number of target sub-regions contained by the target region.

[0148] In an embodiment of the present application, when determining the noise region from the selected target region, a corresponding region of the selected target region in the first image frame or the second image frame can be determined, and a traditional difference image between the two image frames of the corresponding region can be obtained; for the corresponding region of any selected target region, an average difference value of the difference values in the traditional difference image of the region can be calculated; a difference threshold value can be set for any sub-region according to the number of sub-regions in any selected target region; if the average difference value of any selected target region is less than or equal to the difference threshold value corresponding to any sub-region, it can be considered that the any sub-region is a noise region.

[0149] Through the above embodiments, the corresponding difference threshold value can be determined according to the number of sub-regions of each target region, so that the noise region in the target region can be determined according to the area size of each target region, thereby improving the recognition accuracy of the motion region.

[0150] In an embodiment of the present application, the method for determining the noise region in the target region can also refer to the description of the embodiments shown in Figure 16 .

[0151] In an embodiment of the present application, after the denoising processing of the target region, the corresponding region of each remaining target region in the first image frame can be regarded as a region with a moving object, and the second speed of each remaining target region can be regarded as the motion speed of the moving object in the corresponding region with the moving object.

[0152] Through the above multiple embodiments, the optical flow estimation algorithm of the chip of the terminal device can be called when the image is captured to obtain the initial optical flow image between the two adjacent image frames; through the denoising processing of the initial optical flow image, the noise and abnormal optical flow in the initial optical flow image can be gradually removed; by dividing the target optical flow image into multiple sub-regions, the connected region in the target optical flow image that may have a moving object can be determined according to the optical flow value in the target optical flow image; by screening the connected region, the region in the target optical flow image that has a moving object can be accurately determined, so as to avoid misjudging the noise region in the static state as a moving object, and improve the accuracy of calculating the motion speed of the moving object. Further, the exposure parameter of the shooting device can be adjusted according to the region with the moving object and the motion speed of the moving object, so as to improve the image quality (such as definition) of the captured image.

[0153] For an embodiment of the present application, a flow chart of the method for denoising the initial optical flow image is shown in Figure 8 . The method is applied in a terminal device, and the method for denoising the initial optical flow image comprises the following steps:

[0154] S301, performing denoising processing on the initial optical flow map based on the optical flow value of each pixel point in the initial optical flow map and a preset optical flow value range, to obtain an updated optical flow map.

[0155] In an embodiment of the present application, if the displacement value (hereinafter referred to as "third displacement value" for distinguishing from other displacement values in the above) of any pixel point in the initial optical flow map is not within the optical flow value range, the optical flow value of any pixel point is updated to a preset first value; or, if the third displacement value of any pixel point is within the optical flow value range, the optical flow value of any pixel point is maintained. The optical flow value range can be set according to actual needs, for example, the optical flow value range can be [a, b] = [1, 60], which is not specifically limited in the present application.

[0156] The above denoising processing method can also be represented as: if g(x, y)≤a or g(x, y)≥b, then G(x, y) = 0; or, if a < g(x, y) < b, then G(x, y) = g(x, y), wherein g(x, y) represents the optical flow value (for example, the third displacement value) of the pixel point with coordinates (x, y) in the initial optical flow map, and G(x, y) represents the optical flow value (for example, the third displacement value) of the pixel point with coordinates (x, y) in the updated optical flow map.

[0157] Through the above embodiments, the denoising of abnormal small optical flow value noise (for example, small optical flow value noise caused by small shaking of the shooting device) and abnormal large optical flow value noise (for example, large optical flow value noise caused by optical flow estimation algorithm) in the initial optical flow map can be realized.

[0158] S302, obtaining a denoised difference map by using an improved difference algorithm based on the first image frame and the second image frame in the two image frames.

[0159] In an embodiment of the present application, referring to FIG. 2, a flowchart of a detailed process of S302 provided by an embodiment of the present application is shown, and the detailed process of S302 includes: Figure 9

[0160] S3021, determining the first gray value of each pixel point in the first image frame and the second gray value of the corresponding pixel point in the second image frame.

[0161] S3022, performing difference calculation on the first image frame and the second image frame by using the improved difference algorithm based on the first gray value and the second gray value of each pixel point, to obtain an initial difference image between the first image frame and the second image frame.

[0162] In an embodiment of the present application, the formula used by the improved difference algorithm can refer to the description of S203 in the process shown in FIG. 2. Figure 4

[0163] ​​S3023, binarize the initial difference image to obtain a binarized image.

[0164] In one embodiment of this application, binarization processing of the initial difference image includes: if the gray value of any pixel in the initial difference image is less than a preset gray value threshold, then the gray value of any pixel is updated to a preset first value; or, if the gray value of any pixel is greater than or equal to the gray value threshold, then the gray value of any pixel is updated to a preset second value. Wherein, a pixel with a gray value of the first value (e.g., 0) represents a pixel in a black background area (or a static area), and a pixel with a gray value of the second value (e.g., 1 or 255) represents a pixel in a white foreground area (or a moving area). The specific values ​​of the first and second values ​​can be set according to actual needs, and this application does not impose specific limitations on them.

[0165] In one embodiment of this application, the formula used for binarization processing may include:

[0166]

[0167] Among them, D n (x,y) represents the initial difference image D. n The grayscale value of the pixel with coordinates (x, y) in the image, R n (x,y) represents the binary image R. n The grayscale value of the pixel with coordinates (x, y) is represented by 0, 1, and T, which represents the grayscale threshold (e.g., 3).

[0168] Through the above embodiments, the gray values ​​of pixels in the initial difference image can be updated to preset values, thereby accurately identifying moving pixels and stationary pixels based on the obtained binarized image.

[0169] S3024 performs post-processing on the binarized image to obtain a denoised difference image.

[0170] In one embodiment of this application, the post-processing operation may include, but is not limited to, a combination of one or more of the following methods: image erosion processing and image dilation processing.

[0171] In one embodiment of this application, image erosion processing may include: defining a structuring element (e.g., a rectangle, circle, cross, etc.); placing the center point of the structuring element at each pixel of the image (e.g., a binarized image); determining the minimum gray value of all pixels within the area covered by the structuring element; and updating the gray value of the pixel at the center point to the minimum value.

[0172] By performing image erosion processing on the binary image, protruding points on the periphery of the foreground region of the binary image can be eroded, noise and small discontinuous regions can be removed by narrowing the boundary of the foreground region, and the boundaries of different moving objects that can exist in the foreground region can be distinguished, so that the moving objects and the static background can be more accurately distinguished.

[0173] In an embodiment of the present application, the image dilation processing can include: defining a structure element (such as a rectangular element, a circular element, a cross-shaped element, etc.); placing the center point of the structure element at each pixel point of the image (such as the binary image), determining the maximum value of the gray values of all pixel points in the coverage area of the structure element, and updating the gray value of the pixel point at the center point to the maximum value.

[0174] By performing image dilation processing on the binary image, pixel points on the boundary of the foreground region in the image (such as the binary image) can be expanded to achieve the effect of widening the boundary of the foreground region, so that the edges of different foreground regions are more smooth and continuous.

[0175] S303, performing denoising processing on the updated optical flow map by using the denoised difference map to obtain a target optical flow map.

[0176] In an embodiment of the present application, the denoised difference map and the updated optical flow map can be subjected to pixel-by-pixel AND operation, and the optical flow value of each pixel point in the updated optical flow map is updated according to the result of the AND operation. For example, if the gray value of any pixel point in the denoised difference map is equal to a preset first value, the optical flow value of the pixel point corresponding to the any pixel point in the updated optical flow map is updated to the first value (such as 0); or, if the gray value of any pixel point is equal to a preset second value (such as 1), the optical flow value of the pixel point corresponding to the any pixel point in the updated optical flow map is maintained.

[0177] The method of performing denoising processing on the updated optical flow map by using the denoised difference map can also be expressed as: if Dq(x, y) = 0, then let G ′ (x, y) = 0; or, if Dq(x, y) = 1, then let G ′ (x, y) = G(x, y), where Dq(x, y) represents the gray value of the pixel point with coordinates (x, y) in the denoised difference map, G(x, y) represents the optical flow value (such as the third displacement value) of the pixel point with coordinates (x, y) in the updated optical flow map, and G ′ (x, y) represents the optical flow value (such as the third displacement value) of the pixel point with coordinates (x, y) in the target optical flow map.

[0178] Through the above embodiment, the corresponding optical flow value in the updated optical flow map can be reset to 0 according to the stationary pixel points (pixel points with a black background gray value of 0) indicated by the denoised difference map, and the optical flow value of the corresponding pixel point in the updated optical flow map can be maintained unchanged according to the moving pixel points (pixel points with a white foreground gray value of 1) indicated by the denoised difference map, so as to further eliminate the noise caused by the abnormal optical flow value in the updated optical flow map.

[0179] Referring to FIG. 13, a flowchart of a method for determining a shooting state of a shooting device is provided in an embodiment of the present application. The method is applied in a terminal device, and the method for determining the shooting state of the shooting device comprises the following steps. Figure 14

[0180] In S401, a ratio of a number of all target sub-regions in the target optical flow map to a total number of all sub-regions in the target optical flow map is determined.

[0181] In an embodiment of the present application, the target sub-region represents a sub-region with a first speed greater than a preset speed threshold. Referring to the description in S205, the number I of elements "1" in the motion state matrix can be determined as the number of all target sub-regions in the target optical flow map. Referring to the description in S204, the total number of all sub-regions in the target optical flow map can be MxN. The ratio of the number I of all target sub-regions in the target optical flow map to the total number MxN of all sub-regions in the target optical flow map can be exemplarily represented as:

[0182]

[0183] In S402, it is determined whether the ratio is greater than a preset proportion threshold. If the ratio is greater than the proportion threshold, S403 is performed; or if the ratio is less than or equal to the proportion threshold, S404 is performed.

[0184] In an embodiment of the present application, the preset proportion threshold can be set according to actual needs, and the value range of the proportion threshold is (0, 1), for example, the proportion threshold can be 0.75, which is not limited in the present application.

[0185] If the ratio is greater than the proportion threshold, in S403, it is determined that the shooting state of the shooting device is a motion state.

[0186] In an embodiment of the present application, when the ratio is greater than the proportion threshold, it can be considered that most of the target optical flow map is moving optical flow, which can be considered as a full-frame change caused by the motion (for example, movement and shaking) of the shooting device, so it can be determined that the shooting state of the shooting device is a motion state.

[0187] ​If the above ratio is less than or equal to the ratio threshold, in S404, the shooting state of the shooting device is determined to be approximately static.

[0188] In one embodiment of this application, when the above ratio is less than or equal to the ratio threshold, it can be considered that the moving optical flow in the target optical flow map is caused by local moving objects in the scene. Therefore, it can be determined that the shooting state of the shooting device is approximately stationary, such as the camera being in user handheld mode or tripod mode.

[0189] Through the above embodiments, the shooting state of the shooting device for the two image frames corresponding to the target optical flow map can be determined based on the proportion of the moving sub-regions in the target optical flow map. For example, if the proportion of the moving sub-regions in the target optical flow map is large, it can be considered that the presence of many moving sub-regions in the target optical flow map is due to the movement of the shooting device, and therefore the shooting device can be considered to be in a moving state; or, if the proportion of the moving sub-regions in the target optical flow map is small, it can be considered that the few moving sub-regions in the target optical flow map are the areas corresponding to moving objects in the shooting scene, and the shooting device is approximately stationary, and therefore the shooting device can be considered to be in an approximately stationary state.

[0190] For reference Figure 15 The diagram shown is a flowchart of a method for determining the third and fourth velocities of a target area according to an embodiment of this application. The method is applied in a terminal device, and the method for determining the third and fourth velocities of the target area includes:

[0191] S501, clean up the outliers in the first velocity of all sub-regions in each target region, and determine the third velocity of each target region based on the average value of the cleaned first velocities.

[0192] In one embodiment of this application, outliers in the first velocity can be cleaned up according to the principle of normal distribution. This principle includes the following: according to the standard normal distribution, approximately 99.7% of the data will fall within the range of the mean μ plus or minus three standard deviations σ. Therefore, cleaning up outliers in the first velocity of all sub-regions in each target region can include: calculating the mean μ and standard deviation σ of the first velocity of all sub-regions in each target region; determining the range of normal values, which can be expressed as [μ-3σ, μ+3σ]; and treating the first velocity of all sub-regions in each target region that is not within the range of normal values ​​as outliers, and cleaning up or resetting the outliers to 0.

[0193] Through the above embodiments, based on the principle of normal distribution, the smaller and larger values ​​in the first velocity can be removed, and the average value of the first velocity after filtering out possible outliers can be used as the third velocity of the target area.

[0194] S502, determining, based on the sliding window algorithm, a plurality of sliding window average values corresponding to the first speeds of all the sub-regions of each target region, and determining the fourth speed of each target region according to a maximum value in the plurality of sliding window average values.

[0195] In an embodiment of the present application, the step of determining, based on the sliding window algorithm, a plurality of sliding window average values corresponding to the first speeds of all the sub-regions of each target region can include: sorting the first speeds of all the sub-regions of each target region in descending order to obtain a sequence of the first speeds of all the sub-regions of each target region; determining the size of the window, i.e., the number of first speeds that the window can accommodate, which can be less than the total number of first speeds in the sequence, for example, the total number of first speeds in the sequence is 5, and the size of the window can be 3; initializing the window position, setting the first data in the window as the first first speed in the sequence, calculating the average value of the first speeds in the window, i.e., the first sliding window average value l1; moving the window position by one data, setting the first data in the window as the second first speed in the sequence, calculating the average value of the first speeds in the window, i.e., the second sliding window average value l2; and continuously moving the sliding window in this way until the last data in the window is the last first speed in the sequence, or until the sliding window average value l p , wherein p represents a preset positive integer, for example, p = 3; and determining the sliding window average value of the first speeds in the window at the i-th position in the above process, i.e., l i , wherein l i+1 ≤ l i , and i represents a positive integer.

[0196] In an embodiment of the present application, when the fourth speed of each target region is determined according to the maximum value in the plurality of sliding window average values, the plurality of sliding window average values can be arranged in descending order, and the first p ′ sliding window average values in the arrangement are selected, i.e., l1, l2,..., l p , wherein p ′ represents a preset positive integer less than or equal to p, for example, when p ′ = 3, the first p ′ sliding window average values are l1, l2, and l3; and if the difference between every two adjacent sliding window average values in the first p ′ sliding window average values is less than or equal to a preset dynamic parameter θ, the maximum value in the plurality of sliding window average values is selected as the fourth speed of the target region, for example, if l1-l2≤ θ and l2-l3≤ θ, the fourth speed is determined to be l1.

[0197] In one embodiment of this application, the dynamic parameter θ can be determined based on the maximum speed of all grids in each connected region. For example, if the maximum speed is greater than a preset threshold of 10, θ can be set to 1; or, if the maximum speed is less than or equal to the preset threshold of 10, θ can be set to maximum speed × 0.1, and the value range of θ can be [0,1].

[0198] Through the above embodiments, the average value corresponding to the larger value in the first velocity can be obtained as the fourth velocity of the target region by using the sliding window algorithm.

[0199] For reference Figure 16 The diagram shown is a flowchart of a method for determining a noise region in a target region according to an embodiment of this application. The method is applied in a terminal device, and the method for determining a noise region in a target region includes:

[0200] S601, determine the corresponding region of each target region in the first image frame or the second image frame, and obtain the conventional difference map between the two image frames of the corresponding region.

[0201] In one embodiment of this application, the position of each pixel in each target region (e.g., each target region selected in S206) in the entire target optical flow map can be determined, thereby determining the corresponding region of each target region in the first image frame or the second image frame based on the set of positions of all pixels in each target region.

[0202] For example Figure 17 The image shown is an example diagram of the pixels in the target region provided in an embodiment of this application. Wherein, Figure 17 The image on the left shows the target optical flow map divided into 64×48 sub-regions, with each target sub-region represented by a small black block and each non-target region represented by a small gray block. In the left image, the area formed by the small black blocks within the rectangular frame represents a target region, where each black block represents a target sub-region, and each target sub-region includes, for example: Figure 17 The image on the right shows 5×5 pixels. The position of each pixel in each target sub-region is known. Therefore, the corresponding region of each target region in the first or second image frame can be determined by the position of each pixel in the entire target optical flow map.

[0203] In one embodiment of this application, each pixel in the traditional difference map corresponds to a difference value. The method for obtaining the traditional difference map can be referred to the description in S203. By calculating the traditional difference map of the corresponding area of ​​each target area, the difference calculation result can be made more accurate.

[0204] S602, determine an average difference value of difference values of all pixel points in the traditional difference map corresponding to each target region.

[0205] In an embodiment of the present application, the average difference value can be equal to a ratio of a sum value of difference values of all pixel points in the traditional difference map corresponding to each target region to a number of all pixel points of each target region.

[0206] S603, determine a difference threshold value corresponding to each target region according to a number of sub-regions in each target region.

[0207] In an embodiment of the present application, the difference threshold value corresponding to each target region can be determined according to the following formula:

[0208]

[0209] wherein threshold represents the difference threshold value, num represents the number of sub-regions (such as target sub-regions) in each target region, and a, l, b, g, k represent preset parameters. For example, a = -0.0125, l = 18.125, b = -0.02, g = 10.0, and k = 250. The above parameters can be determined according to prior data, and the present application does not make specific limitations thereon. Through the above embodiment, a larger difference threshold value can be set for a target sub-region with a larger area.

[0210] S604, determine a target region corresponding to an average difference value less than or equal to a corresponding difference threshold value as a noise region.

[0211] In an embodiment of the present application, if the average difference value is less than or equal to the difference threshold value, the corresponding target region can be considered as a noise region; if the average difference value is greater than the difference threshold value, the corresponding target region can be considered as a non-noise region with a moving object.

[0212] Through the above embodiment, the difference threshold value corresponding to each target region can be determined according to the number of sub-regions of each target region, so that the noise region in the target region can be determined according to the difference average value of the traditional difference map corresponding to each target region and the difference threshold value, and the recognition accuracy of the moving region is improved.

[0213] Reference is made to Figure 18The diagram shows a flowchart of a method for differential denoising optical flow according to an embodiment of this application. The method for differential denoising optical flow may include: performing threshold denoising on the input original optical flow (e.g., an initial difference map), where the threshold range can be [a, b], and resetting optical flow values ​​exceeding the threshold range to 0 to obtain a first-stage threshold-denoised optical flow; performing differential calculation on two input image frames (e.g., a first image frame and a second image frame) using an improved differential algorithm to obtain a difference map; performing binarization processing on the difference map to obtain a binarized difference map; performing erosion processing on the binarized difference map to obtain an eroded difference map; performing dilation processing on the eroded difference map to obtain a denoised difference map; and performing a pixel-by-pixel AND operation on the first-stage threshold-denoised optical flow and the denoised difference map to obtain a second-stage differential denoised optical flow.

[0214] Through the above embodiments, noise in the initial optical flow can be initially removed according to a preset optical flow value range to obtain a first-stage threshold denoising optical flow; denoising difference maps of two image frames are obtained according to the improved difference algorithm, and the first-stage threshold denoising optical flow is further denoised using the denoising difference maps. This can effectively distinguish the differences between moving and stationary pixels in the updated optical flow. By resetting the optical flow value in the first-stage threshold denoising optical flow, optical flow noise that interferes with the distinction between moving and stationary regions can be effectively removed.

[0215] See Figure 19 The diagram shown is a flowchart of a motion detection method provided in an embodiment of this application. The motion detection method may include: dividing a denoised optical flow map (e.g., a target optical flow map) into multiple grids (e.g., multiple sub-regions); calculating the average displacement value (e.g., a first velocity) of the optical flow for each grid; selecting motion grids (e.g., target sub-regions) that may contain moving objects based on the average displacement value; filtering noisy grids in the motion grids, for example, filtering isolated motion grids that are not boundary-defined (e.g., isolated target sub-regions); calculating the connected graph (e.g., the target region) formed by the filtered motion grids; and proceeding according to a preset order (e.g., according to...). Figure 13For each connected graph (numbered 2 to 7 in sequence), the following process is performed: Calculate two possible motion velocities (e.g., the third and fourth velocities) of the current connected graph; determine the camera's motion state (e.g., the camera is in motion, or in a near-static state where it is held by the user or placed on a tripod), verify the two possible motion velocities based on the camera's state, and determine the final velocity of the current connected graph (e.g., the second velocity); determine whether the current connected graph is a noise region; if the current connected graph is not a noise region, output the velocity of the current connected graph (e.g., the second velocity) as the motion velocity of the corresponding moving object in the image frame (e.g., the first image frame); if there are other connected graphs that have not undergone the above process, then perform the above process on the next connected region in the above order. In another embodiment, the input to the motion detection method can also be optical flow without noise removal, because a certain degree of denoising effect can also be achieved during motion detection.

[0216] Through the above embodiments, the motion speed of each connected region can be accurately determined according to the state of the shooting device, and noise regions can be removed from the connected regions, thereby improving the accuracy of calculating the motion speed of the moving region and the moving object.

[0217] See Figure 20 The diagram shown is a flowchart of a method for noise verification of a connected graph according to an embodiment of this application. The method for noise verification of a connected graph may include: performing a full-graph difference AND operation on the regions corresponding to the connected graph in two image frames (e.g., a first image frame and a second image frame) to obtain a connected subgraph region difference image of the region corresponding to the connected graph; determining the average difference value of each connected subgraph region difference image; determining a corresponding difference threshold based on the area of ​​each connected graph; and determining whether the connected graph is a noise region by comparing the average difference value of each connected graph with the difference threshold.

[0218] The above embodiments enable the identification of noisy areas within the target region, thereby improving the accuracy of motion region recognition.

[0219] This application also provides a terminal device 100, see below. Figure 21As shown, the terminal device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a netbook, and a cellular phone, a Personal Digital Assistant (PDA), an Augmented Reality (AR) device, a Virtual Reality (VR) device, an Artificial Intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, and / or a smart city device, and the specific type of the terminal device 100 is not specially limited in the embodiments of the present application.

[0220] The terminal device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0221] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0222] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors.

[0223] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0224] The processor 110 can also be provided with a memory for storing instructions and data. In an embodiment of the present application, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or are used repeatedly by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0225] In an embodiment of the present application, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0226] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In an embodiment of the present application, the processor 110 can contain multiple sets of I2C bus. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, and realize the touch function of the terminal device 100.

[0227] The I2S interface can be used for audio communication. In an embodiment of the present application, the processor 110 can contain multiple sets of I2S bus. The processor 110 can be coupled to the audio module 170 through the I2S bus, and realize the communication between the processor 110 and the audio module 170. In an embodiment of the present application, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, and realize the function of answering the phone through the Bluetooth earphone.

[0228] The PCM interface can also be used for audio communication, which samples, quantizes and encodes analog signals. In an embodiment of the present application, the audio module 170 and the wireless communication module 160 can be coupled through the PCM bus interface. In an embodiment of the present application, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, and realize the function of answering the phone through the Bluetooth earphone. The I2S interface and the PCM interface can both be used for audio communication.

[0229] The UART interface is a universal serial data bus, which is used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In an embodiment of the present application, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, and realizes the Bluetooth function. In an embodiment of the present application, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, and realize the function of playing music through the Bluetooth earphone.

[0230] The MIPI interface can be used to connect the processor 110 and the display screen 194, the camera 193, and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In an embodiment of the present application, the processor 110 and the camera 193 communicate through the CSI interface to implement the photographing function of the terminal device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the terminal device 100.

[0231] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In an embodiment of the present application, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0232] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and a peripheral device. The interface can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other terminal devices 100, such as AR devices, and the like.

[0233] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the terminal device 100. In other embodiments of the present application, the terminal device 100 can also use different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0234] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through a wireless charging coil of the terminal device 100. The charging management module 140 can charge the battery 142 while also supplying power to the terminal device 100 through the power management module 141.

[0235] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some embodiments, the power management module 141 can also be disposed in the processor 110. In some embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0236] The wireless communication function of the terminal device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0237] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some embodiments, the antennas can be used in combination with a tuning switch.

[0238] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the terminal device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and the like the received electromagnetic waves, and transmit the processed signals to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor and radiate the signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0239] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a microphone 170B, etc.), or displays an image or a video through the display screen 194. In an embodiment of the present application, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110 and can be disposed in the same device as the mobile communication module 150 or other functional modules.

[0240] The wireless communication module 160 can provide a wireless communication solution including a wireless local area network (WLAN) (such as a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), and the like, which can be applied to the terminal device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and radiate the signal as an electromagnetic wave via the antenna 2.

[0241] In an embodiment of the present application, the antenna 1 of the terminal device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the terminal device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0242] The terminal device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected with the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0243] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light Emitting Diode (AMOLED), a Flex Light-Emitting Diode (FLED), a Miniled, a Microled, a Micro-OLED, a Quantum Dot Light Emitting Diodes (QLED), or the like. In an embodiment of the present application, the terminal device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0244] The terminal device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0245] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In an embodiment of the present application, the ISP can be arranged in the camera 193.

[0246] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In an embodiment of the present application, the terminal device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0247] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0248] The video codec is used to compress or decompress digital video. The terminal device 100 can support one or more video codecs. In this way, the terminal device 100 can play or record videos in multiple encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0249] The NPU is a neural network (Neural-Network, NN) computing processor. By drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, the NPU can quickly process input information and can also continuously self-learn. Through the NPU, intelligent cognitive applications of the terminal device 100 can be realized, such as image recognition, face recognition, motion object detection, speech recognition, text understanding, etc.

[0250] The internal memory 121 can include one or more random access memories (Random Access Memory, RAM) and one or more non-volatile memories (Non-Volatile Memory, NVM).

[0251] The random access memory can include static random memory (Static Random-Access Memory, SRAM), dynamic random memory (Dynamic Random Access Memory, DRAM), synchronous dynamic random memory (Synchronous Dynamic Random Access Memory, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR SDRAM, for example, the fifth generation of DDR SDRAM is generally referred to as DDR5 SDRAM), etc.

[0252] The non-volatile memory can include a magnetic disk storage device, a flash memory.

[0253] The flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operation principle, can include Single-Level Cell (SLC), Multi-Level Cell (MLC), Triple-Level Cell (TLC), Quad-Level Cell (QLC), etc. according to the storage unit potential order, and can include Universal Flash Storage (UFS), embedded Multi Media Card (eMMC), etc. according to the storage specification.

[0254] The random access memory can be directly read and written by the processor 110, can be used to store executable programs (such as machine instructions) of an operating system or other programs running, and can also be used to store data of users and application programs, etc.

[0255] The non-volatile memory can also store executable programs and store data of users and application programs, etc., and can be loaded in advance into the random access memory for direct reading and writing by the processor 110.

[0256] The external memory interface 120 can be used to connect an external non-volatile memory, to realize the storage capacity expansion of the terminal device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120, to realize the data storage function. For example, files such as music and video are saved in the external non-volatile memory.

[0257] The internal memory 121 or the external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include a plurality of instructions, which when executed by the processor 110, can implement the screen display detection method executed on the terminal device 100 in the above-mentioned embodiments, to realize the screen display detection function of the terminal device 100.

[0258] The terminal device 100 can realize the audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0259] The audio module 170 is configured to convert digital audio information into analog audio signals and to convert analog audio input into digital audio signals. The audio module 170 can also be configured to encode and decode audio signals. In an embodiment of the present application, the audio module 170 can be disposed in the processor 110, or some of the functional modules of the audio module 170 can be disposed in the processor 110.

[0260] The speaker 170A, also referred to as a "loudspeaker", is configured to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0261] The receiver 170B, also referred to as a "earpiece", is configured to convert audio electrical signals into sound signals. When the terminal device 100 answers a call or a voice message, the receiver 170B can be held close to the ear of a person to listen to the voice.

[0262] The microphone 170C, also referred to as a "microphone", "sound transducer", is configured to convert sound signals into electrical signals. When making a call or sending a voice message, a user can speak into the microphone 170C through the mouth to input sound signals into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, in addition to collecting sound signals, noise reduction functions can also be achieved. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C, to achieve the functions of collecting sound signals, noise reduction, and identifying the source of the sound, and to achieve the functions of directional recording, etc.

[0263] The earphone interface 170D is configured to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0264] The keys 190 include a power-on key, a volume key, etc. The keys 190 can be mechanical keys. They can also be touch keys. The terminal device 100 can receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0265] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations acting on different regions of the display screen 194. Different application scenarios (such as time reminders, received messages, alarms, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0266] The indicator 192 can be an indicator light, which can be used to indicate a charging state, a power change, and can also be used to indicate a message, a missed call, a notification, etc.

[0267] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the terminal device 100. The terminal device 100 can support one or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. The same SIM card interface 195 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The terminal device 100 interacts with a network through the SIM card to realize functions such as calling and data communication. In an embodiment of the present application, the terminal device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100. An embodiment of the present application also provides a computer storage medium, and the computer storage medium stores computer instructions. When the computer instructions are executed on the terminal device 100, the terminal device 100 executes the related method steps to realize the motion object detection method in the above embodiment.

[0268] An embodiment of the present application also provides a computer program product, which, when executed on a computer, causes the computer to execute the related steps to realize the motion object detection method in the above embodiment.

[0269] In addition, an embodiment of the present application also provides a device, which can be a chip, a component or a module. The device can include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device is running, the processor can execute the computer execution instructions stored in the memory to enable the chip to execute the motion object detection method in the above method embodiments.

[0270] The terminal device, the computer storage medium, the computer program product or the chip provided in the embodiment are used for executing the corresponding method provided in the above, and thus the beneficial effects achieved by the terminal device, the computer storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided in the above, which will not be described here again.

[0271] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0272] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0273] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0274] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0275] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application or the essential part or all or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The software product is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0276] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for detecting moving objects, applied to a terminal device, characterized in that, The method includes: In response to the user's shooting operation, the terminal device's shooting device acquires two adjacent image frames; The initial optical flow map between the two image frames is calculated using a preset optical flow estimation algorithm; The initial optical flow map is denoised to obtain the target optical flow map, including: if the displacement value of any pixel in the initial optical flow map is not within a preset optical flow value range, the optical flow value of the pixel is updated to a preset first value; or, if the displacement value of any pixel is within the optical flow value range, the optical flow value of the pixel is maintained. The target optical flow map is divided into multiple sub-regions, and the first velocity corresponding to each sub-region is determined based on the optical flow value of each pixel in each sub-region. Based on the first velocity, multiple target regions in the target optical flow map are determined, and a second velocity corresponding to each target region is determined, wherein each target region includes at least one sub-region; The region containing the moving object in the first image frame of the two image frames is determined based on the target region, and the motion speed of the moving object is determined based on the second speed.

2. The moving object detection method according to claim 1, characterized in that, Each pixel in the initial optical flow map includes at least a first channel and a second channel, wherein the first channel includes a first displacement value in a first direction and the second channel includes a second displacement value in a second direction; The optical flow value of each pixel in the initial optical flow map includes the first displacement value, the second displacement value, and the third displacement value corresponding to the displacement vector determined based on the first displacement value and the second displacement value.

3. The moving object detection method according to claim 1 or 2, characterized in that, The step of denoising the initial optical flow map to obtain the target optical flow map includes: Based on the optical flow value of each pixel in the initial optical flow map and the preset optical flow value range, the initial optical flow map is denoised to obtain an updated optical flow map. Based on the first and second image frames of the two image frames, a denoised difference map is obtained using an improved difference algorithm; The updated optical flow map is denoised using the denoised difference map to obtain the target optical flow map.

4. The moving object detection method according to claim 3, characterized in that, The optical flow value of each pixel in the initial optical flow map includes a third displacement value. The denoising process of the initial optical flow map based on the optical flow value of each pixel in the initial optical flow map and a preset optical flow value range includes: If the third displacement value of any pixel in the initial optical flow map is not within the range of the optical flow value, the optical flow value of that pixel is updated to a preset first value; or... If the third displacement value of any pixel is within the range of the optical flow value, the optical flow value of any pixel is maintained.

5. The moving object detection method according to claim 3, characterized in that, The step of obtaining a denoised difference map based on the first and second image frames of the two image frames using an improved difference algorithm includes: Determine the first grayscale value of each pixel in the first image frame, and the second grayscale value of the corresponding pixel in the second image frame; Based on the first gray value and the second gray value corresponding to each pixel, the improved difference algorithm is used to perform difference calculation on the first image frame and the second image frame to obtain the initial difference image between the first image frame and the second image frame. The initial difference image is binarized to obtain a binarized image; The binarized image is post-processed to obtain the denoised difference image.

6. The moving object detection method according to claim 5, characterized in that, The improved difference algorithm uses the following formulas: ， in, The coordinates in the initial difference image are: The grayscale value of the pixel. Indicates the first image frame The coordinates in are The first grayscale value of the pixel, Indicates the second image frame The coordinates in are The second grayscale value of the pixel.

7. The moving object detection method according to claim 5, characterized in that, The binarization process of the initial difference image includes: If the grayscale value of any pixel in the initial difference image is less than a preset grayscale threshold, then the grayscale value of that pixel is updated to a preset first value; or, If the grayscale value of any pixel is greater than or equal to the grayscale threshold, the grayscale value of any pixel is updated to a preset second value.

8. The moving object detection method according to claim 3, characterized in that, The step of denoising the updated optical flow map using the denoised difference map includes: A pixel-by-pixel AND operation is performed on the denoised difference map and the updated optical flow map, and the optical flow value of each pixel in the updated optical flow map is updated based on the result of the AND operation.

9. The moving object detection method according to claim 8, characterized in that, The step of updating the optical flow value of each pixel in the updated optical flow map based on the result of the AND operation includes: If the grayscale value of any pixel in the denoised difference map is equal to a preset first value, the optical flow value of the pixel corresponding to that pixel in the updated optical flow map is updated to the first value; or... If the grayscale value of any pixel is equal to a preset second value, the optical flow value of the pixel corresponding to that pixel in the updated optical flow map is maintained.

10. The moving object detection method according to claim 1, characterized in that, The optical flow value of each pixel in each sub-region includes a first displacement value and a second displacement value. The process of dividing the target optical flow map into multiple sub-regions and determining the first velocity corresponding to each sub-region based on the optical flow value of each pixel within each sub-region includes: Determine the first average of the first displacement values ​​of all pixels within each sub-region, and the second average of the second displacement values ​​of all pixels; The first speed is determined based on the first average value and the second average value.

11. The moving object detection method according to claim 1, characterized in that, The step of determining multiple target regions in the target optical flow map based on the first velocity includes: If the first velocity of any sub-region is greater than a preset velocity threshold, the sub-region is designated as the target sub-region. The connected regions where each target sub-region is located are determined based on the connected graph algorithm, and the connected regions are taken as the target regions, wherein each connected region includes at least one target sub-region.

12. The moving object detection method according to claim 11, characterized in that, Before determining the connected region to which each target sub-region belongs based on the connected graph algorithm, the method further includes noise filtering of the target optical flow graph, including: The optical flow values ​​of isolated target sub-regions at non-boundary locations in the target optical flow map are updated to a preset first value. The isolated target sub-regions include target sub-regions that are not adjacent to other target sub-regions.

13. The moving object detection method according to claim 1, characterized in that, The method further includes: The outliers in the first velocity of all sub-regions in each target region are cleaned up, and the third velocity of each target region is determined based on the average value of the cleaned first velocities. Based on the sliding window algorithm, the average value of multiple sliding windows corresponding to the first velocity of all sub-regions of each target region is determined, and the fourth velocity of each target region is determined according to the maximum value among the multiple sliding window average values.

14. The moving object detection method according to claim 1, characterized in that, The method further includes: determining the shooting state of the shooting device based on the target optical flow map, including: Determine the ratio of the number of all target sub-regions in the target optical flow map to the total number of all sub-regions in the target optical flow map, wherein the target sub-region represents a sub-region whose first velocity is greater than a preset velocity threshold; If the ratio is greater than a preset ratio threshold, the shooting state of the shooting device is determined to be a motion state; or... If the ratio is less than or equal to the ratio threshold, the shooting state of the shooting device is determined to be approximately static.

15. The moving object detection method according to claim 14, characterized in that, The method further includes: determining the second speed based on the shooting state of the shooting device, including: If the shooting state is the near-static state, and the fourth velocity of each target area is less than the product of a preset third value and the third velocity of each target area, the second velocity is determined based on the fourth velocity; or, if the shooting state is the near-static state, and the fourth velocity of each target area is greater than or equal to the product of the third value and the third velocity of each target area, the second velocity is determined based on the third velocity; or, If the shooting state is the motion state, and the fourth speed of each target area is less than the product of a preset fourth value and the third speed of each target area, the second speed is determined based on the fourth speed; or, if the shooting state is the motion state, and the fourth speed of each target area is greater than or equal to the product of the fourth value and the third speed of each target area, the second speed is determined based on the third speed.

16. The moving object detection method according to claim 1, characterized in that, The method further includes: performing noise reduction processing on the target region, including: All target regions are sorted in descending order of area, and a predetermined number of target regions are selected from the sorted sequence. Noise regions are then identified from the selected target regions.

17. The moving object detection method according to claim 1 or 16, characterized in that, Determining the noise region within the target region includes: Determine the corresponding region of each target region in the first image frame or the second image frame, and obtain the traditional difference map between the two image frames for the corresponding region, wherein each pixel in the traditional difference map corresponds to a difference value; Determine the average difference value of all pixels in the traditional difference map corresponding to each target region; The difference threshold corresponding to each target region is determined based on the number of sub-regions within each target region; The target region corresponding to the average difference value that is less than or equal to the corresponding difference threshold is defined as the noise region.

18. The moving object detection method according to claim 17, characterized in that, The difference threshold corresponding to each target region is determined according to the following formula: ， in, This represents the difference threshold. This indicates the number of sub-regions within each target region. , , , , This indicates the preset parameters.

19. A terminal device, characterized in that, The terminal device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the terminal device performs the moving object detection method as described in any one of claims 1 to 18.

20. A computer storage medium, characterized in that, The computer storage medium stores program instructions that, when executed on a terminal device, cause the processor of the terminal device to perform the moving object detection method as described in any one of claims 1 to 18.

Images