Video-based target motion state estimation method and related device

Abstract

The invention discloses a video-based target motion state estimation method and a related device, and relates to the field of image processing, and the method comprises the steps: carrying out the target detection of a real-time monitoring image, obtaining the information of a target bounding box, and obtaining a target motion state estimation result based on the coordinate change of the information of the target bounding box at each moment on a time axis; and calculating to obtain a center coordinate difference corresponding to the target bounding box at adjacent moments, performing normalization processing on the horizontal direction and the vertical direction of the center coordinate difference to obtain a target moving speed of the target object, and further determining a target motion state of the target object. The process does not need to depend on external equipment, early-stage calibration or complex data training work, the motion state of the target object is evaluated in a speed normalization mode, operation is easy and convenient, and various scene requirements can be met.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for estimating target motion state based on video and related devices. Background Art

[0002] In the field of image monitoring, it is necessary to capture real-time images based on surveillance cameras, and then use detection and tracking technology to accurately track the target and draw its real-time trajectory. On this basis, the target's movement speed is further calculated and its movement state is comprehensively evaluated. At present, two main technical solutions have been adopted:

[0003] Integrated speed measurement method integrating speed measurement equipment: This solution first requires precise pre-calibration of the surveillance camera and speed measurement equipment (such as depth camera, laser rangefinder, etc.). Through this complex preliminary preparation, the speed of the target can be measured in real time, and its motion state can be estimated accordingly. This method has been reflected in many patents, such as CN104915628A and CN105184274B. However, the introduction of speed measurement equipment not only increases the complexity of the system, but the cumbersome calibration process and the accuracy requirements also pose a challenge to the accuracy of the final result.

[0004] State detection method based on model training: In this scheme, a state detection model is first established, and then trained to enable it to recognize and judge the real-time state of the target. When the real-time target image is input into this trained model, the model will output the judgment result of the target state. This method is also supported by patents, such as CN111291632B. However, the limitation of this method is that its applicable scenarios are limited. When faced with new scenarios, the model often needs to be retrained, which undoubtedly increases the difficulty of application. In addition, the lack of visualization of the intermediate states in the model evaluation process makes the adjustment of parameters particularly difficult. Summary of the invention

[0005] In view of the above problems, the present application provides a video-based target motion state estimation method and related devices to provide a target motion state estimation solution that does not require external equipment, preliminary calibration or complex data training.

[0006] The first aspect of the present application provides a method for estimating target motion state based on video, comprising:

[0007] Obtain real-time monitoring images;

[0008] Performing target detection on the real-time monitoring image to obtain target bounding box information;

[0009] Convert the target bounding box information into center coordinates;

[0010] Calculate the difference between the center coordinates of adjacent real-time monitoring images on the time axis, and record it as the center coordinate difference;

[0011] The calculated center coordinate differences are normalized in the horizontal and vertical directions;

[0012] Based on the processing result of the normalization processing, the target moving speed is calculated;

[0013] The target motion state is identified based on the target movement speed.

[0014] Optionally, in the above-mentioned video-based target motion state estimation method, converting the target bounding box information into center coordinates includes:

[0015] Based on the formula Px=x+w / 2 and the formula Py=y+h, the target bounding box information is converted into the center coordinates (Px, Py);

[0016] Where x is the horizontal pixel coordinate of the upper left corner of the bounding box in the target bounding box information;

[0017] y is the vertical pixel coordinate of the upper left corner of the bounding box in the target bounding box information;

[0018] w is the bounding box length in the target bounding box information;

[0019] h is the bounding box width in the target bounding box information.

[0020] Optionally, in the above-mentioned method for estimating the target motion state based on video, pixel normalization processing is performed on each calculated center coordinate difference in the horizontal direction and the vertical direction, including:

[0021] Based on the formula Fx=X / w, each calculated center coordinate difference is subjected to pixel normalization processing in the horizontal direction;

[0022] Based on the formula Fy=Y / tan(T) / w, each calculated center coordinate difference is subjected to pixel normalization in the vertical direction;

[0023] Wherein, Fx is the normalized velocity in the horizontal direction;

[0024] Fy is the normalized velocity in the vertical direction;

[0025] X represents the horizontal component of the center coordinate difference;

[0026] Y represents the component of the center coordinate difference in the vertical direction;

[0027] T is the pitch angle of the camera at the center coordinate.

[0028] Optionally, in the above-mentioned method for estimating the target motion state based on video, identifying the target motion state based on the target moving speed includes:

[0029] Determining the speed range to which the target moving speed belongs;

[0030] The motion state corresponding to the speed range to which the target moving speed belongs is taken as the target motion state.

[0031] Optionally, in the above-mentioned method for estimating the target motion state based on video, the motion state corresponding to the speed range to which the target moving speed belongs is taken as the target motion state;

[0032] When it is determined that the speed range to which the target moving speed belongs is greater than 0.6, the target movement state is determined to be a normal walking state;

[0033] When it is determined that the target moving speed belongs to a speed range of [0.12, 0.6], the target moving state is determined to be a slow moving state;

[0034] When it is determined that the speed range to which the target moving speed belongs is less than 0.12, it is determined that the target moving state is a stopped state.

[0035] A device for estimating target motion state based on video, comprising:

[0036] An image acquisition unit, used to obtain real-time monitoring images;

[0037] A bounding box recognition unit, used to perform target detection on the real-time monitoring image to obtain target bounding box information;

[0038] A center coordinate calculation unit, used to convert the target bounding box information into center coordinates;

[0039] A center difference calculation unit is used to calculate the difference of center coordinates corresponding to adjacent real-time monitoring images on the time axis, which is recorded as center coordinate difference;

[0040] A normalization processing unit, used for performing pixel normalization processing on each calculated center coordinate difference in the horizontal direction and the vertical direction;

[0041] A moving speed calculation unit, used for calculating the target moving speed based on the processing result of the normalization processing;

[0042] The motion state recognition unit is used to recognize the target motion state based on the target moving speed.

[0043] A computer program product includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the above-mentioned target motion state estimation methods.

[0044] An electronic device comprises at least one processing device and a storage device connected to the processing device, wherein:

[0045] The storage device is used to store computer programs;

[0046] The processing device is used to execute the computer program so that the electronic device can implement the target motion state estimation method as described in any one of the above.

[0047] Optionally, the electronic device is a portable electronic device.

[0048] A computer storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the target motion state estimation methods described above.

[0049] By means of the above technical solution, the above solution provided by the present application detects the target bounding box information of the target object in the monitoring image, and based on the change of the coordinates of the target bounding box information, calculates the center coordinate difference corresponding to the adjacent target bounding boxes on the time axis, and normalizes the horizontal and vertical directions of the center coordinate difference, so as to obtain the moving speed of the target object, and then determine the target motion state of the target object. This process does not require external equipment, preliminary calibration or complex data training work, and evaluates the motion state of the target object by speed normalization. It is easy to operate and can adapt to various scene requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0051] Figure 1 A system framework diagram for implementing a video-based target motion state estimation solution provided in this application;

[0052] Figure 2 A schematic diagram of the structure of a terminal for implementing a video-based target motion state estimation solution provided by the present application;

[0053] Figure 3 A schematic diagram of the structure of a server for implementing a video-based target motion state estimation solution provided in the present application;

[0054] Figure 4A schematic diagram of a flow chart of a method for estimating a target motion state based on video disclosed in an embodiment of the present application;

[0055] Figure 5 This is the normalized principle diagram of the center coordinate difference in the horizontal direction;

[0056] Figure 6 This is the normalized principle diagram of the center coordinate difference in the vertical direction;

[0057] Figure 7 A schematic diagram of the structure of a video-based target motion state estimation device disclosed in an embodiment of the present application;

[0058] Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] The present application can be applied in the field of image processing. Taking image processing as an example, multiple application scenarios of products will be introduced below.

[0063] First, the application scenario of this application is introduced.

[0064] This application can be applied, but is not limited to, to applications with image processing capabilities or cloud services provided by cloud-side servers.

[0065] See also Figure 1 , Figure 1A schematic diagram of a system architecture for implementing a video-based target motion state estimation solution is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided by the embodiment of the present application for one or more terminals. Among them, the terminal 100 can independently implement the various steps of the video-based target motion state estimation scheme, and can also be combined with the server 200 to implement the various steps of the video-based target motion state estimation scheme. When combined with the server 200, the terminal 100 can be an image acquisition device or an image transmission device for sending the collected real-time monitoring image to the server. When the terminal 100 independently performs the various steps of the video-based target motion state estimation scheme, the terminal 100 can be an image acquisition device or an intelligent terminal connected to the image acquisition device.

[0066] Among them, the terminal 100 can be installed with an application for implementing a video-based target motion state estimation method. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters input by the user on the interactive interface (the relevant parameters can be real-time monitoring images captured by the image acquisition device), and send the above parameters to the server 200. The server 200 can obtain processing results (identified target motion state) based on the received parameters, and return the processing results to the terminal 100.

[0067] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0068] Next describe Figure 1 The product form of the mid-terminal 100;

[0069] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0070] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0071] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0072] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0073] Among them, the input device 132 can receive input data and the like.

[0074] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display the human-computer interaction interface, processing results, etc.

[0075] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0076] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0077] Among them, the memory 120 can be used to store relevant software codes for implementing the video-based target motion state estimation method, the processor 170 can execute the steps of the video-based target motion state estimation method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0078] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0079] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0080] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0081] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0082] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0083] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0084] Next describe Figure 1 The product form of the server 200;

[0085] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0086] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0087] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0088] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0089] Among them, the memory 204 can be used to store software codes related to implementing the video-based target motion state estimation method, the processor 202 can execute the relevant steps of the chip's video-based target motion state estimation method, and can also schedule other units to implement corresponding functions.

[0090] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0091] In order to solve the above problems, the embodiment of the present application provides a method for estimating the target motion state based on video. The method for estimating the target motion state based on video according to the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0092] Reference Figure 4 , Figure 4 The process diagram of a method for estimating target motion state based on video provided in an embodiment of the present application is as follows: Figure 4 As shown, a data processing method provided in an embodiment of the present application may include steps S401 to S407, and these steps are described in detail below.

[0093] S401: Acquire real-time monitoring images.

[0094] In this step, the real-time monitoring image is obtained based on the monitoring camera. The whole process usually involves multiple links such as image acquisition, transmission, processing and display. When acquiring images, monitoring cameras generally use high-definition cameras, which can capture high-resolution videos and images and provide clear picture quality. The imaging module (such as CCD or CMOS) in the high-definition camera is a key component, which is responsible for converting optical signals into electrical signals to achieve high-quality image acquisition. The monitoring camera can be equipped with an electric zoom lens, which can adjust the focal length according to actual needs to achieve clear imaging of distant targets. The monitoring camera needs to be connected to the network (such as a wired network or WiFi) and configured through the corresponding software or APP to ensure that it can work normally and transmit image data. For monitoring cameras in fixed positions, wired transmission methods (such as Ethernet) can be used to transmit real-time monitoring images to related devices that apply the video-based target motion state estimation method. For monitoring scenarios that need to be moved or cables cannot be laid, wireless transmission methods (such as WiFi, 4G / 5G, etc.) can be used to transmit the collected real-time monitoring images to related devices that apply the video-based target motion state estimation method.

[0095] S402: Perform target detection on the real-time monitoring image to obtain target bounding box information.

[0096] After acquiring the real-time monitoring image, the existing mature target detection technology can be applied to the real-time monitoring image to detect the target bounding box information. The specific target bounding box information recognition scheme may include:

[0097] 1. Solutions based on traditional image processing

[0098] Feature extraction and matching:

[0099] By extracting features from the image (such as edges, corners, textures, etc.) and matching them with a preset feature library, the location of the target object can be determined.

[0100] Feature extraction methods include SIFT, SURF, HOG, etc.

[0101] Matching algorithms such as template matching and RANSAC can be used to find areas in the image that are similar to the target features, thereby determining the target bounding box.

[0102] Color and shape analysis:

[0103] Search and match the target object in the image based on its color, shape and other features.

[0104] For example, you can use a color histogram for color matching, or a Shape Context for shape matching.

[0105] 2. Machine learning-based solutions

[0106] Classifier training:

[0107] By training a classifier (such as SVM, decision tree, random forest, etc.), pixels or areas in the image are classified as target objects or background.

[0108] Training a classifier requires a large amount of labeled image data.

[0109] Sliding Window Method:

[0110] Slide a fixed-size window across the image and classify each image patch within the window.

[0111] According to the classification results, the position and size of the target object are determined to obtain the target bounding box.

[0112] 3. Solutions based on deep learning

[0113] Convolutional Neural Networks (CNN):

[0114] Use convolutional neural networks to extract features and classify images.

[0115] CNN can automatically learn features in images through a large amount of training data without the need to manually design feature extraction algorithms.

[0116] Object Detection Framework:

[0117] Such as R-CNN, Fast R-CNN, Faster R-CNN, YOLO, SSD, etc. These frameworks combine the advantages of region proposal and CNN to achieve efficient target detection.

[0118] They can directly output the location (i.e., target bounding box) and category of the target object.

[0119] One-stage object detector:

[0120] For example, the YOLO series can predict the bounding boxes and categories of multiple targets through a single forward propagation, which is fast.

[0121] 4. Solutions based on semi-supervised and unsupervised learning

[0122] Weakly supervised learning:

[0123] When the training data is only partially labeled or the labels are inaccurate, weakly supervised learning methods are used for object detection.

[0124] For example, use image-level labels to train the model instead of object-level labels.

[0125] Self-supervised learning:

[0126] The statistical characteristics or contextual information of the image itself is used to train the model without manual annotation.

[0127] The application of self-supervised learning in object detection is gradually being explored and developed.

[0128] For the detection results of the above target bounding box information, the existing mature target tracking solution is used to realize the front and back association of the target, so that the bounding box information (x, y, w, h) of the target object at each time node within a period of time can be obtained, where x is the horizontal pixel coordinate of the upper left corner of the bounding box in the target bounding box information; y is the vertical pixel coordinate of the upper left corner of the bounding box in the target bounding box information; w is the length of the bounding box in the target bounding box information; h is the width of the bounding box in the target bounding box information.

[0129] S403: Convert the target bounding box information into center coordinates.

[0130] In this step, after determining the target bounding box information (x, y, w, h), coordinate conversion is performed on it to convert it into the center coordinate. Here, the center coordinate may refer to the bottom center coordinate, top center coordinate, left center coordinate, right center coordinate or positive center coordinate of the target bounding box information. In this solution, the bottom center coordinate is preferred, that is, the target bounding box information (x, y, w, h) is converted into the bottom center coordinate (Px, Py);

[0131] Px=x+w / 2

[0132] Py=y+h

[0133] Bottom center coordinates (Px, Py) = (x+w / 2, y+h)

[0134] S404: Calculate the difference between the center coordinates of the adjacent real-time monitoring images on the time axis, and record it as the center coordinate difference.

[0135] This step is used to calculate the difference between the center coordinates of adjacent real-time monitoring images on the time axis, calculate the pixel speed of the center coordinates in these two adjacent time nodes, and decompose the pixel speed of the center coordinate to obtain the pixel speed of the center coordinate in the horizontal and vertical directions.

[0136] Calculate the difference between the center coordinates of adjacent real-time monitoring images on the time axis, that is, calculate the pixel speed of the center coordinate. In this solution, the time difference on the time axis between adjacent real-time monitoring images can be the time difference of the image acquisition time. In this solution, the real-time monitoring image can be acquired every one second or other time period. At this time, the time difference on the time axis between the real-time monitoring images is 1 second. By subtracting the center coordinates, the speed per second (X, Y) of the center coordinate can be obtained in pixels.

[0137] Assume that the center coordinates of two adjacent real-time monitoring images on the time axis are (Px1, Py1) and (Px2, Py2) respectively, and make a difference between (Px1, Py1) and (Px2, Py2) to obtain the pixel speed (X, Y) of the center coordinates;

[0138] X=Px1-Px2

[0139] Y=Py1-Py2

[0140] Wherein, X is the pixel speed of the center coordinate in the horizontal direction, and Y is the pixel speed of the center coordinate in the vertical direction.

[0141] S405: performing pixel normalization processing on each calculated center coordinate difference in the horizontal direction and the vertical direction.

[0142] In this step, each center coordinate difference calculated is normalized in the horizontal direction and the vertical direction respectively.

[0143] When normalizing in the horizontal direction, Fx=X / w, where Fx is the normalized velocity of the center coordinate in the horizontal direction, see Figure 5 For a target object moving horizontally in the picture, with the same displacement of 1m, when the target object moves near, the target object in the monitoring image moves more pixels in the picture; when the target object moves far, the target object in the monitoring image moves less pixels in the picture. Figure 5 In the figure, the dotted area between AB is the area of ​​the target bounding box, AB is the horizontal length of the target object, BC is the displacement of 1m, OH is the vertical distance between the target object and the screen, OH' is the focal length, A' B' C' are the projection positions of ABC in the screen, respectively. According to the principle of similar triangles, we can get

[0144] AB / A'B'=OH / OH'

[0145] BC / B'C'=OH / OH'

[0146] Further

[0147] AB / A'B'=BC / B'C'

[0148] So

[0149] A'B' / B'C'=AB / BC

[0150] Since AB and BC are real distances, the ratio at any position is a fixed value, so A'B' / B'C' is also a fixed value, that is, the value obtained by dividing the pixel distance moved in the picture by the pixel length of the target in the picture is a fixed value. Furthermore, the distance divided by the time is the speed, from which we can conclude that for a target object, at any position in the picture, the target object moves at a fixed speed, and its pixel speed in the picture divided by the pixel length is a fixed value, so this parameter can be used as a unified metric to evaluate the speed of the target.

[0151] For example, if the target object moves horizontally at a speed of 1m / s, its horizontal length is 0.4m, and 1 / 0.4=2.5. Then when the target object is 10m, 20m, or even 50m away from the camera, the pixel speed of the camera's horizontal movement in the picture divided by the pixel length is 2.5, and remains unchanged. 2.5 can represent the moving speed of the target object. At this time, the pixel speed in the horizontal direction can represent the actual moving speed of the target object compared to the camera in the horizontal direction. Therefore, in this solution, the pixel speed in the horizontal direction is used as the horizontal moving speed of the target object compared to the camera.

[0152] By normalizing the pixels of each center coordinate difference in the horizontal direction using the formula Fx=X / w, the horizontal moving speed-time curve of the target object compared to the camera in the horizontal direction can be obtained.

[0153] When the pixel speed of each center coordinate difference in the vertical direction is only normalized, the pixel normalization process is performed on each center coordinate difference calculated in the vertical direction based on the formula Fy=Y / tan(T) / w;

[0154] Fy is the normalized velocity;

[0155] T is the corresponding pitch angle of the camera at the target position (center coordinate);

[0156] T is calculated as follows:

[0157] Given the camera's current pitch angle t, field of view viewt, resolution (width, height) and current target center coordinates (Px, Py), the following is a detailed explanation of each step and the necessary Python code implementation (if applicable):

[0158] a. Calculate the distance dis between the camera center and the image center

[0159] Formula: dis=height / 2 / tan(view_t / 2);

[0160] Height represents the height of the image, that is, the number of pixels or actual size of the image in the vertical direction.

[0161] view_t represents the vertical field of view of the camera, that is, the angular range of the scene that the camera lens can capture in the vertical direction.

[0162] This distance is the vertical distance from the camera to the center point of its field of view.

[0163] b. Calculate the offset of the center coordinates relative to the center of the screen

[0164] Formula: dx=Px-width / 2

[0165] dy=Py-height / 2

[0166] Width represents the width of the image, that is, the number of pixels or actual size of the image in the horizontal direction.

[0167] These offsets represent the position of the target point on the image plane relative to the center of the image.

[0168] c. Calculate the three-dimensional coordinates of the center point (Wx, Wy, Wz)

[0169] Here, rad should be the distance from the camera to the target plane (i.e. the radial distance from the camera to the target point). Usually, this distance needs to be obtained by other means (such as depth sensor, stereo vision, known object size, etc.).

[0170] Assuming rad is known, then:

[0171] Formula: Wx=dx Wy=rad×cos(t)−dy×sin(t) Wz=rad×sin(t)+dy×cos(t)

[0172] Note: Wx, Wy, Wz here are coordinates in a coordinate system with the camera as the origin, the camera's line of sight as the Z axis, the horizontal rightward direction as the X axis, and the vertical upward direction as the Y axis. This coordinate system is hypothetical and may need to be adjusted according to actual conditions.

[0173] d. Calculate the pitch angle T of the center coordinate

[0174] formula:

[0175] This formula calculates the angle between the line from the camera to the target point and the horizontal plane (i.e. the pitch angle). At this point, the pitch angle T corresponding to the camera at the center coordinate is calculated. Substituting the pitch angle T into the formula Fy=Y / tan(T) / w, the calculated center coordinate differences can be normalized in the vertical direction.

[0176] like Figure 6 As shown, Figure 6 CAB is located in the real coordinate system, CA is the actual displacement of the target, CA first needs to be projected to the position of CB, and then further pinhole imaging can be converted into the displacement h' of the screen pixel;

[0177] CB and h1 are the corresponding pinhole imaging relationships, so the image pixel displacement corresponding to the actual CA is:

[0178] h' / tan(T)

[0179] The distance h' is extended to the vertical velocity Y, and the vertical velocity is normalized in the same way as the horizontal velocity to obtain the normalized vertical velocity Fy.

[0180] Fy=Y / tan(T) / w

[0181] For any position in the monitoring image, the Fx and Fy corresponding to the position are combined to obtain the final pixel velocity Fv of the center point. , Fv can be used to measure the moving speed of the target object.

[0182] S406: Based on the normalization processing result, the target moving speed is calculated.

[0183] After obtaining the normalized results of the center coordinate difference in the horizontal and vertical directions, based on the formula The normalized results Fx and Fy in the horizontal and vertical directions corresponding to the same center coordinate difference are processed to obtain the pixel speed corresponding to the center coordinate difference, and the pixel speed is used as the target moving speed of the target object on the monitoring image.

[0184] S407: Identify the target motion state based on the target moving speed.

[0185] In this solution, the user can pre-configure multiple motion states. Different motion states correspond to different target moving speeds of the target object. After determining the target moving speed of the target object, the motion state of the target object is also determined. These motion states may include but are not limited to normal walking, slow walking, and stationary states. In this solution, after calculating the target moving speed in step S406, a preset motion state that matches the target moving speed is obtained, thereby realizing the recognition of the motion state of the target object.

[0186] The current patents related to target motion state analysis all measure the target speed through speed measuring equipment (including speedometers, radars, depth cameras, etc.) to further evaluate the target motion state; or establish and train state analysis models to evaluate the motion state. The former requires mutual calibration between multiple devices, the preliminary work is complicated, and the accuracy is greatly limited by the calibration accuracy; the latter requires the collection of various data for training, and also needs to be optimized for different scenes, which is costly and difficult to adjust. The video-based target motion state estimation method proposed in this application detects the target bounding box information of the target object in the monitoring image, and based on the change of the coordinates of the target bounding box information, calculates the center coordinate difference corresponding to the adjacent target bounding boxes on the time axis, and normalizes the horizontal and vertical directions of the center coordinate difference to obtain the moving speed of the target object, and then determine the target motion state of the target object. This process does not require external equipment, preliminary calibration or complex data training work, and evaluates the motion state of the target object by speed normalization. It is easy to operate and can adapt to various scene requirements.

[0187] In this embodiment, before identifying the target motion state based on the target moving speed, multiple motion states can be pre-selected and divided, and each motion state corresponds to a speed range. By comparing the speed range to which the target moving speed belongs, the moving state of the target object can be determined. The process of identifying the target motion state based on the target moving speed can specifically be: determining the speed range to which the target moving speed belongs; and using the motion state corresponding to the speed range to which the target moving speed belongs as the target motion state of the target object.

[0188] In a specific embodiment, the speed range may include greater than 0.6, [0.12, 0.6] and less than 0.12. When the speed range to which the target moving speed belongs is greater than 0.6, the target motion state is determined to be a normal walking state; when the speed range to which the target moving speed belongs is determined to be [0.12, 0.6], the target motion state is determined to be a slow moving state; when the speed range to which the target moving speed belongs is determined to be less than 0.12, the target motion state is determined to be a stopped state. Of course, the user can also set multiple other motion states and speed ranges that match the motion states according to design requirements, so that this solution can obtain a more accurate motion state of the target object.

[0189] In a possible implementation, a video-based target motion state estimation method provided by an embodiment of the present application is introduced above, and a device for executing the above video-based target motion state estimation method will be introduced below.

[0190] See also Figure 7 , Figure 7 The structure diagram of a video-based target motion state estimation device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the video-based target motion state estimation device includes:

[0191] An image acquisition unit 10 is used to acquire real-time monitoring images;

[0192] The bounding box recognition unit 20 is used to perform target detection on the real-time monitoring image to obtain target bounding box information;

[0193] A center coordinate calculation unit 30, used to convert the target bounding box information into center coordinates;

[0194] The center difference calculation unit 40 is used to calculate the difference of the center coordinates corresponding to the adjacent real-time monitoring images on the time axis, which is recorded as the center coordinate difference;

[0195] A normalization processing unit 50 is used to perform pixel normalization processing on each calculated center coordinate difference in the horizontal direction and the vertical direction;

[0196] A moving speed calculation unit 60, configured to calculate a target moving speed based on a processing result of the normalization processing;

[0197] The motion state recognition unit 70 is used to recognize the target motion state based on the target moving speed.

[0198] In the above device, the specific working process of each unit can be found in the introduction of the above method embodiment, and will not be described here in detail.

[0199] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may be a portable mobile terminal, which may include but is not limited to a mobile phone, a notebook computer, a PDA (personal digital assistant), a PAD (tablet computer), and of course a fixed terminal such as a desktop computer. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0200] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0201] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0202] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the video-based target motion state estimation methods provided in the embodiments of the present application.

[0203] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any video-based target motion state estimation method provided in an embodiment of the present application.

[0204] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0205] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0206] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0207] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for estimating target motion state based on video, characterized in that: include: Obtain real-time monitoring images; Performing target detection on the real-time monitoring image to obtain target bounding box information; Convert the target bounding box information into center coordinates; Calculate the difference between the center coordinates of adjacent real-time monitoring images on the time axis, and record it as the center coordinate difference; The calculated center coordinate differences are normalized in the horizontal and vertical directions; Based on the processing result of the normalization processing, the target moving speed is calculated; The target motion state is identified based on the target movement speed.

2. The method for estimating target motion state based on video according to claim 1, characterized in that: Converting the target bounding box information into center coordinates includes: Based on the formula Px=x+w / 2 and the formula Py=y+h, the target bounding box information is converted into the center coordinates (Px, Py); Where x is the horizontal pixel coordinate of the upper left corner of the bounding box in the target bounding box information; y is the vertical pixel coordinate of the upper left corner of the bounding box in the target bounding box information; w is the bounding box length in the target bounding box information; h is the bounding box width in the target bounding box information.

3. The method for estimating target motion state based on video according to claim 2, characterized in that: The calculated center coordinate differences are normalized in the horizontal and vertical directions, including: Based on the formula Fx=X / w, each calculated center coordinate difference is subjected to pixel normalization processing in the horizontal direction; Based on the formula Fy=Y / tan(T) / w, each calculated center coordinate difference is subjected to pixel normalization in the vertical direction; Wherein, Fx is the normalized velocity in the horizontal direction; Fy is the normalized velocity in the vertical direction; X represents the horizontal component of the center coordinate difference; Y represents the component of the center coordinate difference in the vertical direction; T is the pitch angle of the camera at the center coordinate.

4. The method for estimating target motion state based on video according to claim 3, characterized in that: Identifying the target motion state based on the target moving speed includes: Determining the speed range to which the target moving speed belongs; The motion state corresponding to the speed range to which the target moving speed belongs is taken as the target motion state.

5. The method for estimating target motion state based on video according to claim 4, characterized in that: The motion state corresponding to the speed range to which the target moving speed belongs is taken as the target motion state; When it is determined that the speed range to which the target moving speed belongs is greater than 0.6, the target movement state is determined to be a normal walking state; When it is determined that the target moving speed belongs to a speed range of [0.12, 0.6], the target moving state is determined to be a slow moving state; When it is determined that the speed range to which the target moving speed belongs is less than 0.12, it is determined that the target moving state is a stopped state.

6. A device for estimating target motion state based on video, characterized in that: include: An image acquisition unit, used to obtain real-time monitoring images; A bounding box recognition unit, used to perform target detection on the real-time monitoring image to obtain target bounding box information; A center coordinate calculation unit, used to convert the target bounding box information into center coordinates; A center difference calculation unit is used to calculate the difference of center coordinates corresponding to adjacent real-time monitoring images on the time axis, which is recorded as center coordinate difference; A normalization processing unit, used for performing pixel normalization processing on each calculated center coordinate difference in the horizontal direction and the vertical direction; A moving speed calculation unit, used for calculating the target moving speed based on the processing result of the normalization processing; The motion state recognition unit is used to recognize the target motion state based on the target moving speed.

7. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the method for estimating the target motion state based on video as claimed in any one of claims 1 to 5.

8. An electronic device, characterized in that: The method comprises at least one processing device and a storage device connected to the processing device, wherein: The storage device is used to store computer programs; The processing device is used to execute the computer program so that the electronic device can implement the video-based target motion state estimation method as described in any one of claims 1 to 5.

9. The electronic device according to claim 8, characterized in that: The electronic device is a portable electronic device.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the video-based target motion state estimation method as described in any one of claims 1 to 5.

Images