Construction machinery statistical analysis method, device and product based on video monitoring

Abstract

The embodiment of the invention provides a construction machinery statistical analysis method and device based on video monitoring and a product. The method specifically comprises the following steps: performing character recognition on an initial video stream image; determining a target working mode of the acquisition equipment according to the identified acquisition place information; if the collection site information corresponds to a preset large-range scene, the target working mode is a moving mode; acquiring a video stream image in the target working mode; performing feature extraction on the regional image in the bounding box to obtain a first appearance feature vector including a texture feature and a color feature; the weight of the texture feature is greater than that of the color feature; determining the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the analysis result of the construction machine is determined for the construction machine corresponding to the trajectory ID of the confirmed state. According to the embodiment of the invention, the accuracy of construction machinery counting analysis in a hydropower engineering scene can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of video surveillance of hydropower projects, and in particular to a construction machinery statistical analysis method, device and product based on video surveillance. Background Art

[0002] During the construction of hydropower projects, video surveillance systems are often deployed. Compared with installing positioning equipment on each construction machine, the analysis of the number and type of construction machines entering the site based on the video surveillance system can timely assist the construction management of hydropower projects with less investment and operation and maintenance costs.

[0003] At present, the research on vehicle counting based on video surveillance mainly focuses on traffic scenes. In specific implementation, a monocular camera is usually used to capture surveillance videos with a fixed viewing angle, and a target detection algorithm is used to accurately identify vehicles, and then a target tracking algorithm is combined to achieve continuous tracking of vehicles. In addition, by setting a virtual detection line, the vehicle flow on a specified road section can be effectively counted.

[0004] However, there are significant differences between traffic scenes and hydropower engineering scenes. In traffic scenes, the similarity between vehicles in the surveillance videos collected within a period of time is relatively low. However, at the construction site of a hydropower project, the models and appearance colors of construction machinery of the same category are often highly similar, which makes the vehicle counting method in traffic scenes face challenges in hydropower engineering scenes, and easily leads to a significant reduction in the accuracy of vehicle counting results in hydropower engineering scenes. Summary of the invention

[0005] The embodiment of the present application provides a construction machinery statistical analysis method based on video surveillance, which can improve the accuracy of construction machinery counting analysis in hydropower engineering scenarios.

[0006] Correspondingly, the embodiments of the present application also provide a construction machinery statistical analysis device based on video surveillance, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application discloses a construction machinery statistical analysis method based on video monitoring, the method comprising:

[0008] Perform text recognition on the initial video stream image of the construction area to be counted, and the obtained text recognition result includes: collection location information;

[0009] Determine the target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode;

[0010] Acquire the video stream image acquired by the acquisition device in the target working mode;

[0011] Performing target detection on the i-th frame of the video stream image, the target detection result obtained includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set includes: at least one bounding box; i is a positive integer;

[0012] For the bounding box in the first frame of the video stream image, create a track ID in the track collection;

[0013] Extracting features of a region image within a bounding box in an i-th frame of a video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector includes: a texture feature and a color feature; a weight of the texture feature is greater than a weight of the color feature;

[0014] Determine the predicted trajectory of the trajectory ID in the trajectory set at the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set;

[0015] Determine the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state;

[0016] Assign a machine ID to the construction machine corresponding to the trajectory ID of the confirmed status;

[0017] According to the machine identification, the analysis results of the construction machines in the construction area to be counted are determined.

[0018] The present application also discloses a construction machinery statistical analysis device based on video monitoring, the device comprising:

[0019] The text recognition module is used to perform text recognition on the initial video stream image of the construction area to be counted. The obtained text recognition result includes: collection location information;

[0020] A working mode module, used to determine a target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode;

[0021] An image acquisition module, used to acquire a video stream image acquired by an acquisition device in the target working mode;

[0022] The target detection module is used to perform target detection on the i-th frame of the video stream image, and the target detection result obtained includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set includes: at least one bounding box; i is a positive integer;

[0023] A track ID creation module is used to create a track ID in a track set for a bounding box in the first frame of the video stream image;

[0024] An appearance feature extraction module is used to extract features from a region image within a bounding box in an i-th frame of a video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature;

[0025] A trajectory prediction module, used to determine the predicted trajectory of the trajectory ID in the trajectory set in the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set;

[0026] A state determination module, used to determine the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state;

[0027] An identification allocation module, used to allocate a machine identification to the construction machine corresponding to the trajectory ID of the confirmed state;

[0028] The analysis module is used to determine the analysis results of the construction machinery in the construction area to be counted according to the machinery identification.

[0029] Optionally, the appearance feature extraction module includes:

[0030] A first extraction module, used for extracting a first feature vector of the regional image by using a convolution module;

[0031] A second extraction module is used to extract a second feature vector of the regional image using a local binary pattern device;

[0032] The fusion module is used to fuse the first feature vector and the second feature vector to obtain an appearance feature vector.

[0033] Optionally, the fusion module includes:

[0034] A weighting module is used to perform weighted processing on the same-dimensional features in the first feature vector and the second feature vector to obtain a weighted processing result; the same-dimensional features include: texture features;

[0035] The texture enhancement module is used to increase the corresponding value according to the weighted processing result corresponding to the texture feature.

[0036] Optionally, the state determination module includes:

[0037] A first matching module is used to perform a first match between a trajectory ID in a confirmed state in the trajectory set and a bounding box in the bounding box set to obtain a corresponding first matching result; wherein, in the first matching process, a cosine distance between a first appearance feature vector of the bounding box and a second appearance feature vector corresponding to the trajectory ID is determined, and a first matching result is determined according to the cosine distance; the first matching result includes: a trajectory ID for which the first matching succeeds, a trajectory ID for which the first matching fails, and a bounding box for which the first matching fails;

[0038] A second matching module, used for performing a second matching on the trajectory ID that failed the first matching or the trajectory ID in the unconfirmed state with the bounding box that failed the first matching, so as to obtain a corresponding second matching result; the second matching is an intersection-over-union matching; the second matching result includes: the trajectory ID that succeeded the second matching, the trajectory ID that failed the second matching, and the bounding box that failed the second matching;

[0039] A second track ID creation module, used for creating a track ID in a non-confirmed state in the track set for the bounding box where the second match fails;

[0040] The state updating module is used to update the state of the trajectory ID to the confirmed state when the second matching result of the trajectory ID in the unconfirmed state is a successful match, and delete the trajectory ID when the number of second matching failures of the trajectory ID in the confirmed state exceeds a first threshold.

[0041] Optionally, the first matching module includes:

[0042] a cosine distance determination module, configured to determine a cosine distance between a first appearance feature vector of a bounding box and a second appearance feature vector corresponding to a track ID whose state is a confirmation state;

[0043] A Mahalanobis distance determination module, used to determine the Mahalanobis distance between the bounding box and the track ID whose state is the confirmed state;

[0044] A matrix construction module, used to construct a cost matrix; the elements in the cost matrix represent the matching cost between a bounding box and a trajectory; the values ​​of the elements are determined according to the cosine distance and the Mahalanobis distance;

[0045] The solution module is used to solve the cost matrix with the goal of minimizing the total matching cost to obtain a first matching result.

[0046] Optionally, the device further comprises:

[0047] The lost age setting module is used to set the lost age of the trajectory ID to 0 after creating the trajectory ID;

[0048] A first lost age updating module, configured to set the lost age of the trajectory ID to 0 when the first matching result or the second matching result of the trajectory ID is a successful match;

[0049] A second lost age updating module, configured to increase the lost age of the trajectory ID by 1 when the second matching result of the trajectory ID in the confirmed state is a matching failure;

[0050] The first matching module includes:

[0051] A hierarchical division module is used to divide the trajectories in the trajectory set into multiple levels according to the loss age;

[0052] The layer-by-layer matching module is used to perform a first match between the track IDs in the track set with the confirmed state and the bounding boxes in the bounding box set layer by layer, starting from the layer corresponding to the lost age 0.

[0053] Optionally, the trajectory information corresponding to the trajectory ID includes: position information and a second appearance feature vector;

[0054] The device also includes:

[0055] The trajectory update module is used to update the position information corresponding to the trajectory ID according to the position information of the successfully matched bounding box when the first matching result or the second matching result of the trajectory ID is a successful match, and to update the second appearance feature vector of the trajectory ID according to the first appearance feature vector of the successfully matched bounding box.

[0056] Optionally, the device further comprises:

[0057] The deletion module is used to delete the trajectory ID whose status is unconfirmed and whose second matching result is second matching failure in the trajectory set.

[0058] Optionally, the first text recognition module includes:

[0059] An extraction module, used for extracting an image of a region of interest from an initial video stream image;

[0060] A space conversion module, used for converting the region of interest image into an HSV hue, saturation and brightness color space;

[0061] A target position determination module is used to determine a target position corresponding to white or approximately white according to the HSV value of the image of the region of interest in the HSV space;

[0062] A color negation module is used to perform a color channel negation operation on the pixel point at the target position;

[0063] A magnification module, used for magnifying the image of the region of interest after the color channel inversion operation;

[0064] The optical character recognition module is used to perform optical character recognition on the image of the region of interest after the zoom operation to obtain a text recognition result.

[0065] Optionally, the process of determining the movement mode includes:

[0066] Determine the acquisition space range corresponding to the acquisition device in the four directions of up, down, left, and right:

[0067] According to the acquisition space range, determine the angle and height of the acquisition device:

[0068] Determining the magnification of the acquisition device according to the acquisition space range and the angle and height of the acquisition device;

[0069] The moving path of the acquisition device is determined according to the acquisition space range, the angle and height of the acquisition device, and the magnification of the acquisition device.

[0070] Optionally, the target detection result further includes: the category of the construction machinery in the i-th frame of the video stream image; the analysis result includes: the number and category distribution of the construction machinery;

[0071] The analysis module is used to include:

[0072] A first analysis module is used to determine the number of construction machines in the construction area to be counted according to the type of the machine identification;

[0073] The second analysis module is used to determine the category distribution of the construction machinery in the construction area to be counted according to the type of the machinery identification and the category of the construction machinery corresponding to the machinery identification.

[0074] Optionally, the device further comprises:

[0075] The second text recognition module is used to perform text recognition on the i-th frame of the video stream image, and the obtained text recognition result includes: acquisition time information;

[0076] An appearance time determination module, used to determine the appearance time information corresponding to the mechanical identifier according to the acquisition time information contained in the video stream image to which the mechanical identifier belongs;

[0077] The saving module is used to save the mapping relationship between the machine identification and the appearance time information.

[0078] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0079] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0080] The embodiments of the present application include the following advantages:

[0081] The embodiment of the present application comprehensively utilizes multiple technical means such as image acquisition, target detection, appearance feature extraction of bounding boxes, trajectory tracking and status determination of trajectory ID to improve the accuracy of construction machinery counting and analysis in hydropower engineering scenarios.

[0082] First, for image acquisition means, the embodiment of the present application performs text recognition on the initial video stream image of the construction area to obtain the acquisition location information, and then determines the target working mode of the acquisition device, which is divided into mobile mode (corresponding to the preset large-scale scene) and static mode (corresponding to the preset small and medium-scale scene). In this way, the appropriate acquisition mode can be selected according to different construction area scenes, improving the pertinence and effectiveness of video acquisition.

[0083] Secondly, in the hydropower engineering scenario, construction machinery of the same category is similar in appearance, shape or color, which can easily lead to errors in the first matching process between the trajectory ID and the bounding box by the trajectory tracking means. In the process of calculating the first appearance feature vector of the image of the area within the bounding box, the embodiment of the present application controls the weight of the texture feature to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, and the color feature has a lower degree of distinction when the model and appearance color are similar, on this basis, the embodiment of the present application determines the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the trajectory ID. This can more accurately distinguish different construction machinery, reduce the probability of misjudgment due to similar appearance, and improve the distinction of construction machinery.

[0084] On the basis of improving the distinguishability of construction machinery, the embodiment of the present application can more accurately assign the mechanical identification of construction machinery, so that it can effectively prevent different construction machinery from being erroneously assigned the same mechanical identification. Therefore, the embodiment of the present application can improve the accuracy of construction machinery counting and analysis in hydropower engineering scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1It is a schematic diagram of an application environment of a construction machinery statistical analysis method based on video monitoring according to an embodiment of the present application;

[0086] Figure 2 It is a schematic diagram of the steps of a construction machinery statistical analysis method based on video monitoring according to an embodiment of the present application;

[0087] Figure 3 is a schematic diagram of a video stream image of a pumped storage power station according to an embodiment of the present application;

[0088] Figure 4 is a flowchart of the steps of a method for performing text recognition on an initial video stream image of a construction area to be counted according to an embodiment of the present application;

[0089] Figure 5 This is a flowchart of the steps of a method for extracting features of an area image within a boundary box in an i-th frame of a video stream image according to an embodiment of the present application;

[0090] Figure 6 It is a structural schematic diagram of a construction machinery statistical analysis device based on video monitoring according to an embodiment of the present application;

[0091] Figure 7 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0093] The embodiment of the present application can be applied to engineering industries such as hydropower and civil engineering, and can be used to perform statistical analysis on construction machinery in hydropower engineering scenarios based on video surveillance. The statistical analysis may include: counting analysis of construction machinery, etc.

[0094] The statistical analysis of construction machinery is of great significance. For example, the above statistical analysis can monitor the number and distribution of mechanical equipment at the construction site in real time to ensure the safe operation of construction machinery. This helps prevent mechanical equipment from overloading, collision or other safety accidents. For another example, counting analysis can help project managers understand the use of construction machinery, optimize resource allocation, avoid idleness or overuse of mechanical equipment, thereby improving resource utilization efficiency and reducing costs. For another example, counting analysis of construction machinery helps monitor the progress of the project. By understanding the working status and quantity of mechanical equipment in real time, project managers can better arrange construction plans and ensure that the project is completed on time.

[0095] At present, the research on vehicle counting based on video surveillance mainly focuses on traffic scenes. In specific implementation, a monocular camera is usually used to capture surveillance videos with a fixed viewing angle, and a target detection algorithm is used to accurately identify vehicles, and then a target tracking algorithm is combined to achieve continuous tracking of vehicles. In addition, by setting a virtual detection line, the vehicle flow on a specified road section can be effectively counted.

[0096] However, there are significant differences between traffic scenes and hydropower engineering scenes. In traffic scenes, the similarity between vehicles in the surveillance videos collected within a period of time is relatively low. However, at the construction site of a hydropower project, the models and appearance colors of construction machinery of the same category are often highly similar, which makes the vehicle counting method in traffic scenes face challenges in hydropower engineering scenes, and easily leads to a significant reduction in the accuracy of vehicle counting results in hydropower engineering scenes.

[0097] In order to solve the technical problem of low accuracy of vehicle counting results in hydropower engineering scenarios, the present application provides a construction machinery statistical analysis method based on video surveillance, which may specifically include the following steps:

[0098] Perform text recognition on the initial video stream image of the construction area to be counted, and the obtained text recognition result includes: collection location information;

[0099] Determine the target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode;

[0100] Acquire the video stream image acquired by the acquisition device in the target working mode;

[0101] Performing target detection on the i-th frame of the video stream image, the target detection result obtained includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set includes: at least one bounding box; i can be a positive integer;

[0102] For the bounding box in the first frame of the video stream image, create a track ID in the track collection;

[0103] Extracting features of a region image within a bounding box in an i-th frame of a video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector includes: a texture feature and a color feature; a weight of the texture feature is greater than a weight of the color feature;

[0104] Determine the predicted trajectory of the trajectory ID in the trajectory set at the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set;

[0105] Determine the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state;

[0106] Assign a machine ID to the construction machine corresponding to the trajectory ID of the confirmed status;

[0107] Based on the above-mentioned machinery identification, the analysis results of the construction machinery in the construction area to be counted are determined.

[0108] The embodiment of the present application comprehensively utilizes multiple technical means such as image acquisition, target detection, appearance feature extraction of bounding boxes, trajectory tracking and status determination of trajectory ID to improve the accuracy of construction machinery counting and analysis in hydropower engineering scenarios.

[0109] First, for image acquisition means, the embodiment of the present application performs text recognition on the initial video stream image of the construction area to obtain the acquisition location information, and then determines the target working mode of the acquisition device, which is divided into mobile mode (corresponding to the preset large-scale scene) and static mode (corresponding to the preset small and medium-scale scene). In this way, the appropriate acquisition mode can be selected according to different construction area scenes, improving the pertinence and effectiveness of video acquisition.

[0110] Secondly, in the hydropower engineering scenario, construction machinery specifically includes: dump trucks and excavators, etc. Construction machinery of the same category often has similar models, appearances, and colors, which brings challenges to the counting analysis of construction machinery. Specifically, the similarities between construction machinery of the same category in appearance, shape, or color can easily lead to errors in the first matching process between the trajectory ID and the bounding box by the trajectory tracking means. For example, when two or more construction machines are very similar in appearance, the trajectory tracking means may mistakenly assign the bounding box of one construction machine to the trajectory ID of another construction machine, or, when a construction machine with a trajectory ID disappears briefly and then reappears in the bounding box, the briefly disappeared bounding box may be mistakenly identified as a new trajectory ID.

[0111] In the process of calculating the first appearance feature vector of the image of the area within the bounding box, the embodiment of the present application controls the weight of the texture feature to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, while the color feature has a low degree of distinction when the model and appearance color are similar, on this basis, the embodiment of the present application determines the state of the track ID based on the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID, so that different construction machinery can be distinguished more accurately, the probability of misjudgment due to similar appearance is reduced, and the degree of distinction of construction machinery is improved.

[0112] On the basis of improving the distinguishability of construction machinery, the embodiment of the present application can more accurately assign the mechanical identification of construction machinery, so that it can effectively prevent different construction machinery from being erroneously assigned the same mechanical identification. Therefore, the embodiment of the present application can improve the accuracy of construction machinery counting and analysis in hydropower engineering scenarios.

[0113] Reference Figure 1 , showing a schematic diagram of the application environment of a construction machinery statistical analysis method based on video surveillance in one embodiment of the present application, wherein the image acquisition terminal 101 and the service terminal 102 can perform data exchange based on a wireless network or a wired network.

[0114] In practical applications, the image acquisition terminal 101 may be equipped with acquisition devices with image acquisition functions such as cameras and video cameras. The acquisition devices may acquire video stream images and send the video stream images to the server 102 according to a preset time period.

[0115] After receiving the video stream image sent by the image acquisition terminal 101, the server 102 can use the method of the embodiment of the present application to perform statistical analysis on the construction machinery in the hydropower engineering scene based on video monitoring. The obtained analysis results may include: the number of construction machinery and the category distribution of construction machinery.

[0116] Method Embodiment

[0117] refer to Figure 2 , shows a schematic flow chart of the steps of a construction machinery statistical analysis method based on video monitoring according to an embodiment of the present application, and the method may specifically include the following steps:

[0118] Step 201, performing text recognition on the initial video stream image of the construction area to be counted, and the obtained text recognition result includes: collection location information;

[0119] Step 202: Determine the target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode;

[0120] Step 203: acquiring the video stream image collected by the acquisition device in the above target working mode;

[0121] Step 204, performing target detection on the i-th frame of the video stream image, and obtaining a target detection result including: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set specifically includes: at least one bounding box; i is a positive integer;

[0122] Step 205: create a track ID in the track set for the bounding box in the first frame of the video stream image;

[0123] Step 206: extract features of the region image within the bounding box in the i-th frame of the video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector specifically includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature;

[0124] Step 207: Determine the predicted trajectory of the trajectory ID in the trajectory set in the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set;

[0125] Step 208: determining the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state;

[0126] Step 209: assign a machine identification to the construction machine corresponding to the track ID in the confirmed state;

[0127] Step 210: Determine the analysis results of the construction machinery in the construction area to be counted based on the above-mentioned machinery identification.

[0128] Figure 2 The method embodiment shown can be executed by the server to perform statistical analysis on construction machinery in hydropower engineering scenarios based on video surveillance. Figure 2 The specific execution subject of the illustrated method embodiment is not limited.

[0129] In step 201, the server may receive the initial video stream image sent by the image acquisition terminal. The image acquisition terminal may set one or more acquisition devices in the construction area to be counted, and the acquisition devices may be used to acquire the initial video stream image of the construction area to be counted.

[0130] Examples of acquisition equipment may include: cameras and video cameras, etc. Among them, a ball camera is a monitoring device that integrates a camera and a pan / tilt head, and its appearance is spherical.

[0131] The features of the ball camera include:

[0132] All-round monitoring: The ball camera equipped with a pan / tilt system can rotate horizontally and vertically to achieve 360-degree monitoring without blind spots, ensuring coverage of a wide area.

[0133] Strong concealment: The spherical design makes the specific monitoring direction of the camera difficult to be directly identified, which increases the concealment of monitoring and protects the safety of the camera.

[0134] Strong adaptability: It usually has functions such as dustproof and waterproof, can work stably in a variety of indoor and outdoor environments, and adapt to various climatic conditions.

[0135] High-definition image quality: Supports high-definition and even 4K resolution, ensuring clear and detailed monitoring images under any lighting conditions. 4K resolution means that the pixel value per horizontal line reaches or approaches 4096.

[0136] The construction area to be counted can be an area in a hydropower project scene. One or more collection devices can be set up in a construction area to be counted.

[0137] It should be noted that when multiple collection devices are set up in a construction area to be counted, the collection space ranges corresponding to the multiple collection devices can be independent of each other and have no intersection, so that the accuracy of the statistical analysis of construction machinery in the construction area to be counted can be improved. Of course, when multiple collection devices are set up in a construction area to be counted, the collection space ranges corresponding to the multiple collection devices can have intersections. In this case, special processing technology can be used to handle the intersection problem of the collection space ranges between multiple collection devices.

[0138] The following mainly takes setting up one collection device in a construction area to be counted as an example to illustrate the method of the embodiment of the present application. The case where multiple collection devices are set up in a construction area to be counted can be referenced to each other.

[0139] The collection device may have multiple working modes, such as a stationary mode and a mobile mode. The embodiment of the present application may determine the target working mode of the collection device according to the collection location where the collection device is located.

[0140] Step 201 performs text recognition on the initial video stream image of the construction area to be counted, which can be used to obtain the text information contained in the initial video stream image. The initial video stream image of step 201 and the video stream image of step 203 are both subordinate concepts of video stream images, which are used to distinguish video stream images with different acquisition times. The acquisition time of the initial video stream image is earlier than the acquisition time of the video stream image of step 203. It can be understood that the embodiments of the present application do not limit the specific initial video stream image and the video stream image of step 203.

[0141] Reference Figure 4 , shows a flowchart of the steps of a method for performing text recognition on an initial video stream image of a construction area to be counted according to an embodiment of the present application, the method specifically comprising:

[0142] Step 401: extracting an image of a region of interest from an initial video stream image;

[0143] Step 402: convert the region of interest image to HSV (Hue Saturation Value) color space;

[0144] Step 403: Determine the target position corresponding to white or approximately white according to the HSV value of the image of the region of interest in the HSV space;

[0145] Step 404: performing a color channel inversion operation on the pixel at the target position;

[0146] Step 405, performing a zoom operation on the region of interest image after the color channel inversion operation;

[0147] Step 406: Perform optical character recognition on the image of the region of interest after the zoom operation to obtain a text recognition result.

[0148] The image region corresponding to the region of interest image may be an image region where text may appear.

[0149] Reference Figure 3 , shows a schematic diagram of a video stream image of a pumped storage power station according to an embodiment of the present application, wherein the upper left corner area 301 of the video stream frame image contains acquisition time information, and the lower right corner area 302 of the video stream frame image contains acquisition location information. Therefore, the embodiment of the present application can respectively acquire the corresponding region of interest images in the upper left corner area and the lower right corner area.

[0150] In a specific implementation, a rectangular boundary may be used to describe the image of the region of interest, and the coordinates, height, and width of the center point may be defined, and the image of the region of interest may be extracted from the initial video stream image based on image cropping.

[0151] Since the HSV space separates the three attributes of color (hue, saturation, and brightness), it is more intuitive and convenient when processing specific colors or color ranges. Specifically in the embodiment of the present application, the HSV value of the region of interest image in the HSV space can be used to determine the target position corresponding to white or approximately white.

[0152] In the HSV color space, hue indicates the type of color. For white, the hue value usually does not have obvious characteristics because white does not have a specific position on the hue circle. Saturation indicates the purity of the color. The saturation of white and other neutral colors (such as gray and black) is usually low. Lightness indicates the brightness of the color, and white has a higher lightness value. Therefore, the embodiment of the present application can set a specific threshold to identify the target position of white or approximately white. For example, a lower saturation threshold and a higher lightness threshold can be set to filter out the target position.

[0153] By performing a color channel inversion operation on the pixel point at the target position, white characters or nearly white characters can be converted into black characters to improve the contrast between the text and the image background and improve the accuracy of text recognition.

[0154] The process of performing the color channel inversion operation on the pixel point at the target position may be: new_img=255-img, where img represents the original pixel value at the target position, and new_img represents the new pixel value after the color channel inversion operation.

[0155] The image of the region of interest after the color channel negation operation is enlarged, which can improve the recognizability of the text. In practical applications, a bilinear interpolation technique can be used to enlarge the image of the region of interest after the color channel negation operation. Of course, the embodiment of the present application does not limit the specific enlargement operation.

[0156] Those skilled in the art may adopt a desired optical character recognition method to perform optical character recognition on the image of the region of interest after the zoom operation, and the embodiments of the present application do not limit the specific optical character recognition method.

[0157] In one implementation, an optical character recognition model may be constructed based on a CRNN (Convolutional Recurrent Neural Network), and the optical character recognition model is used to recognize text sequences of indefinite length in an image of a region of interest.

[0158] The optical character recognition model may specifically include: a convolutional neural network, a deep bidirectional LSTM (Long Short-Term Memory) network and a CTC (Connectionist Temporal Classification) module. The process of performing optical character recognition on the image of the region of interest after the zoom operation specifically includes: first, using a convolutional neural network to extract the feature vector sequence of the image of the region of interest; then determining the text sequence features corresponding to the feature vector sequence based on the deep bidirectional LSTM network; and then converting the text sequence features into a text sequence based on the CTC module.

[0159] The embodiment of the present application can first determine the collection scene category based on the collection location information of the initial video stream frame image; then, determine the working mode of the collection device based on the collection scene category.

[0160] For example, in the scenario of a pumped storage power station, the scenes collected by the cameras installed on the shoulders of the main / auxiliary dams of the upper / lower reservoirs can be defined as preset large-scale scenes. For another example, the scenes collected by the cameras installed in the material yard, sand-mixing system, slag yard, and switch station of the upper / lower reservoirs can be defined as preset medium-scale scenes. For another example, the scenes collected by the cameras installed in the main transformer tunnel, underground powerhouse, inclined shaft, construction branch tunnel entrance of the underground cavern group, as well as the diversion tunnel gate well platform, vertical shaft spillway platform, and the intersection of the dam road can be defined as preset small-scale scenes.

[0161] In a specific implementation, the embodiment of the present application can pre-save the correspondence between the acquisition scene category and the keyword, so that the acquisition location information of the initial video stream frame image can be matched with the keyword in the correspondence to obtain the acquisition scene category corresponding to the acquisition location information.

[0162] In the embodiment of the present application, when the collection location information corresponds to a preset large-scale scene, the target working mode is set to mobile mode. Since the mobile mode can increase the collection space range of the collection equipment, the mobile mode of the embodiment of the present application can achieve full coverage of the construction area to be counted, reduce the omission of the counting of the construction machinery entering the site, and improve the counting effect of the construction machinery with small target size.

[0163] In the embodiment of the present application, the process of determining the above movement mode specifically includes:

[0164] Step B1: Determine the acquisition space range corresponding to the acquisition device in the four directions of up, down, left, and right:

[0165] Step B2: Determine the angle and height of the acquisition device according to the acquisition space range:

[0166] Step B3, determining the magnification of the acquisition device according to the acquisition space range and the angle and height of the acquisition device;

[0167] Step B4: determining the moving path of the acquisition device according to the acquisition space range, the angle and height of the acquisition device, and the magnification of the acquisition device.

[0168] In step B1, the acquisition space range specifically includes:

[0169] Upward moving direction: Determine the farthest point of the construction scene, which is usually the highest point or farthest point of the construction scene.

[0170] Downward moving direction: Determine the nearest point of the construction scene, which is usually the lowest or closest point of the construction scene.

[0171] Left moving direction: Determine the farthest point of the construction scene, which is usually the far left end of the construction scene.

[0172] Right moving direction: Determine the farthest point of the construction scene, which is usually the far right end of the construction scene.

[0173] In step B2, the angle and height of the camera can be adjusted according to the above-mentioned acquisition space range so that the camera can cover the entire construction scene. The angle of the camera should be able to move smoothly from the farthest to the closest to avoid missing areas in the acquisition space range.

[0174] In step B3, the magnification of the camera is determined based on the ability to clearly see the construction machinery in the construction scene to the farthest upward, left, and right. The magnification should be large enough to clearly identify and monitor the construction machinery, but at the same time ensure that the field of view is wide enough to cover the entire construction scene.

[0175] In step B4, after the angle, height and magnification of the camera are determined, the moving path of the acquisition device can be determined.

[0176] In one example, assuming that the acquisition device moves within a rectangle, the moving path can be a vertex of the rectangle as the starting point. Assuming that the starting point is the upper left vertex of the rectangle, the acquisition device can move from the starting point in the first direction (from left to right), and after moving to the upper right vertex, the acquisition device can move in the second direction (from top to bottom) by a preset step length, and then move in the third direction (from right to left) to the left side of the rectangle, and then move in the second direction (from top to bottom) by a preset step length, and then move in the first direction (from left to right) to the right side of the rectangle, and so on, until the acquisition device moves to the lower right vertex (end point) of the rectangle. Then, the acquisition device can be controlled to move from the lower right vertex of the rectangle to the upper left vertex according to the opposite path from the starting point to the end point.

[0177] It is understood that the upper left vertex of the rectangle is used as the starting point for the above example only. In fact, the starting point can be any vertex of the rectangle or the center of the rectangle. In addition, the acquisition device moves within a rectangle only as an example. In fact, the acquisition device can also move within a circle.

[0178] Regardless of whether the target working mode is a stationary mode or a mobile mode, the server can send a control instruction to the acquisition device, and the control instruction is used to make the acquisition device enter the target working mode.

[0179] In step 204, the target detection model may be used to perform target detection on the i-th frame of the video stream image. The first frame of the video stream image may be the first frame of the video stream image after the acquisition device enters the target working mode.

[0180] The object detection model is a deep learning model in the field of computer vision. It can identify objects in video stream images and determine the location and category of the object. The object detection model is usually implemented based on convolutional neural networks.

[0181] The embodiments of the present application do not limit the specific target detection model. For example, examples of target detection models may include: YOLO (You Only Look Once) series, SSD (Single Shot MultiBox Detector), etc.

[0182] The embodiment of the present application can use the construction machinery target recognition data set in advance to train the target detection model, so that the trained target detection model has the target detection capability of the video stream image in the construction scene.

[0183] The acquisition process of the construction machinery target recognition dataset specifically includes: collecting the working images of construction machinery, and using annotation tools to annotate the categories and target bounding boxes of the construction machinery in the working images, saving the annotation files, and establishing the construction machinery target recognition dataset. Among them, the image acquisition method of the working images is not limited to video surveillance systems, drones, mobile phones, cameras, etc.; the image acquisition covers various types of construction machinery involved in the construction of hydropower and civil engineering projects, different postures of construction machinery, different lighting conditions, different weather conditions, etc. The number of working images in the construction machinery target recognition dataset can exceed a certain number, such as 10,000, and ensure that the types and sizes of construction machinery are evenly distributed.

[0184] The target detection result obtained by the target detection model specifically includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result may also include: a set of bounding boxes of the construction machine in the i-th frame of the video stream image; the bounding box set specifically includes: at least one bounding box. The bounding box may be a rectangular box. The target detection result may also include: the category of the construction machine corresponding to the bounding box. The category of the construction machine may include: a dump truck or an excavator, etc.

[0185] In step 205, a trajectory ID is created in a trajectory set for the bounding box in the first frame of the video stream image, which can provide historical trajectory information for trajectory prediction of video stream images after the first frame of the video stream image.

[0186] In step 206, feature extraction is performed on the area image within the bounding box in the i-th frame of the video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image.

[0187] In the process of determining the characteristic features of the image in the region within the boundary box, the embodiment of the present application controls the weight of the texture feature to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, while the color feature has a low degree of distinction when the model and appearance color are similar, the embodiment of the present application can improve the degree of distinction between the two construction machinery based on the texture feature when the colors of the two construction machinery are similar.

[0188] For example, for two excavators of the same model, the colors and appearances of the two are similar, but the two may be different in terms of wear and scratches, cleanliness, corrosion, warning signs and safety signs, etc. The texture features of the embodiments of the present application can precisely reflect the differences between the two excavators of the same model in terms of wear and scratches, cleanliness, corrosion, warning signs and safety signs, etc.

[0189] Reference Figure 5 , shows a flowchart of the steps of a method for extracting features of a region image within a bounding box in an i-th frame of a video stream image according to an embodiment of the present application, the method specifically comprising:

[0190] Step 501: extract the first feature vector of the regional image using a convolution module;

[0191] Step 502: extracting a second eigenvector of the regional image using a local binary pattern method;

[0192] Step 503: Fuse the first feature vector and the second feature vector to obtain an appearance feature vector.

[0193] In step 501, the convolution module can be constructed based on models such as CNN (Convolutional Neural Network) and VGG (Visual Geometry Group).

[0194] The first feature vector extracted by the convolution module is high-dimensional, usually containing hundreds, thousands or even more dimensions. Examples of the dimensions of the first feature vector may include: spatial features, hierarchical features, scale features, directional features, color features, depth features, context features, etc. Among them, the spatial features represent the spatial structural information of the construction machinery, such as edges, corners, textures, etc. Therefore, the spatial features may include: texture features.

[0195] LBP (Local Binary Patterns) is a simple and effective algorithm for describing image texture features. The LBP algorithm generates a binary code by comparing the grayscale values ​​of each pixel in the image with its neighboring pixels, and then converts these binary codes into decimal numbers to form a texture descriptor. Since LBP can effectively describe the local texture features of the image and capture the subtle changes and patterns in the image; therefore, the second eigenvector can contain: texture features with strong distinguishing ability.

[0196] The process of fusing the first feature vector and the second feature vector in the embodiment of the present application specifically includes: weighting the features of the same dimension in the first feature vector and the second feature vector to obtain a weighted processing result; the features of the same dimension include: texture features; and increasing the corresponding numerical value for the weighted processing result corresponding to the texture feature.

[0197] Assuming that the weight of the first eigenvector is w1, the value of the texture feature in the first eigenvector is f1, the weight of the second eigenvector is w2, and the value of the texture feature in the second eigenvector is f2, the weighted processing result corresponding to the texture feature can be: w=w1*f1+w2*f2. In order to highlight the importance of the second eigenvector, w2 can be greater than w1. It can be understood that those skilled in the art can determine the specific values ​​of w1 and w2 according to actual application requirements.

[0198] The embodiment of the present application increases the corresponding value for the weighted processing result corresponding to the texture feature, which can achieve the technical effect that the weight of the texture feature is greater than the weight of the color feature. The increased value can be determined by those skilled in the art according to actual application requirements.

[0199] In step 207, the predicted trajectory of the trajectory ID in the trajectory set in the i-th frame of the video stream image is determined according to the historical trajectory information of the trajectory ID in the trajectory set.

[0200] The historical track information may include: the location information of the construction machine or the boundary box corresponding to the track ID in the past video stream images, and the location information generally includes coordinates, timestamp, speed, direction and other information.

[0201] The embodiment of the present application can use the Kalman filter method to determine the predicted trajectory of the trajectory ID in the trajectory set in the i-th frame of the video stream image. Kalman filtering is an efficient autoregressive filter that can estimate the state of a linear dynamic system from a series of incomplete and noisy measurements.

[0202] In step 208, the state of the track ID may include: a non-confirmed state or a confirmed state. The confirmed state may mean that the track ID has a corresponding target bounding box after matching calculation. The non-confirmed state may mean that the track ID is generated due to the initialization of the first frame image or the appearance of a new construction machinery target object, and does not have a corresponding target bounding box after matching calculation.

[0203] Step 208 is a process of determining the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID, which specifically includes:

[0204] Performing a first match between a trajectory ID in the trajectory set whose status is a confirmed status and a bounding box in the bounding box set to obtain a corresponding first matching result; wherein, in the first matching process, determining a cosine distance between a first appearance feature vector of the bounding box and a second appearance feature vector corresponding to the trajectory ID, and determining the first matching result according to the cosine distance; the first matching result includes: a trajectory ID for which the first matching succeeds, a trajectory ID for which the first matching fails, and a bounding box for which the first matching fails;

[0205] Performing a second match on the trajectory ID that failed the first match or the trajectory ID in the unconfirmed state with the bounding box that failed the first match, so as to obtain a corresponding second matching result; the second matching is an intersection-over-union matching; the second matching result includes: the trajectory ID that succeeded the second match, the trajectory ID that failed the second match, and the bounding box that failed the second match;

[0206] For the bounding box that fails the second matching, it is determined to be a new construction machinery target object, and a track ID with a non-confirmed state is created in the track set;

[0207] If the second matching result of the trajectory ID in the unconfirmed state is a successful match, the state of the trajectory ID is updated to the confirmed state; and if the number of second matching failures of the trajectory ID in the confirmed state exceeds a first threshold, the trajectory ID is deleted.

[0208] The embodiment of the present application can determine the status of the newly created track ID according to the actual application requirements. For example, when creating a track ID in a track set for the bounding box in the first frame of the video stream image, the status of the track ID corresponding to the bounding box in the first frame of the video stream image can be set to a non-confirmed state. For another example, for the detection box that failed the second match, it is determined to be a new construction machinery target object, and a track ID with a non-confirmed state is created.

[0209] It should be noted that the embodiment of the present application provides a switching method from the non-confirmed state to the confirmed state. Specifically, for the track ID in the non-confirmed state, the embodiment of the present application will verify the authenticity of the construction machinery corresponding to the track ID in the non-confirmed state based on the subsequent second matching result. The above verification means can identify and eliminate errors in the target detection results, thereby reducing errors in the construction machinery counting results caused by errors in the target detection results.

[0210] In the first matching process, the embodiment of the present application determines the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the trajectory ID, and determines the first matching result according to the cosine distance.

[0211] The second appearance feature vector is an appearance feature vector associated with the track ID, and represents the appearance features of the bounding box corresponding to the track ID in the historical video stream image.

[0212] In practical applications, the second appearance feature vector corresponding to the trajectory ID may be determined according to the first appearance feature vector of the bounding box corresponding to the trajectory ID in the historical video stream image.

[0213] For example, if the track ID1 is created based on the bounding box in the first frame of the video stream image, then the second appearance feature vector B2 of the track ID1 in the second frame of the video stream image can be determined based on the first appearance vector A1 of the bounding box A in the first frame of the video stream image. The second appearance feature vector B2 can be equal to the first appearance vector A1.

[0214] For example, if the track ID2 is created based on the bounding box B in the first frame of the video stream image, then the track ID2 successfully matches the bounding box C in the second frame of the video stream image, and the first appearance vector C2 of the bounding box C in the second frame of the video stream image. Then, the second appearance feature vector B3 of the track ID2 in the third frame of the video stream image can be jointly determined based on the first appearance vector B1 of the bounding box B in the first frame of the video stream image of the track ID2, and the first appearance vector C2 of the bounding box C in the second frame of the video stream image. Of course, the embodiment of the present application does not limit the specific process of jointly determining the second appearance feature vector B3 of the track ID2 in the third frame of the video stream image.

[0215] Since the texture features contained in the first appearance feature vector or the second appearance feature vector of the embodiment of the present application can better reflect the surface structure and details of the construction machinery, on this basis, the embodiment of the present application determines the first matching result according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the trajectory ID, which can effectively reduce the errors that occur in the first matching process between the trajectory ID and the bounding box by the trajectory tracking means. In other words, the first matching of the embodiment of the present application can correctly associate the bounding box of each construction machinery with its corresponding trajectory ID based on accurate appearance feature matching, which can effectively reduce matching errors caused by similar appearance, and thus can improve the accuracy of construction machinery counting analysis in hydropower engineering scenarios.

[0216] In an implementation of the present application, the process of first matching the track ID in the track set with the bounding box in the bounding box set specifically includes:

[0217] Step C1, determining a cosine distance between a first appearance feature vector of a bounding box and a second appearance feature vector corresponding to a track ID whose state is a confirmation state;

[0218] Step C2, determining the Mahalanobis distance between the bounding box and the track ID whose state is the confirmed state;

[0219] Step C3, constructing a cost matrix; the elements in the cost matrix represent the matching cost between a bounding box and a trajectory; the values ​​of the elements are determined according to the cosine distance and the Mahalanobis distance;

[0220] Step C4: solving the cost matrix with the goal of minimizing the total matching cost to obtain a first matching result.

[0221] In step C1, cosine distance is a commonly used similarity measurement method, which measures the similarity between two vectors by calculating the cosine value of the angle between them. The smaller the cosine distance, the more similar the two vectors are, indicating that the bounding box may correspond to the same construction machine as the trajectory ID. The cosine distance reflects the similarity between the bounding box and the trajectory ID in terms of appearance features.

[0222] In step C2, the Mahalanobis distance is a statistic used to measure the similarity between two random vectors. The Mahalanobis distance takes into account the covariance structure in the data, so it can better handle the correlation and scale differences in the data. Generally speaking, the smaller the Mahalanobis distance, the higher the similarity between the two vectors. The Mahalanobis distance characterizes the similarity between the bounding box and the trajectory ID in terms of positional features.

[0223] In the process of determining the Mahalanobis distance between the bounding box and the track ID in the confirmed state, a first position vector corresponding to the bounding box may be determined, a second position vector corresponding to the track ID may be determined, and the Mahalanobis distance between the first position vector and the second position vector may be calculated.

[0224] The second position vector may be determined based on the aforementioned position information. The position information corresponding to the trajectory ID may be determined based on a Kalman filter method.

[0225] In step C3, the cost matrix is ​​a two-dimensional matrix, each element of which represents the matching cost between a bounding box and a track ID. This matching cost is usually determined based on a combination of cosine distance and Mahalanobis distance. For example, the cosine distance and the Mahalanobis distance can be weighted summed, or other functions can be used to combine the cosine distance and the Mahalanobis distance to form the final matching cost.

[0226] It should be noted that before constructing the cost matrix, the cosine distance and the Mahalanobis distance may be filtered to filter out the trajectory IDs with larger cosine distance or Mahalanobis distance.

[0227] In step C4, an optimization method such as the Hungarian algorithm and the linear assignment problem solver can be used to find a matching solution that minimizes the total matching cost. The Hungarian algorithm is a classic combinatorial optimization algorithm that is suitable for solving the minimum weight matching problem of a bipartite graph. By solving the cost matrix, it can be determined which bounding boxes should be matched with which trajectory IDs.

[0228] In another implementation of the present application, the method may further include: after creating the trajectory ID, setting the lost age of the trajectory ID to 0; when the first matching result or the second matching result of the trajectory ID is a successful match, setting the lost age of the trajectory ID to 0; when the second matching result of the trajectory ID in the confirmation state is a failed match, increasing the lost age of the trajectory ID by 1;

[0229] The process of performing a first match between the trajectory ID in the trajectory set whose status is the confirmed state and the bounding box in the bounding box set in the embodiment of the present application specifically includes: dividing the trajectories in the trajectory set into multiple levels according to the lost age; starting from the level corresponding to the lost age 0, performing a first match between the trajectory ID in the trajectory set whose status is the confirmed state and the bounding box in the bounding box set layer by layer.

[0230] The present application embodiment introduces the concept of lost age, which can process the trajectory ID in layers and match them layer by layer according to the lost age. The advantages of this processing mainly include the following points:

[0231] Priority matching: By matching from the level with a loss age of 0, the most recently detected trajectory IDs are given priority. This allows newly appeared construction machinery or construction machinery that has just been re-detected to be matched in a timely manner, thereby improving the continuity and accuracy of trajectory prediction.

[0232] Dynamically adjust the matching order: As the loss age increases, the matching priority of the trajectory ID gradually decreases. This strategy of dynamically adjusting the matching order helps the system prioritize construction machinery that is more likely to be re-detected when resources are limited.

[0233] Improve matching efficiency: Through layered processing, we can avoid global matching of all trajectory IDs and instead match trajectory IDs at specific levels. This can reduce unnecessary calculations and improve matching efficiency.

[0234] Enhanced robustness: During the tracking of construction machinery, the construction machinery may disappear or be blocked for a short time. By losing the age, it is possible to quickly identify and resume tracking when the construction machinery reappears, thereby enhancing the robustness of the system.

[0235] The first matching result of the embodiment of the present application specifically includes: a trajectory ID of a first successful match, a trajectory ID of a first failed match, and a bounding box of a first failed match.

[0236] For the first successfully matched trajectory ID, the Kalman filter method can be used to update the fusion of historical trajectory information. Specifically, according to the position information of the successfully matched bounding box, the position information corresponding to the first successfully matched trajectory ID is updated, and according to the first appearance feature vector of the successfully matched bounding box, the second appearance feature vector of the first successfully matched trajectory ID is updated.

[0237] In this embodiment of the present application, step 207 may be performed for the first failed matching trajectory ID and the first failed matching bounding box.

[0238] In the embodiment of the present application, the trajectory ID that failed the first match or the trajectory ID in the non-confirmed state can be matched with the bounding box that failed the first match to obtain the corresponding second matching result. The second match is a supplement to the first match, and can handle those bounding boxes and trajectory IDs that failed to match successfully in the appearance feature match. It is suitable for handling situations where the appearance feature match fails due to occlusion, temporary disappearance, etc. Therefore, the second match of the embodiment of the present application can further improve the accuracy of the construction machinery counting analysis in the hydropower engineering scene.

[0239] The second match can be an intersection-and-union match. The intersection-and-union match process specifically includes: calculating the area intersection-and-union ratio between the prediction box corresponding to the trajectory ID and the boundary box. If the area intersection-and-union ratio is greater than a set threshold, the match is considered successful, otherwise the match fails. The prediction box corresponding to the trajectory ID can also be a rectangular box, which can be determined by a Kalman filter.

[0240] The second matching result specifically includes: a trajectory ID of a second successful match, a trajectory ID of a second failed match, and a bounding box of a second failed match.

[0241] For the second successfully matched trajectory ID, the Kalman filter method can be used to update the fusion of historical trajectory information. Specifically, according to the position information of the successfully matched bounding box, the position information corresponding to the second successfully matched trajectory ID is updated, and according to the first appearance feature vector of the successfully matched bounding box, the second appearance feature vector of the second successfully matched trajectory ID is updated.

[0242] In the embodiment of the present application, step 208 may be performed for the bounding box that fails the second matching. In step 208, a track ID with a non-confirmed state is created in the track set for the bounding box that fails the second matching.

[0243] The reason for the existence of the bounding box where the second match fails may be: new construction machinery has appeared in the construction area to be counted, or it may be: there is an error in the target detection result, and an object that is not a construction machinery is mistakenly identified as a construction machinery. The embodiment of the present application creates a track ID with a non-confirmed state for the detection box where the secondary match fails, and verifies the authenticity of the construction machinery corresponding to the track ID in the non-confirmed state based on the subsequent second matching result. The above means can identify and eliminate errors in the target detection results, thereby reducing errors in the construction machinery counting results caused by errors in the target detection results.

[0244] When the second matching failure number of the trajectory ID in the confirmation state exceeds the first threshold, the trajectory ID is deleted. The first threshold can be determined by those skilled in the art according to actual application requirements, for example, the first threshold can be 60 or other values.

[0245] For the trajectory ID that failed the second match, it can be determined whether it is in a non-confirmed state. If so, the trajectory ID can be directly deleted; if not, it can be determined whether the loss age of the trajectory ID is greater than a second threshold value, which can be a value such as 60. If it is greater, the trajectory ID is directly deleted, and if it is less, the trajectory ID is retained. Therefore, the method of the embodiment of the present application can also include: deleting the trajectory ID that is in a non-confirmed state and whose second matching result is a second match failure in the trajectory set.

[0246] In step 209, a machine identifier is assigned to the construction machine corresponding to the track ID in the confirmed state. The machine identifier uniquely identifies the construction machine in the construction area to be counted, and can be a string such as 0001, 0002, etc.

[0247] In summary, the embodiment of the present application can continuously obtain and confirm the construction machinery corresponding to the trajectory ID of the state by continuously updating the i-th frame video stream image, using advanced image processing and target tracking technology. This continuous tracking and confirmation process enables the system to accurately assign and identify the machinery identification of each construction machinery in the construction area.

[0248] In step 210, required statistical analysis may be performed based on the above-mentioned machine identification to determine the analysis results of the construction machines in the construction area to be counted.

[0249] In one implementation, the target detection result further includes: the category of the construction machinery in the i-th frame of the video stream image; the analysis result includes: the number and category distribution of the construction machinery;

[0250] Then step 210 is a process of determining the analysis results of the construction machinery in the construction area to be counted according to the machinery identification, which specifically includes:

[0251] Determining the number of construction machines in the construction area to be counted according to the type of the machine identification;

[0252] According to the type of the machine identification and the category of the construction machine corresponding to the machine identification, the category distribution of the construction machines in the construction area to be counted is determined.

[0253] Since each machine ID corresponds to a unique construction machine, the number of construction machines can be accurately obtained by counting the types of machine IDs. For example, machine IDs are assigned in ascending order. Assuming that the machine ID of the first construction machine is 0001, the number ABCD corresponding to the current largest machine ID can represent the number of construction machines.

[0254] The category distribution of construction machinery in the embodiment of the present application specifically includes: the number of construction machinery corresponding to each category of construction machinery. Examples of the above category distribution may include: X excavators and Y dump trucks in the construction area to be counted, and X and Y may be positive integers.

[0255] By analyzing the category distribution, the embodiment of the present application can provide detailed information about the distribution of different categories of construction machinery in the construction area to be counted, which is very useful for construction management and resource allocation.

[0256] In another implementation, the above may also include: performing text recognition on the i-th frame of the video stream image, and the obtained text recognition result includes: acquisition time information; determining the appearance time information corresponding to the mechanical identifier based on the acquisition time information contained in the video stream image to which the mechanical identifier belongs; and saving the mapping relationship between the mechanical identifier and the appearance time information.

[0257] The embodiment of the present application saves the mapping relationship between the machine identification and the appearance time information, which is helpful for making detailed ledger records. The ledger records may include the machine identification of each construction machine, the time of first appearance, and the activity record at the construction site.

[0258] In summary, the construction machinery statistical analysis method based on video surveillance in the embodiment of the present application comprehensively utilizes multiple technical means such as image acquisition, target detection, appearance feature extraction of bounding boxes, trajectory tracking and status determination of trajectory ID to improve the accuracy of construction machinery counting analysis in hydropower engineering scenarios.

[0259] First, for image acquisition means, the embodiment of the present application performs text recognition on the initial video stream image of the construction area to obtain the acquisition location information, and then determines the target working mode of the acquisition device, which is divided into mobile mode (corresponding to the preset large-scale scene) and static mode (corresponding to the preset small and medium-scale scene). In this way, the appropriate acquisition mode can be selected according to different construction area scenes, improving the pertinence and effectiveness of video acquisition.

[0260] Secondly, in the hydropower engineering scenario, construction machinery of the same category is similar in appearance, shape or color, which can easily lead to errors in the first matching process between the trajectory ID and the bounding box by the trajectory tracking means. In the process of calculating the first appearance feature vector of the image of the area within the bounding box, the embodiment of the present application controls the weight of the texture feature to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, and the color feature has a lower degree of distinction when the model and appearance color are similar, on this basis, the embodiment of the present application determines the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the trajectory ID. This can more accurately distinguish different construction machinery, reduce the probability of misjudgment due to similar appearance, and improve the distinction of construction machinery.

[0261] On the basis of improving the distinguishability of construction machinery, the embodiment of the present application can more accurately assign the mechanical identification of construction machinery, so that it can effectively prevent different construction machinery from being erroneously assigned the same mechanical identification. Therefore, the embodiment of the present application can improve the accuracy of construction machinery counting and analysis in hydropower engineering scenarios.

[0262] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0263] Based on the above embodiment, this embodiment also provides a construction machinery statistical analysis device based on video monitoring, referring to Figure 6 The device may specifically include: a text recognition module 601, a working mode module 602, an image acquisition module 603, a target detection module 604, a trajectory ID creation module 605, an appearance feature extraction module 606, a trajectory prediction module 607, a state determination module 608, an identification allocation module 609 and an analysis module 610.

[0264] The text recognition module 601 is used to perform text recognition on the initial video stream image of the construction area to be counted, and the obtained text recognition result includes: collection location information;

[0265] The working mode module 602 is used to determine the target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode;

[0266] An image acquisition module 603 is used to acquire a video stream image acquired by an acquisition device in the target working mode;

[0267] The target detection module 604 is used to perform target detection on the i-th frame of the video stream image, and the target detection result obtained includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set includes: at least one bounding box; i is a positive integer;

[0268] A trajectory ID creation module 605 is used to create a trajectory ID in a trajectory set for a bounding box in the first frame of the video stream image;

[0269] The appearance feature extraction module 606 is used to extract features from the regional image within the bounding box in the i-th frame of the video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature;

[0270] The trajectory prediction module 607 is used to determine the predicted trajectory of the trajectory ID in the trajectory set in the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set;

[0271] A state determination module 608 is used to determine the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state;

[0272] The identification allocation module 609 is used to allocate a machine identification to the construction machine corresponding to the track ID of the confirmed state;

[0273] The analysis module 610 is used to determine the analysis results of the construction machinery in the construction area to be counted according to the machinery identification.

[0274] Optionally, the appearance feature extraction module includes:

[0275] A first extraction module, used for extracting a first feature vector of the regional image by using a convolution module;

[0276] A second extraction module is used to extract a second feature vector of the regional image using a local binary pattern device;

[0277] The fusion module is used to fuse the first feature vector and the second feature vector to obtain an appearance feature vector.

[0278] Optionally, the fusion module includes:

[0279] A weighting module is used to perform weighted processing on the same-dimensional features in the first feature vector and the second feature vector to obtain a weighted processing result; the same-dimensional features include: texture features;

[0280] The texture enhancement module is used to increase the corresponding value according to the weighted processing result corresponding to the texture feature.

[0281] Optionally, the state determination module includes:

[0282] A first matching module is used to perform a first match between a trajectory ID in a confirmed state in the trajectory set and a bounding box in the bounding box set to obtain a corresponding first matching result; wherein, in the first matching process, a cosine distance between a first appearance feature vector of the bounding box and a second appearance feature vector corresponding to the trajectory ID is determined, and a first matching result is determined according to the cosine distance; the first matching result includes: a trajectory ID for which the first matching succeeds, a trajectory ID for which the first matching fails, and a bounding box for which the first matching fails;

[0283] A second matching module, used for performing a second matching on the trajectory ID that failed the first matching or the trajectory ID in the unconfirmed state with the bounding box that failed the first matching, so as to obtain a corresponding second matching result; the second matching is an intersection-over-union matching; the second matching result includes: the trajectory ID that succeeded the second matching, the trajectory ID that failed the second matching, and the bounding box that failed the second matching;

[0284] A second track ID creation module, used for creating a track ID in a non-confirmed state in the track set for the bounding box where the second match fails;

[0285] The state updating module is used to update the state of the trajectory ID to the confirmed state when the second matching result of the trajectory ID in the unconfirmed state is a successful match; and to delete the trajectory ID when the number of second matching failures of the trajectory ID in the confirmed state exceeds a first threshold.

[0286] Optionally, the first matching module includes:

[0287] a cosine distance determination module, configured to determine a cosine distance between a first appearance feature vector of a bounding box and a second appearance feature vector corresponding to a track ID whose state is a confirmation state;

[0288] A Mahalanobis distance determination module, used to determine the Mahalanobis distance between the bounding box and the track ID whose state is the confirmed state;

[0289] A matrix construction module, used to construct a cost matrix; the elements in the cost matrix represent the matching cost between a bounding box and a trajectory; the values ​​of the elements are determined according to the cosine distance and the Mahalanobis distance;

[0290] The solution module is used to solve the cost matrix with the goal of minimizing the total matching cost to obtain a first matching result.

[0291] Optionally, the device further comprises:

[0292] The lost age setting module is used to set the lost age of the trajectory ID to 0 after creating the trajectory ID;

[0293] A first lost age updating module, configured to set the lost age of the trajectory ID to 0 when the first matching result or the second matching result of the trajectory ID is a successful match;

[0294] A second lost age updating module, configured to increase the lost age of the trajectory ID by 1 when the second matching result of the trajectory ID in the confirmed state is a matching failure;

[0295] The first matching module includes:

[0296] A hierarchical division module is used to divide the trajectories in the trajectory set into multiple levels according to the loss age;

[0297] The layer-by-layer matching module is used to perform a first match between the track IDs in the track set with the confirmed state and the bounding boxes in the bounding box set layer by layer, starting from the layer corresponding to the lost age 0.

[0298] Optionally, the trajectory information corresponding to the trajectory ID includes: position information and a second appearance feature vector;

[0299] The device also includes:

[0300] The trajectory update module is used to update the position information corresponding to the trajectory ID according to the position information of the successfully matched bounding box when the first matching result or the second matching result of the trajectory ID is a successful match, and to update the second appearance feature vector of the trajectory ID according to the first appearance feature vector of the successfully matched bounding box.

[0301] Optionally, the device further comprises:

[0302] The deletion module is used to delete the trajectory ID whose status is unconfirmed and whose second matching result is second matching failure in the trajectory set.

[0303] Optionally, the first text recognition module includes:

[0304] An extraction module, used for extracting an image of a region of interest from an initial video stream image;

[0305] A space conversion module, used for converting the region of interest image into an HSV hue, saturation and brightness color space;

[0306] A target position determination module is used to determine a target position corresponding to white or approximately white according to the HSV value of the image of the region of interest in the HSV space;

[0307] A color negation module is used to perform a color channel negation operation on the pixel point at the target position;

[0308] A magnification module, used for magnifying the image of the region of interest after the color channel inversion operation;

[0309] The optical character recognition module is used to perform optical character recognition on the image of the region of interest after the zoom operation to obtain a text recognition result.

[0310] Optionally, the process of determining the movement mode includes:

[0311] Determine the acquisition space range corresponding to the acquisition device in the four directions of up, down, left, and right:

[0312] According to the acquisition space range, determine the angle and height of the acquisition device:

[0313] Determining the magnification of the acquisition device according to the acquisition space range and the angle and height of the acquisition device;

[0314] The moving path of the acquisition device is determined according to the acquisition space range, the angle and height of the acquisition device, and the magnification of the acquisition device.

[0315] Optionally, the target detection result further includes: the category of the construction machinery in the i-th frame of the video stream image; the analysis result includes: the number and category distribution of the construction machinery;

[0316] The analysis module is used to include:

[0317] A first analysis module is used to determine the number of construction machines in the construction area to be counted according to the type of the machine identification;

[0318] The second analysis module is used to determine the category distribution of the construction machinery in the construction area to be counted according to the type of the machinery identification and the category of the construction machinery corresponding to the machinery identification.

[0319] Optionally, the device further comprises:

[0320] The second text recognition module is used to perform text recognition on the i-th frame of the video stream image, and the obtained text recognition result includes: acquisition time information;

[0321] An appearance time determination module, used to determine the appearance time information corresponding to the mechanical identifier according to the acquisition time information contained in the video stream image to which the mechanical identifier belongs;

[0322] The saving module is used to save the mapping relationship between the machine identification and the appearance time information.

[0323] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0324] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0325] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Figure 7 An exemplary apparatus 1100 that can be used to implement various embodiments described in this application is schematically shown.

[0326] For one embodiment, Figure 7 An exemplary apparatus 1100 is shown having one or more processors 1102, a control module (chip set) 1104 coupled to at least one of the (one or more) processors 1102, a memory 1106 coupled to the control module 1104, a non-volatile memory / storage device 1108 coupled to the control module 1104, one or more input / output devices 1110 coupled to the control module 1104, and a network interface 1112 coupled to the control module 1104.

[0327] The processor 1102 may include one or more single-core or multi-core processors, and the processor 1102 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1100 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0328] In some embodiments, the device 1100 may include one or more computer-readable media (e.g., memory 1106 or non-volatile memory / storage device 1108) having instructions 1114 and one or more processors 1102 configured to execute the instructions 1114 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0329] For one embodiment, the control module 1104 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1102 and / or any suitable device or component in communication with the control module 1104 .

[0330] The control module 1104 may include a memory controller module to provide an interface to the memory 1106. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0331] The memory 1106 may be used, for example, to load and store data and / or instructions 1114 for the device 1100. For one embodiment, the memory 1106 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1106 may include a double data rate type four synchronous dynamic random access memory.

[0332] For one embodiment, the control module 1104 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1108 and the input / output device(s) 1110 .

[0333] For example, non-volatile memory / storage 1108 may be used to store data and / or instructions 1114. Non-volatile memory / storage 1108 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0334] The non-volatile memory / storage device 1108 may include storage resources that are physically part of the device on which the apparatus 1100 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1108 may be accessed via (one or more) input / output devices 1110 over a network.

[0335] The input / output device(s) 1110 may provide an interface for the apparatus 1100 to communicate with any other appropriate device, and the input / output device 1110 may include a communication component, an audio component, a sensor component, etc. The network interface 1112 may provide an interface for the apparatus 1100 to communicate through one or more networks, and the apparatus 1100 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0336] For one embodiment, at least one of the processor(s) 1102 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be packaged together with the logic of one or more controllers of the control module 1104 to form a system-in-package. For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with the logic of one or more controllers of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with the logic of one or more controllers of the control module 1104 to form a system-on-chip.

[0337] In various embodiments, the device 1100 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1100 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1100 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0338] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0339] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0340] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0341] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0342] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0343] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0344] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0345] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0346] The above is a detailed introduction to a construction machinery statistical analysis method and device based on video surveillance, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A construction machinery statistical analysis method based on video monitoring, characterized in that: The method comprises: Perform text recognition on the initial video stream image of the construction area to be counted, and the obtained text recognition result includes: collection location information; Determine the target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode; Acquire the video stream image acquired by the acquisition device in the target working mode; Performing target detection on the i-th frame of the video stream image, the target detection result obtained includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set includes: at least one bounding box; i is a positive integer; For the bounding box in the first frame of the video stream image, create a track ID in the track collection; Extracting features of a region image within a bounding box in an i-th frame of a video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector includes: a texture feature and a color feature; a weight of the texture feature is greater than a weight of the color feature; Determine the predicted trajectory of the trajectory ID in the trajectory set at the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set; Determine the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state; Assign a machine ID to the construction machine corresponding to the trajectory ID of the confirmed status; According to the machine identification, the analysis results of the construction machines in the construction area to be counted are determined.

2. The method according to claim 1, characterized in that The feature extraction of the area image within the boundary box of the i-th frame of the video stream image includes: Using the convolution module, extract the first eigenvector of the regional image; Using the local binary pattern method, extract the second eigenvector of the regional image; The first feature vector is fused with the second feature vector to obtain an appearance feature vector.

3. The method according to claim 2, characterized in that The fusing the first feature vector and the second feature vector comprises: Performing weighted processing on the same-dimensional features in the first feature vector and the second feature vector to obtain a weighted processing result; the same-dimensional features include: texture features; For the weighted processing result corresponding to the texture feature, increase the corresponding value.

4. The method according to claim 1, characterized in that The text recognition of the initial video stream image of the construction area to be counted includes: Extracting a region of interest image from an initial video stream image; Convert the region of interest image to HSV hue, saturation and brightness color space; According to the HSV value of the image of the region of interest in the HSV space, determine the target position corresponding to white or approximately white; Perform a color channel inversion operation on the pixel at the target position; A zoom operation is performed on the image of the region of interest after the color channel inversion operation; Optical character recognition is performed on the image of the region of interest after the zoom operation to obtain a text recognition result.

5. The method according to claim 1, characterized in that The process of determining the movement mode includes: Determine the acquisition space range corresponding to the acquisition device in the four directions of up, down, left, and right: According to the acquisition space range, determine the angle and height of the acquisition device: Determining the magnification of the acquisition device according to the acquisition space range and the angle and height of the acquisition device; The moving path of the acquisition device is determined according to the acquisition space range, the angle and height of the acquisition device, and the magnification of the acquisition device.

6. The method according to claim 1, characterized in that The determining the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID includes: Performing a first match between a trajectory ID in the trajectory set whose status is a confirmed status and a bounding box in the bounding box set to obtain a corresponding first matching result; wherein, in the first matching process, determining a cosine distance between a first appearance feature vector of the bounding box and a second appearance feature vector corresponding to the trajectory ID, and determining the first matching result according to the cosine distance; the first matching result includes: a trajectory ID for which the first matching succeeds, a trajectory ID for which the first matching fails, and a bounding box for which the first matching fails; Performing a second match on the trajectory ID that failed the first match or the trajectory ID in the unconfirmed state with the bounding box that failed the first match, so as to obtain a corresponding second matching result; the second matching is an intersection-over-union matching; the second matching result includes: the trajectory ID that succeeded the second match, the trajectory ID that failed the second match, and the bounding box that failed the second match; For the bounding box that fails the second matching, it is determined to be a new construction machinery target object, and a track ID with a non-confirmed state is created in the track set; When the second matching result of the trajectory ID in the unconfirmed state is a successful match, the state of the trajectory ID is updated to the confirmed state; and when the number of second matching failures of the trajectory ID in the unconfirmed state exceeds the first threshold, the state of the trajectory ID is updated to the confirmed state and deleted.

7. The method according to claim 6, characterized in that The first matching of the track IDs in the track set whose status is confirmed with the bounding boxes in the bounding box set includes: Determine a cosine distance between a first appearance feature vector of the bounding box and a second appearance feature vector corresponding to the track ID whose state is a confirmed state; Determine the Mahalanobis distance between the bounding box and the track ID whose state is confirmed; Constructing a cost matrix; the elements in the cost matrix represent the matching cost between a bounding box and a trajectory; the values ​​of the elements are determined according to the cosine distance and the Mahalanobis distance; With the goal of minimizing the total matching cost, the cost matrix is ​​solved to obtain a first matching result.

8. The method according to claim 6, characterized in that The method further comprises: After creating the trajectory ID, set the lost age of the trajectory ID to 0; When the first matching result or the second matching result of the trajectory ID is a successful match, the lost age of the trajectory ID is set to 0; When the second matching result of the trajectory ID in the confirmation state is a matching failure, the lost age of the trajectory ID is increased by 1; The first matching of the track IDs in the track set whose status is confirmed with the bounding boxes in the bounding box set includes: According to the age of loss, the trajectories in the trajectory set are divided into multiple levels; Starting from the level corresponding to the lost age 0, the first matching is performed layer by layer between the track IDs in the track set whose status is confirmed and the bounding boxes in the bounding box set.

9. The method according to claim 1, characterized in that: The trajectory information corresponding to the trajectory ID includes: position information and the second appearance feature vector; The method further comprises: When the first matching result or the second matching result of the trajectory ID is a successful match, the position information corresponding to the trajectory ID is updated according to the position information of the successfully matched bounding box, and the second appearance feature vector of the trajectory ID is updated according to the first appearance feature vector of the successfully matched bounding box.

10. The method according to claim 1, characterized in that The method further comprises: The trajectory ID whose status is unconfirmed and whose second matching result is second matching failure is deleted from the trajectory set.

11. The method according to any one of claims 1 to 10, characterized in that: The target detection result also includes: the category of the construction machinery in the i-th frame of the video stream image; the analysis result includes: the number and category distribution of the construction machinery; The step of determining the analysis results of the construction machinery in the construction area to be counted according to the machinery identification includes: Determining the number of construction machines in the construction area to be counted according to the type of the machine identification; According to the type of the machine identification and the category of the construction machine corresponding to the machine identification, the category distribution of the construction machines in the construction area to be counted is determined.

12. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: Perform text recognition on the i-th frame of the video stream image, and the obtained text recognition result includes: acquisition time information; Determine the appearance time information corresponding to the mechanical identifier according to the acquisition time information contained in the video stream image to which the mechanical identifier belongs; Save the mapping relationship between the machine identification and the occurrence time information.

13. A construction machinery statistical analysis device based on video monitoring, characterized in that: The device comprises: The text recognition module is used to perform text recognition on the initial video stream image of the construction area to be counted. The obtained text recognition result includes: collection location information; A working mode module, used to determine a target working mode of the collection device according to the collection location information; if the collection location information corresponds to a preset large-scale scene, the target working mode is a mobile mode; if the collection location information corresponds to a preset small- to medium-scale scene, the target working mode is a stationary mode; An image acquisition module, used to acquire a video stream image acquired by an acquisition device in the target working mode; The target detection module is used to perform target detection on the i-th frame of the video stream image, and the target detection result obtained includes: whether there is a construction machine in the i-th frame of the video stream image; if there is a construction machine in the i-th frame of the video stream image, the target detection result also includes: a bounding box set of the construction machine in the i-th frame of the video stream image; the bounding box set includes: at least one bounding box; i is a positive integer; A track ID creation module is used to create a track ID in a track set for a bounding box in the first frame of the video stream image; An appearance feature extraction module is used to extract features from a region image within a bounding box in an i-th frame of a video stream image to obtain a first appearance feature vector of the bounding box in the i-th frame of the video stream image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature; A trajectory prediction module, used to determine the predicted trajectory of the trajectory ID in the trajectory set in the i-th frame of the video stream image according to the historical trajectory information of the trajectory ID in the trajectory set; A state determination module, used to determine the state of the track ID according to the cosine distance between the first appearance feature vector of the bounding box and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state; An identification allocation module, used to allocate a machine identification to the construction machine corresponding to the trajectory ID of the confirmed state; The analysis module is used to determine the analysis results of the construction machinery in the construction area to be counted according to the machinery identification.

14. The device according to claim 13, characterized in that The appearance feature extraction module comprises: A first extraction module, used for extracting a first feature vector of the regional image by using a convolution module; A second extraction module is used to extract a second feature vector of the regional image using a local binary pattern device; The fusion module is used to fuse the first feature vector and the second feature vector to obtain an appearance feature vector.

15. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 12.

16. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 12.

Images