Intelligent building material counting method and counting device

Abstract

The present invention provides an intelligent building material counting method and counting device, the counting method comprising: extracting video frames of a building material placement scene video, using a multi-layer perceptron to calculate the time domain and spatial domain key frame scores {s1, s2, ..., s t}; extract the feature units of the corresponding frames according to the K high value method to represent the spatiotemporal information between frames {f1,f2,…,f t}; After position encoding of feature units, input into the pre-trained network structure for attention calculation, and obtain the feature sequence with attention {o1,o2,…,o t}; for each unit’s characteristic sequence {o1,o2,…,o t} perform global average pooling operation to obtain the fusion feature sequence {F1, F2, ..., F t The concatenated fused feature sequences are regressed towards numbers using a regression head to obtain a predicted number. A classification head analyzes the concatenated fused feature sequences to determine the building material categories in the video. This method uses a neural network approach to reduce network computational complexity and improve detection speed. A global attention computation mechanism extracts spatiotemporal information from the image to achieve accurate number and category estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an intelligent building material counting method and counting equipment. Background Art

[0002] The preparation and delivery of construction materials are an important part of construction projects. During the procurement, receipt, storage and distribution of construction materials, the traditional method is to organize personnel to manually count the relevant parts, which is inefficient and human errors may occur in multiple links. Summary of the Invention

[0003] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides an intelligent building material counting method and counting equipment to solve the technical problems of the current manual counting method, which leads to high workload, cumbersome counting process, easy counting errors and low work efficiency.

[0004] To solve the above problems, the first object of the present invention is to provide an intelligent building material counting method for estimating the types and quantities of building materials. The counting method includes:

[0005] S 100 : Extract video frames from building material placement scene videos and use multi-layer perceptron to calculate temporal and spatial keyframe scores ;

[0006] S 200 : Extract the feature units of the corresponding frames according to the K high value method to represent the spatiotemporal information between frames ;

[0007] S 300 : After position encoding of feature units, input into the pre-trained network structure for attention calculation to obtain feature sequences with attention ;

[0008] S 400 :For each unit's characteristic sequence Perform global average pooling operation to obtain a fusion feature sequence ;

[0009] S 500 : Use the regression head to regress the spliced ​​fusion feature sequence to the number and get the predicted number N;

[0010] S 600 : Use the classification head to analyze the spliced ​​fusion feature sequence and obtain the building material category C present in the video.

[0011] Optionally, in step S 100 In the above, the temporal and spatial key frame scores are calculated using a multilayer perceptron. The specific operations are:

[0012] S 101 : Input the image data of each frame in the original video into the multi-layer perceptron that can learn independently for refinement,

[0013] S 102 : Perform global averaging on all refined frame data to obtain the global feature frame.

[0014] S 103 :After each frame is connected with the global feature frame, it passes through another multi-layer perceptron that can learn independently to obtain the corresponding score g. The calculation expression of this score is as follows:

[0015] .

[0016] Optionally, in step S 100 In , the selection of time domain key frames includes:

[0017] The time domain frame scores calculated by the multilayer perceptron Arrange the K highest value order for each frame, and the operation is recorded as:

[0018]

[0019] Where: Y t It is a one-hot matrix used to extract the data at the corresponding position in the feature unit.

[0020] Optionally, in step S 100 In , the selection of spatial keyframes includes:

[0021] Set K in the K high value operation to 1, use the anchor point-based partitioning method, and calculate the spatial domain score of each anchor area through the maximum pooling operation. ;

[0022] Extract the top-1 region and generate a one-hot matrix:

[0023]

[0024] in: is the extracted spatial score matrix, represents the airspace fraction, -1 represents the maximum value of the airspace score.

[0025] Optionally, in step S 300 In the above, after position encoding of the feature units, the specific operations of inputting them into the pre-trained network structure for attention calculation include:

[0026] The selection operation can be regarded as a linear programming problem, and the calculation of forward propagation and backward propagation is implemented using the maximum perturbation method.

[0027] Optionally, in step S 500 In

[15] , the regression head is a network structure that implements the regression task and is composed of a fully connected layer, an activation operation, and a dropout operation. The data is processed and flattened by other network layers before being input into the regression head.

[0028] Optionally, in step S 600 In [1], the classification head is a network structure that implements the classification task and consists of a fully connected layer, an activation operation, and a drop operation. The data is processed and flattened by other network layers before being input into the classification head. The classification head outputs the probability that the current video segment belongs to a given classification.

[0029] Optionally, the loss function of the pre-trained network structure includes two parts: cross entropy loss and number regression loss. The loss function expression is as follows:

[0030]

[0031] Where: M is the number of building material categories, N is the number of video segments, Indicates whether the building materials in this video are of type k. represents the prediction of the number of building materials in the i-th video, represents the actual number of building materials in the i-th video, Represents the parameters set for actual application.

[0032] The second object of the present invention is to provide an intelligent building material counting device, comprising: a processor, a display, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor executes the computer program instructions for the intelligent building material counting method described above.

[0033] A third object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the intelligent building material counting method as described above.

[0034] Compared with the prior art, the present invention has significant advantages and beneficial effects, which are specifically reflected in the following aspects:

[0035] The present invention estimates the category and quantity of building materials by analyzing the content of video images. This method adopts the neural network method in deep learning to solve the problem of using a computer to automatically calculate the number of building materials and automatically identify the category of building materials. It also uses a video information screening structure to reduce the amount of network structure calculation and improve the detection speed. Finally, the Transformer's superior global attention calculation mechanism is used to extract the spatiotemporal information in the image to achieve more accurate quantity and category estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of the intelligent counting method for building materials in an embodiment of the present invention;

[0037] Figure 2 This is a structural diagram of a building material intelligent counting device in an embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of a feature unit screening module in an embodiment of the present invention;

[0039] Figure 4 This is a basic unit structure diagram of the TransformerEncoder module in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0041] See also Figure 1-4 As shown, an embodiment of the present invention provides an intelligent building material counting method for estimating the number of building materials. The counting method includes:

[0042] S 100 : Extract video frames from building material placement scene videos and use multi-layer perceptron to calculate temporal and spatial keyframe scores .

[0043] The selection of time domain key frames includes:

[0044] The time domain frame scores calculated by the multilayer perceptron Arrange the K highest value order for each frame, and the operation is recorded as:

[0045]

[0046] Where: Y t It is a one-hot matrix used to extract the data at the corresponding position in the feature unit (tokens). Represents the maximum value of the first k spatial domain scores.

[0047] The selection of spatial keyframes includes:

[0048] Set K in the K high value operation to 1, use the anchor point-based partitioning method, and calculate the spatial domain score of each anchor area through the maximum pooling operation. ;

[0049] Extract the top-1 region and generate a one-hot matrix:

[0050]

[0051] in: is the extracted spatial score matrix, represents the airspace fraction, -1 represents the maximum value of the airspace score.

[0052] S 200 : Extract the feature units (tokens) of the corresponding frames according to the K high value method to represent the spatiotemporal information between frames Therefore, the model using the K high value method to select key frames performs better than the Sinkhorn algorithm in practice.

[0053] The following is a detailed description of the K high value method:

[0054] The operation of the K high value method is recorded as: ,in is the normalized key frame score, and D is the corresponding key frame number. To speed up the calculation, the matrix operation is:

[0055]

[0056]

[0057] in: Represents the first k data taken from the original data sequence d, Y is a one-hot matrix, Represents the transpose of a matrix.

[0058] At this time, the operation is not differentiable and needs to be smoothed. The maximum perturbation method is introduced, and the K high value method can be regarded as a linear programming:

[0059]

[0060]

[0061]

[0062] Then, in the forward propagation, the K high value method can be implemented by adding random perturbations sampled from a standard Gaussian distribution: .

[0063] The differentiable K-high value method can be obtained through the Jacobian matrix: Complete the backpropagation calculation.

[0064] S 300 : After position encoding of the feature units (tokens), both are input into a pre-trained Transformer Encoder for attention calculation to obtain a feature sequence with attention .

[0065] Therefore, adding a Transformer network to the structure can utilize global context information, so that the features of the time domain and space domain can be associated with the corresponding time domain or space domain nearby, reducing the counting error and improving the classification accuracy.

[0066] Among them, the following is a detailed description of the feature extractor Transformer network:

[0067] See also Figure 4 As shown, Figure 4 This is the basic unit structure diagram of the Transformer Encoder module in the embodiment of the present invention. The pre-trained Transformer Encoder is used as the feature extractor. The basic unit of the Transformer network has multi-head attention ( ) and Multilayer Perceptron ( ) two parts, remember The layer output is , the basic network structure can be expressed as:

[0068]

[0069]

[0070] Among them, LN represents the hierarchical normalization operation, and the MSA operation consists of multiple self-attention operations, namely .

[0071] The self-attention operation can be expressed as:

[0072]

[0073]

[0074] in: All matrices are self-learning parameter matrices.

[0075] The main reason for using this network structure is that Transformer with a global receptive field is more suitable for counting tasks than CNNs with a limited receptive field.

[0076] S 400 :For each unit's characteristic sequence Perform global average pooling operation to obtain a fusion feature sequence .

[0077] S 500 : Use a regression head to regress the concatenated fusion feature sequence to a number to obtain the predicted number N.

[0078] The regression head is described in detail as follows:

[0079] The output of the Transformer is used as the input of the regression module, and the estimated number is obtained through a multi-layer perceptron whose output vector sequence is gradually reduced. .

[0080] S 600 : Use a classification head to analyze the spliced ​​fusion feature sequence and obtain the building material category C present in the video.

[0081] The classification header is described in detail as follows:

[0082] The output of Transformer is used as the input of the classification module, and the feature information unit token is converted into a probability representation as the predicted classification result through Softmax .

[0083] Specifically, in the embodiment of the present invention, step S 100 The specific steps of score calculation are as follows:

[0084] S 101 : Input the image data of each frame of the original video into the multi-layer perceptron that can learn independently for refinement;

[0085] S 102 : Perform global averaging on all refined frame data to obtain the global feature frame;

[0086] S 103 : After each frame is connected with the global feature frame, it passes through another multi-layer perceptron that can learn independently to obtain the corresponding score g.

[0087] The calculation expression of the fraction g is:

[0088]

[0089] Adjust the value range of fraction g to (0,1) through the calculation expression of fraction g.

[0090] Specifically, in the embodiment of the present invention, step S 200In order to ensure that the selection operation of high values ​​of K can pass differentials, the selection operation can be regarded as a linear programming problem, and the calculation of forward propagation and backward propagation is implemented using the maximum perturbation method.

[0091] Specifically, in the embodiment of the present invention, step S 500 In [1], the regression head specifically refers to a network structure that implements the regression task and consists of a fully connected layer, an activation operation, and a dropout operation. Before entering the regression head, the data is processed and flattened by other network layers. The classification head outputs the probability that the current video segment belongs to a given classification.

[0092] Specifically, in the embodiment of the present invention, the multi-layer perceptron and the Transformer Encoder in the feature extraction module are networks trained using a gradient descent training method.

[0093] Specifically, in the embodiment of the present invention, the loss function during network training includes two parts: cross entropy loss and number regression loss. The calculation expression of the loss function is as follows:

[0094]

[0095] Where: M is the number of building material categories, N is the number of video segments, Indicates whether the building materials in this video are of type k. The value of is 0 or 1. represents the prediction of the number of building materials in the i-th video, Represents the actual number of building materials in the i-th video.

[0096] Therefore, by adopting the neural network method in deep learning, the problem of using computers to automatically calculate the number of building materials and automatically identify the types of building materials is solved. A video information screening structure is used to reduce the computational complexity of the network structure and improve the detection speed. Finally, the Transformer's better global attention calculation mechanism is used to extract the spatiotemporal information in the image to achieve more accurate number and category estimation.

[0097] The first perception that humans use to recognize the features of an object is transmitted back to the brain through the senses of the eyes. By using computers to deeply learn the process of human senses transmitting information back to the brain, it is more natural to use and experience than other counting methods. Visual existence is the number, and there is no need for specific behavior of counting objects.

[0098] In addition, the counting process of computer vision does not require the counting object to come into contact with the device or terminal, and the image capture process is faster, which reduces the interference of the counting process on the object.

[0099] Finally, by analyzing the large-scale changes in the number of objects over time and space, and utilizing big data analysis and mining techniques, we can understand the behavior or nature of specific objects. This process requires no human intervention, as cloud computing and big data systems handle the entire process. The development and maturity of this technology is driving the application of intelligent analysis technology across all industries, replacing manual labor with machines, and significantly improving and promoting the efficiency and safety of social work.

[0100] An embodiment of the present invention also provides an intelligent building material counting device, comprising: a processor, a display, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor executes the computer program instructions to implement the above-mentioned intelligent building material counting method.

[0101] The counting device provided in the embodiment of the present application can be used to execute the intelligent building material counting method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0102] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the above-mentioned intelligent building material counting method.

[0103] It should be noted that the computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0104] Optionally, a readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0105] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. An intelligent building material counting method, applied to the estimation of building material types and quantities, characterized by: The counting method comprises: S 100 : Extract video frames from building material placement scene videos and use multi-layer perceptron to calculate temporal and spatial keyframe scores ; The multi-layer perceptron is used to calculate the time domain and spatial domain key frame scores. The specific operations are: S 101 : Input the image data of each frame in the scene video into the multi-layer perceptron that can learn autonomously for refinement. S 102 : Perform global averaging on all refined frame data to obtain the global feature frame. S 103 : After each frame is connected with the global feature frame, it passes through another multi-layer perceptron that can learn independently to obtain the corresponding score g. The calculation expression of the score is as follows: S 200 : Extract the feature units of the corresponding frames according to the K high value method to represent the spatiotemporal information between frames ; S 300 : After position encoding of feature units, input into pre-trained Transformer Encoder for attention calculation to obtain feature sequence with attention ; S 400 :For each unit's characteristic sequence Perform global average pooling operation to obtain a fusion feature sequence ; S 500 : Use the regression head to regress the spliced ​​fusion feature sequence to the number and obtain the predicted number N; S 600 : Use the classification head to analyze the spliced ​​fusion feature sequence and obtain the building material category C present in the video.

2. The intelligent building material counting method according to claim 1, characterized in that: In step S 100 In , the selection of time domain key frames includes: The time domain frame scores calculated by the multilayer perceptron Arrange the K highest value order for each frame, and the operation is recorded as: Where: Y t It is a one-hot matrix used to extract the data at the corresponding position in the feature unit; Represents the score of each frame, Represents the maximum value of the first k spatial domain scores.

3. The intelligent building material counting method according to claim 1, characterized in that: In step S 100 In , the selection of spatial keyframes includes: Set K in the K high value operation to 1, use the anchor point-based partitioning method, and calculate the spatial domain score of each anchor area through the maximum pooling operation. ; Extract the top-1 region and generate a one-hot matrix: in: is the extracted spatial score matrix, represents the airspace fraction, -1 represents the maximum value of the airspace score.

4. The intelligent building material counting method according to claim 1, characterized in that: In step S 300 In the above, after position encoding of the feature units, the specific operations of inputting them into the pre-trained network structure for attention calculation include: The selection operation can be regarded as a linear programming problem, and the calculation of forward propagation and backward propagation is implemented using the maximum perturbation method.

5. The intelligent building material counting method according to claim 1, characterized in that: In step S 500 In [15], the regression head is a network structure that implements the regression task and is composed of a fully connected layer, an activation operation, and a dropout operation. The data is processed and flattened by other network layers before being input into the regression head.

6. The intelligent building material counting method according to claim 1, characterized in that: In step S 600 In [1], the classification head is a network structure that implements the classification task and consists of a fully connected layer, an activation operation, and a drop operation. The data is processed and flattened by other network layers before being input into the classification head. The classification head outputs the probability that the current video segment belongs to a given classification.

7. The intelligent building material counting method according to claim 1, characterized in that: The loss function of the pre-trained network structure includes two parts: cross entropy loss and number regression loss. The loss function expression is as follows: Where: M is the number of building material categories, N is the number of video segments, Indicates whether the building material in the video is of type k, represents the prediction of the number of building materials in the i-th video, represents the actual number of building materials in the i-th video, Represents the parameters set for actual application.

8. A video counting device for building materials, comprising: A processor, a display, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor is used to implement the intelligent building material counting method according to any one of claims 1 to 7 when executing the computer program instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the intelligent building material counting method according to any one of claims 1 to 7.

Images