Method and device for recognizing motion state of mechanical equipment, electronic device and medium

Abstract

The application discloses a mechanical equipment motion state recognition method and device, electronic equipment and medium. The mechanical equipment motion state recognition method comprises the following steps: inputting an acquired to-be-tested image into a target category recognition model for recognition to obtain the equipment category of the to-be-tested equipment; inputting the to-be-tested image into a target position recognition model for recognition to obtain target position data of the to-be-tested equipment; grouping the to-be-tested image to obtain a reference image database; acquiring a reference timestamp; taking at least two reference images adjacent to the reference timestamp as target images; acquiring image pixel values of the target images; and determining the motion result of the to-be-tested equipment according to the image pixel values, wherein the motion result indicates whether the to-be-tested equipment is in a motion state or a non-motion state. The mechanical equipment motion state recognition method can determine whether to-be-tested equipment of different equipment categories is in a motion state, thereby improving the accuracy of construction mechanical equipment supervision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a motion state recognition method and device of mechanical equipment, an electronic device and a medium. BACKGROUND

[0002] At present, construction mechanical equipment of a construction site will produce noise, dust, exhaust gas and other influences, causing inconvenience to residents around the construction site. In order to reduce the influence of the above mechanical equipment, the working state of mechanical equipment of different categories needs to be supervised. The working state of the mechanical equipment can be determined according to the motion state of the construction equipment.

[0003] In the related art, only the device category of the mechanical equipment can be determined, and the motion state of the mechanical equipment of different categories cannot be determined, thereby affecting the accuracy of the supervision of the construction mechanical equipment. Therefore, how to determine the motion state of the mechanical equipment of different categories in the construction site has become a technical problem to be solved. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the prior art. To this end, the present application provides a motion state recognition method of mechanical equipment, which can determine the motion state of mechanical equipment of different categories.

[0005] The present application also provides a motion state recognition device of mechanical equipment, an electronic device applying the motion state recognition method of the mechanical equipment, and a computer readable storage medium applying the motion state recognition method of the mechanical equipment.

[0006] The motion state recognition method of mechanical equipment according to the first aspect embodiment of the present application comprises:

[0007] obtaining a to-be-tested image of a to-be-tested device;

[0008] inputting the to-be-tested image into a preset target category recognition model for recognition to obtain a device category of the to-be-tested device;

[0009] inputting the to-be-tested image into a preset target position recognition model for recognition to obtain target position data of the to-be-tested device;

[0010] grouping the to-be-tested image according to the target category and the target position data to obtain a reference image database; wherein a to-be-tested image in the reference image database is taken as a reference image;

[0011] obtaining a reference timestamp of the reference image, and taking at least two reference images adjacent to the reference timestamp as target images;

[0012] Obtaining image pixel values of the target image, and determining a motion result of the to-be-tested device according to the image pixel values; wherein the motion result is used to indicate that the to-be-tested device is in a motion state or a non-motion state.

[0013] According to the motion state recognition method of the mechanical device, the to-be-tested image is input into the preset target category recognition model for recognition, and the device category of the to-be-tested device is obtained. The to-be-tested image is input into the preset target position recognition model for recognition, and the target position data of the to-be-tested device is obtained. The to-be-tested image is grouped according to the target category and the target position data, and the reference image database is obtained. The to-be-tested image in the reference image database is used as a reference image. The reference timestamp of the reference image is obtained, and at least two reference images adjacent to the reference timestamp are used as target images. Image pixel values of the target images are obtained, and a motion result of the to-be-tested device is determined according to the image pixel values. The motion result indicates that the to-be-tested device is in a motion state or a non-motion state. The motion state recognition method of the mechanical device can determine whether the to-be-tested device of different device categories is in a motion state by judging the position change of the to-be-tested device in the to-be-tested image, thereby improving the accuracy of the supervision of the construction mechanical device.

[0014] According to some embodiments of the present application, the obtaining of the image pixel values of the target image and the determination of the motion result of the to-be-tested device according to the image pixel values include:

[0015] Difference processing is performed according to the image pixel values to obtain pixel difference values;

[0016] The motion result is determined according to the pixel difference values and a preset reference value.

[0017] According to some embodiments of the present application, the target position recognition model includes an image segmentation layer and a classification layer.

[0018] The inputting of the to-be-tested image into the preset target position recognition model for recognition to obtain the target position data of the to-be-tested device includes:

[0019] An area segmentation operation is performed on the to-be-tested image according to the image segmentation layer to obtain a candidate area image;

[0020] An image recognition operation is performed on the candidate area image according to the classification layer to obtain a region recognition result; wherein the region recognition result is used to indicate that the candidate area image includes the to-be-tested device or does not include the to-be-tested device; and the candidate area image indicated by the region recognition result as including the to-be-tested device is used as a target area image.

[0021] acquire a target pixel position of the target region image in the to-be-tested image, and take the target pixel position as the target position data.

[0022] According to some embodiments of the present application, the grouping of the to-be-tested image according to the target category and the target position data to obtain a reference image database comprises:

[0023] grouping the target region image according to the target category and the target position data to obtain a preliminary image database; wherein the target region image in the preliminary image database is taken as a preliminary image;

[0024] performing object boundary recognition on the preliminary image to obtain boundary line data of the to-be-tested device;

[0025] performing a clipping operation on the preliminary image according to the boundary line data to obtain the reference image;

[0026] constructing the reference image database according to the reference image.

[0027] According to some embodiments of the present application, before the to-be-tested image is input into a preset target category recognition model for recognition to obtain the device category of the to-be-tested device, the method further comprises training the target category recognition model, specifically comprising:

[0028] acquiring a sample image of a sample device and a sample image label of the sample image; wherein the sample image label is used to represent the device category of the sample device;

[0029] inputting the sample image into a preset original category recognition model for recognition to obtain an original image label; the original image label is used to represent the device category of the sample device;

[0030] performing parameter adjustment on the original category recognition model according to the sample image label and the original image label to obtain the target category recognition model.

[0031] According to some embodiments of the present application, before the to-be-tested image of the to-be-tested device is acquired, the method further comprises:

[0032] acquiring to-be-tested video data of the to-be-tested device;

[0033] performing single-frame processing on the to-be-tested video data to obtain the to-be-tested image.

[0034] According to some embodiments of the present application, the method further comprises:

[0035] if the motion result indicates that the to-be-tested device is in a motion state, taking the target image as a marked image.

[0036] repeating the acquiring of the reference timestamps of the reference images, taking at least two reference images adjacent in the reference timestamps as target images until all the reference images are traversed;

[0037] acquiring a mark timestamp of all the mark images, and performing video clip cutting on the to-be-tested video data according to the mark timestamp to obtain a target video clip.

[0038] The motion state recognition device of the mechanical equipment according to the second aspect of the present application comprises:

[0039] an image acquisition module, configured to acquire a to-be-tested image of a to-be-tested device;

[0040] a category recognition module, configured to input the to-be-tested image into a preset target category recognition model for recognition to obtain a target category of the to-be-tested device;

[0041] a position recognition module, configured to input the to-be-tested image into a preset target position recognition model for recognition to obtain target position data of the to-be-tested device;

[0042] a grouping module, configured to group the to-be-tested image according to the target category and the target position data to obtain a reference image database; wherein, a to-be-tested image in the reference image database is taken as a reference image;

[0043] a target image determination module, configured to acquire a reference timestamp of a reference image, and take at least two reference images adjacent in the reference timestamp as target images;

[0044] a motion state recognition module, configured to acquire an image pixel value of the target image, and determine a motion result of the to-be-tested device according to the image pixel value; wherein, the motion result is used to represent that the to-be-tested device is in a motion state or a non-motion state.

[0045] The motion state recognition device of the mechanical equipment according to the present application has at least the following beneficial effects: the motion state recognition device of the mechanical equipment can determine whether a to-be-tested device of different device categories is in a motion state by adopting the motion state recognition method of the mechanical equipment, thereby improving the accuracy of the supervision of the construction mechanical equipment.

[0046] The electronic device according to the third aspect of the present application comprises:

[0047] at least one memory;

[0048] at least one processor;

[0049] at least one computer program;

[0050] The computer program is stored in the memory, and the processor executes the at least one computer program to implement the motion state recognition method of the mechanical equipment according to the first aspect embodiment.

[0051] The computer readable storage medium according to the fourth aspect embodiment of the present application stores computer executable instructions, and the computer executable instructions are used to make a computer execute the motion state recognition method of the mechanical equipment according to the first aspect embodiment.

[0052] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0053] The present application will be further described below in conjunction with the accompanying drawings and embodiments, wherein:

[0054] Figure 1 The flow chart of the motion state recognition method of the mechanical equipment provided in the embodiments of the present application;

[0055] Figure 2 The flow chart of the specific method of step S160 in the embodiment of the present application; Figure 1

[0056] The flow chart of the specific method of step S130 in the embodiment of the present application; Figure 3 Figure 1 The schematic diagram of the to-be-tested image in the method provided in the embodiments of the present application;

[0057] Figure 4 The flow chart of the specific method of step S140 in the embodiment of the present application;

[0058] Figure 5 Figure 1 The schematic diagram of the preliminary image in the method provided in the embodiments of the present application;

[0059] Figure 6 The flow chart of the specific method of training the target category recognition model in the embodiment of the present application;

[0060] Figure 7 The flow chart of the motion state recognition method of the mechanical equipment provided in the second specific embodiment of the present application; Figure 1

[0061] The flow chart of the motion state recognition method of the mechanical equipment provided in the second specific embodiment of the present application; Figure 8

[0062] Figure 9 ​​​The flow chart of the motion state recognition method of the mechanical equipment of the third specific embodiment provided in the present application is shown in FIG. 6.

[0063] Figure 10 The module block diagram of the motion state recognition device of the mechanical equipment provided in the embodiment of the present application is shown in FIG. 7.

[0064] Figure 11 The hardware structure schematic diagram of the electronic device provided in the embodiment of the present application is shown in FIG. 8.

[0065] Reference signs:

[0066] The image acquisition module 110, the category recognition module 120, the position recognition module 130, the grouping module 140, the target image determination module 150, the motion state recognition module 160, the processor 210, the memory 220, the input / output interface 230, the communication interface 240, and the bus 250. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0068] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flow chart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the description, claims and above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0070] First, the several terms involved in the present application are analyzed:

[0071] Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] Machine Learning: is a special study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. In machine learning, deep learning is to learn the internal rules and representation levels of sample data. The information obtained in these learning processes is very helpful for the interpretation of data such as text, images and sound. The ultimate goal is to enable machines to have analytical learning ability like humans, and to recognize text, images and sound data. Deep learning is a complex machine learning algorithm, and the effect achieved in speech and image recognition far exceeds that of previous related technologies.

[0073] At present, construction machinery and equipment on construction sites will produce noise, dust, exhaust gas and other influences, causing inconvenience to residents around the construction site. In order to reduce the influence of the above mechanical equipment, it is necessary to supervise the working state of different types of mechanical equipment. Among them, the working state of the mechanical equipment can be determined according to the motion state of the construction equipment.

[0074] In related technologies, only the device category of the mechanical equipment can be determined, and the motion state of the mechanical equipment of different categories cannot be determined, thereby affecting the accuracy of the supervision of the construction machinery and equipment. Therefore, how to determine the motion state of the mechanical equipment of different categories in the construction site has become a technical problem to be solved.

[0075] Based on this, the embodiments of the present disclosure provide a mechanical equipment motion state recognition method and device, electronic equipment and medium, which can determine the motion state of mechanical equipment of different categories.

[0076] As Figure 1As shown, the embodiment of the present application provides a motion state recognition method of a mechanical device, which comprises but is not limited to steps S110 to S160, and the six steps are described in detail below.

[0077] Step S110: obtaining a to-be-tested image of a to-be-tested device;

[0078] Step S120: inputting the to-be-tested image into a preset target category recognition model for recognition to obtain a device category of the to-be-tested device;

[0079] Step S130: inputting the to-be-tested image into a preset target position recognition model for recognition to obtain target position data of the to-be-tested device;

[0080] Step S140: grouping the to-be-tested image according to the target category and the target position data to obtain a reference image database; wherein the to-be-tested image in the reference image database is taken as a reference image;

[0081] Step S150: obtaining a reference timestamp of the reference image, and taking at least two reference images adjacent to the reference timestamp as target images;

[0082] Step S160: obtaining image pixel values of the target images, and determining a motion result of the to-be-tested device according to the image pixel values; wherein the motion result is used to represent a motion state or a non-motion state of the to-be-tested device.

[0083] In step S110 of some embodiments, the to-be-tested image of the to-be-tested device is obtained, and the number of to-be-tested images can be multiple. The to-be-tested device is a mechanical device to be subjected to motion state recognition. In the following embodiments, the mechanical device used for building construction is taken as an example to be described.

[0084] In step S120 of some embodiments, the obtained to-be-tested image is input into a preset target category recognition model, and the to-be-tested image is recognized by the target category recognition model to obtain the device category of the to-be-tested device. Wherein, the device category can include the categories of mechanical devices such as excavators, pile drivers, and spoil transport vehicles.

[0085] In step S130 of some embodiments, the obtained to-be-tested image is input into a preset target position recognition model, and the to-be-tested image is recognized by the target position recognition model to obtain the target position data of the to-be-tested device. Wherein, the target position data is used to represent the pixel coordinates of the position of the to-be-tested device in the to-be-tested image.

[0086] In step S140 of some embodiments, after obtaining the device category and the target position data of the to-be-tested device, all to-be-tested images are grouped. To-be-tested images with the same device category and the same target position data are grouped into the same group, so that a reference image database is constructed according to to-be-tested images in the same group. The to-be-tested images in the reference image database are reference images. For example, the device category of the first to-be-tested image and the second to-be-tested image is both excavator, and the target position data of the first to-be-tested image and the second to-be-tested image is both the first target position, so that the first to-be-tested image and the second to-be-tested image are grouped into a group to construct a reference image database. By constructing the reference image database, to-be-tested images with the same device category and the same target position of the to-be-tested device can be collected in the same database, so as to facilitate subsequent judgment of the motion state of the to-be-tested device with the same device category and target position.

[0087] In step S150 of some embodiments, reference timestamps of each reference image in the reference image database are obtained. At least two reference images with adjacent reference timestamps are selected from the reference image database, and the selected reference images are taken as target images. For example, the first reference timestamp of the first reference image is “00:00:01”, and the second reference timestamp of the second reference image is “00:00:02”, so that the first reference timestamp and the second reference timestamp are adjacent to each other, and at this time, the first reference image is taken as the first target image, and the second reference image is taken as the second target image. If the reference image database further includes a third reference image, and the third reference timestamp corresponding to the third reference image is “00:00:03”, then the third reference timestamp and the second reference timestamp are adjacent to each other, and as above, the third reference image is taken as the third target image.

[0088] In step S160 of some embodiments, image pixel values of any two target images with adjacent reference timestamps are obtained respectively. Since the reference timestamps of the target images are adjacent, when there is a certain degree of change between the two image pixel values, it indicates that the position of the to-be-tested device in the image changes, that is, the to-be-tested device is in a motion state. Therefore, the transformation amount of the pixel values is determined according to the two image pixel values, and the motion result of the to-be-tested device is obtained. The motion result is used to represent that the to-be-tested device is in a motion state or a non-motion state in the corresponding two reference timestamps. When the motion result indicates that the to-be-tested device is in a motion state, the to-be-tested device is in a working state. For example, the device category of the to-be-tested device is excavator, and when the motion result indicates that the to-be-tested device is in a motion state, the working state of the to-be-tested device corresponds to the excavating working state.

[0089] The motion state recognition method of the mechanical equipment provided in the embodiments of the present application inputs the obtained to-be-tested image into a preset target category recognition model for recognition to obtain the equipment category of the to-be-tested equipment. The to-be-tested image is input into a preset target position recognition model for recognition to obtain target position data of the to-be-tested equipment. The to-be-tested image is grouped according to the target category and the target position data to obtain a reference image database, and the to-be-tested image in the reference image database is taken as a reference image. A reference timestamp of the reference image is obtained, and at least two reference images adjacent in time to the reference timestamp are taken as target images. Image pixel values of the target images are obtained, and a motion result of the to-be-tested equipment is determined according to the image pixel values, which indicates whether the to-be-tested equipment is in a motion state or a non-motion state. The motion state recognition method of the mechanical equipment in the embodiments can determine the position change of the to-be-tested equipment in different target images, thereby determining whether the to-be-tested equipment of different equipment categories is in a motion state, and the accuracy of the supervision of the construction mechanical equipment is improved.

[0090] As shown in Figure 2 In some embodiments of the present application, step S160 includes but is not limited to step S210 and step S220, which are described in detail as follows.

[0091] Step S210: performing differential processing according to the image pixel values to obtain pixel differential values;

[0092] Step S220: determining a motion result according to the pixel differential values and a preset reference value.

[0093] In step S210 of some embodiments, image differential processing is performed on any two target images adjacent in time to the reference timestamp to obtain a differential image, and the pixel values of the differential image are the pixel differential values. The image differential processing refers to subtracting the pixels of two images adjacent in time to obtain a differential image for representing the change between the two images.

[0094] In step S220 of some embodiments, the obtained pixel differential values are compared with a preset reference value. When the pixel differential values are less than the reference value, it indicates that the to-be-tested equipment does not move, and the motion result indicates that the to-be-tested equipment is in a non-motion state. When the pixel differential values are greater than the reference value, it indicates that the to-be-tested equipment moves, and the motion result indicates that the to-be-tested equipment is in a motion state. The specific judgment formula can refer to the following formula (1):

[0095]

[0096] Wherein, |Sc(X,t)-Sc(X,t-Δt)| is the pixel differential value, λ is the suppression coefficient, N ATotal pixel number of the target image. As can be seen from formula (1), when the pixel difference value is greater than the preset maximum value (i.e., the above-mentioned preset reference value), it indicates that the pixel value in the target image changes obviously, and at this time, it can be judged that the to-be-tested device is in a motion state.

[0097] As shown in the drawings, Figure 3 In some embodiments of the present application, the target position recognition model includes an image segmentation layer and a classification layer. Step S130 includes, but is not limited to, steps S310 to S330, which are described in detail below.

[0098] Step S310: performing a region segmentation operation on the to-be-tested image according to the image segmentation layer to obtain a candidate region image;

[0099] Step S320: performing image recognition on the candidate region image according to the classification layer to obtain a region recognition result; wherein the region recognition result is used to indicate whether the candidate region image includes the to-be-tested device or not; and the candidate region image indicating that it includes the to-be-tested device is taken as a target region image;

[0100] Step S330: obtaining a target pixel position of the target region image in the to-be-tested image, and taking the target pixel position as target position data.

[0101] In step S310 of some embodiments, the image segmentation layer of the target position recognition model is used to perform region segmentation on the to-be-tested image to obtain a plurality of candidate region images. The region segmentation operation can divide the to-be-tested image into a plurality of candidate region images with equal areas, and the area of each candidate region image can be set according to requirements. For example, referring to Figure 4 , it is assumed that the resolution of the to-be-tested image is 1920x1080, and the to-be-tested image is evenly divided into 12 candidate region images with a resolution of 480x270.

[0102] In step S320 of some embodiments, the classification layer of the target position recognition model is used to perform image recognition on the candidate region image to identify whether the to-be-tested device is included in the candidate region image, i.e., to obtain the region recognition result. Secondly, the candidate region image with the region recognition result indicating that it includes the to-be-tested device is taken as the target region image.

[0103] In step S330 of some embodiments, the target pixel position of the target region image in the to-be-tested image is obtained, and the target pixel position is taken as the target position data. For example, referring to Figure 4The to-be-tested device A is located in one of the target region images S, and pixel coordinates of the target region image S in the to-be-tested image are obtained, and the pixel coordinates are the target position data. The pixel coordinates of the to-be-tested image can take a first side of the to-be-tested image as an X axis of a coordinate system, take a second side adjacent to the first side as a Y axis of the coordinate system, take a vertex where the first side and the second side intersect as an origin of the coordinate system, and the pixel coordinates are coordinates of each pixel of the to-be-tested image in the coordinate system.

[0104] As shown in FIG. 13, in some embodiments of the present application, step S140 includes but is not limited to steps S510 to S540, which are described in detail as follows. Figure 5

[0105] Step S510: Grouping the target region images according to the target category and the target position data to obtain a preliminary image database; wherein the target region images in the preliminary image database are preliminary images.

[0106] Step S520: Performing object boundary recognition on the preliminary images to obtain boundary line data of the to-be-tested device.

[0107] Step S530: Performing a cutting operation on the preliminary images according to the boundary line data to obtain reference images.

[0108] Step S540: Constructing a reference image database according to the reference images.

[0109] In step S510 of some embodiments, all target region images are grouped, the target region images with the same device category and the same target position data are taken as preliminary images, and the preliminary images are divided into the same group, so that the preliminary image database is constructed according to the preliminary images in the same group.

[0110] In step S520 of some embodiments, the to-be-tested device in the preliminary image is recognized to obtain the boundary line data of the to-be-tested device. For example, referring to Figure 6 The to-be-tested device A in the preliminary image is recognized by the SoftNMS algorithm to obtain the boundary line L data in the preliminary image. The boundary line data is used to represent the image contour of the to-be-tested device in the preliminary image. It can be understood that the boundary line data obtained by optimizing and adjusting the parameters of the SoftNMS algorithm can be closer to the image contour of the to-be-tested device in the preliminary image.

[0111] ​In step S530 of some embodiments, the obtained boundary line data is used to crop the preliminary image, to retain an image containing the to-be-tested device, and the image is taken as a reference image. The cropped reference image is as close as possible to the image contour of the to-be-tested device, and thus in subsequent step S160, the recognition degree of the to-be-tested device in the image can be improved, and thus the recognition accuracy of the motion state of the to-be-tested device can be improved.

[0112] In step S540 of some embodiments, a reference image database is constructed by using the obtained reference image containing the to-be-tested device.

[0113] As shown in FIG. 7, Figure 7 In some embodiments of the present application, before step S120, the motion state recognition method of the mechanical device further includes training a target category recognition model, specifically including but not limited to steps S710 to S730, which are described in detail below.

[0114] Step S710: obtaining a sample image of a sample device and a sample image label of the sample image; wherein the sample image label is used to represent the device category of the sample device;

[0115] Step S720: inputting the sample image into a preset original category recognition model for recognition to obtain an original image label; the original image label is used to represent the device category of the sample device;

[0116] Step S730: adjusting parameters of the original category recognition model according to the sample image label and the original image label to obtain a target category recognition model.

[0117] In step S710 of some embodiments, a sample image of a sample device and a sample image label of the sample image are obtained. The sample device is a mechanical device, and in the following embodiments, the sample device is taken as a mechanical device used for building construction. The sample image label represents the device category of the sample device, and the device category can include categories of mechanical devices such as excavators, pile drivers, and debris transport vehicles.

[0118] In step S720 of some embodiments, the sample image is input into the original category recognition model, and the original category recognition model is used to recognize the sample image to obtain an original image label. The original image label represents the device category of the sample device recognized by the original category recognition model. The original category recognition model uses a Cascade R-CNN algorithm to recognize the sample image, and the specific recognition process is as follows: a dimension reduction layer of the original category recognition model performs dimension reduction processing on the sample image, and a shared convolution layer of the original category recognition model extracts the original image label of the sample image from the dimension-reduced sample image.

[0119] In step S730 of some embodiments, the original image label and the sample image label are compared to obtain a label comparison result. If the label comparison result indicates that the original image label is inconsistent with the sample image label, the original boundary point detection model is adjusted in parameters to obtain a target category recognition model. The target category recognition model obtained by training in this embodiment can more accurately predict the device category of the to-be-tested device.

[0120] As shown in Figure 8 In some embodiments of the present application, before step S110, the mechanical equipment motion state recognition method further includes but is not limited to steps S810 and S820, which will be described in detail below.

[0121] Step S810: obtaining to-be-tested video data of the to-be-tested device;

[0122] Step S820: performing single-frame processing on the to-be-tested video data to obtain to-be-tested images.

[0123] In step S810 of some embodiments, the to-be-tested video data is obtained by a camera such as a high-definition camera arranged at a construction site. The to-be-tested video data can be collected in the form of a video stream file, and the frequency of video stream acquisition is 1 minute, i.e., a 60-second video stream file each time.

[0124] In step S820 of some embodiments, the to-be-tested video data is processed by single-frame processing, the total frame number of the to-be-tested video data is obtained by software such as OpenCV, and the video is image-captured at a frequency of 15 frames per time to obtain a plurality of to-be-tested images. The resolution of the to-be-tested images can be 1920x1080, and the to-be-tested images can be stored in BMP, JPEG, PSD format, etc.

[0125] As shown in Figure 9 In some embodiments of the present application, the mechanical equipment motion state recognition method further includes but is not limited to steps S910 to S930, which will be described in detail below.

[0126] Step S910: if the motion result indicates that the to-be-tested device is in a motion state, taking the target image as a marked image;

[0127] Step S920: performing step S150 again until all reference images are traversed;

[0128] Step S930: obtaining a marked timestamp of all marked images, and performing video segment capture on the to-be-tested video data according to the marked timestamp to obtain a target video segment.

[0129] In step S910 of some embodiments, after obtaining the image pixel values ​​of any two target images with adjacent timestamps in step S160, the motion result is obtained according to the transformation amount between the two image pixel values. If the motion result indicates that the device under test is in motion, the two target images are used as marker images.

[0130] In step S920 of some embodiments, step S150 is executed again to use two other reference images with adjacent reference timestamps as target images and determine the motion result of the device under test in the target images. For example, if the first reference timestamp of the first reference image is "00:00:01" and the second reference timestamp of the second reference image is "00:00:02", then the first and second reference timestamps are adjacent reference timestamps. In this case, the first reference image is used as the first target image and the second reference image is used as the second target image. Subsequently, based on the first image pixel value of the first target image and the second image pixel value of the second target image, the motion result of the device under test in the first and second target images is determined. If the motion result indicates that the device under test is in motion, then the third reference timestamp of the third reference image in the reference image database is obtained. Let the third reference timestamp be "00:00:03". Then the third reference timestamp and the second reference timestamp are adjacent reference timestamps, and the third reference image is used as the third target image. Subsequently, based on the second image pixel values ​​of the second target image and the third image pixel values ​​of the third target image, the motion results of the device under test in the second and third target images are determined. This process is repeated for all reference images in the reference image database.

[0131] In step S930 of some embodiments, the timestamps of all marked images are obtained, and video segments with timestamps matching the marked timestamps in the video data to be tested are extracted to obtain the target video segment. In this target video segment, all devices under test are in motion. After obtaining the target video segment, the mechanical equipment in operation on the construction site, as well as its specific operating mode and operating time, can be directly determined by viewing the target video segment.

[0132] like Figure 10 As shown, this embodiment of the invention also provides a motion state recognition device for mechanical equipment, comprising:

[0133] Image acquisition module 110 is used to acquire the image of the device under test;

[0134] The category recognition module 120 is used to input the image to be tested into a preset target category recognition model for recognition, so as to obtain the target category of the device under test;

[0135] The position recognition module 130 is configured to input the to-be-tested image into a preset target position recognition model to obtain target position data of the to-be-tested device.

[0136] The grouping module 140 is configured to group the to-be-tested image according to the target category and the target position data to obtain a reference image database; wherein the to-be-tested image in the reference image database is used as a reference image.

[0137] The target image determination module 150 is configured to obtain a reference timestamp of the reference image, and use at least two reference images adjacent to the reference timestamp as target images.

[0138] The motion state recognition module 160 is configured to obtain image pixel values of the target images, and determine a motion result of the to-be-tested device according to the image pixel values; wherein the motion result is used to represent that the to-be-tested device is in a motion state or a non-motion state.

[0139] It can be seen that the content in the motion state recognition method for the mechanical device is applicable to the motion state recognition device for the mechanical device, the motion state recognition device for the mechanical device specifically implements the same functions as the motion state recognition method for the mechanical device, and achieves the same beneficial effects as the motion state recognition method for the mechanical device.

[0140] The following will be described in detail Figure 11 The electronic device of the embodiment of the present application is described in detail.

[0141] As Figure 11 , Figure 11 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device includes:

[0142] The processor 210 can be implemented in a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0143] The memory 220 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc. The memory 220 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 220 and are called and executed by the processor 210 to implement the motion state recognition method of the mechanical equipment according to the embodiments of the present disclosure.

[0144] The input / output interface 230 is configured to realize information input and output.

[0145] The communication interface 240 is configured to realize communication interaction between the device and other devices, and the communication can be realized in a wired manner (for example, a USB, a network cable, etc.) or in a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0146] The bus 250 is configured to transmit information between various components (for example, the processor 210, the memory 220, the input / output interface 230 and the communication interface 240) of the device.

[0147] The processor 210, the memory 220, the input / output interface 230 and the communication interface 240 are connected to each other in the device through the bus 250.

[0148] The embodiment of the present disclosure further provides a computer readable storage medium, and the computer readable storage medium stores computer executable instructions, and the computer executable instructions are configured to make a computer execute the motion state recognition method of the mechanical equipment according to any one of the above embodiments.

[0149] It can be seen that the content in the motion state recognition method of the mechanical equipment embodiment is applicable to the computer readable storage medium embodiment, the function realized by the computer readable storage medium embodiment is the same as that of the motion state recognition method of the mechanical equipment, and the beneficial effects achieved by the computer readable storage medium embodiment are also the same as those achieved by the motion state recognition method of the mechanical equipment.

[0150] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present embodiment.

[0151] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0152] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover the embodiments of the application described herein as appropriately followed by the term "for example". It is also to be understood that the terms "comprising", "including", "containing" and "having" and their conjugates, when used herein, are used in the sense of "including" and not of "consisting of" to cover also the case of "for example". It is also to be understood that the use of the term "or" when used in this document includes the conjunctive meaning of "and / or".

[0153] It should be understood that, in the present application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0154] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0156] In addition, each functional unit in various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0157] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0158] The preferred embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the present application is not limited thereto. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the present application shall fall within the scope of the present application.

Claims

1. A method of recognizing a motion state of a mechanical device, characterized by, The method comprises the following steps: acquiring a to-be-tested image of a to-be-tested device; inputting the to-be-tested image into a preset target category recognition model for recognition to obtain a device category of the to-be-tested device; inputting the to-be-tested image into a preset target position recognition model for recognition to obtain target position data of the to-be-tested device, the target position data being used to represent a pixel coordinate of a position of the to-be-tested device in the to-be-tested image; grouping the to-be-tested image according to the device category and the target position data to obtain a reference image database; wherein the to-be-tested image in the reference image database is taken as a reference image, the reference image is a to-be-tested image including the to-be-tested device, and to-be-tested images with the same device category and the same target position data form a group; acquiring a reference timestamp of the reference image, and taking at least two reference images adjacent in the reference timestamp as target images; acquiring image pixel values of the target images, and determining a motion result of the to-be-tested device according to the image pixel values; wherein the motion result is used to represent a motion state or a non-motion state of the to-be-tested device.

2. The method of claim 1, wherein The method of acquiring the image pixel values of the target images and determining the motion result of the to-be-tested device according to the image pixel values comprises the following steps: performing differential processing on the image pixel values to obtain pixel differential values; determining the motion result according to the pixel differential values and a preset reference value.

3. The method of claim 2, wherein The target position recognition model comprises an image segmentation layer and a classification layer. The method of inputting the to-be-tested image into the preset target position recognition model for recognition to obtain the target position data of the to-be-tested device comprises the following steps: performing region segmentation operation on the to-be-tested image according to the image segmentation layer to obtain a candidate region image; performing image recognition on the candidate region image according to the classification layer to obtain a region recognition result; wherein the region recognition result is used to represent whether the candidate region image includes the to-be-tested device or not; and taking the candidate region image represented by the region recognition result as a target region image; acquiring a target pixel position of the target region image in the to-be-tested image, and taking the target pixel position as the target position data.

4. The method of claim 3, wherein The method of grouping the to-be-tested image according to the device category and the target position data to obtain the reference image database comprises the following steps: grouping the target region image according to the device category and the target position data to obtain a preliminary image database; wherein a target region image in the preliminary image database is taken as a preliminary image; performing object boundary recognition on the preliminary image to obtain boundary line data of the to-be-tested device; performing intercepting operation on the preliminary image according to the boundary line data to obtain the reference image; constructing the reference image database according to the reference image.

5. The method of claim 4, wherein Before the step of inputting the to-be-tested image into the preset target category recognition model for recognition to obtain the device category of the to-be-tested device, the method further comprises training the target category recognition model, specifically comprising the following steps: obtain a sample image of a sample device, and a sample image label of the sample image; the sample image label is used to represent a device category of the sample device; input the sample image into a preset original category recognition model for recognition, to obtain an original image label; the original image label is used to represent the device category of the sample device; perform parameter adjustment on the original category recognition model according to the sample image label and the original image label, to obtain the target category recognition model.

6. The method of identifying a motion state of a mechanical device according to any one of claims 1 to 5, characterized by, Before the method of obtaining the to-be-tested image of the to-be-tested device, the method further comprises: obtain to-be-tested video data of the to-be-tested device; perform single-frame processing on the to-be-tested video data, to obtain the to-be-tested image.

7. The method of claim 6, wherein The method further comprises: if the motion result indicates that the to-be-tested device is in a motion state, take the target image as a marked image; again perform the obtaining of the reference timestamps of the reference images, and take at least two reference images adjacent in reference timestamp as target images, until all the reference images are traversed; obtain marked timestamps of all the marked images, and perform video clip cutting on the to-be-tested video data according to the marked timestamps, to obtain a target video clip.

8. A motion state recognition device for mechanical equipment, characterized in that, comprise: an image obtaining module, which is used to obtain a to-be-tested image of a to-be-tested device; a category recognition module, which is used to input the to-be-tested image into a preset target category recognition model for recognition, to obtain a device category of the to-be-tested device; a position recognition module, which is used to input the to-be-tested image into a preset target position recognition model for recognition, to obtain target position data of the to-be-tested device; the target position data is used to represent a pixel coordinate of a position of the to-be-tested device in the to-be-tested image; a grouping module, which is used to group the to-be-tested images according to the device category and the target position data, to obtain a reference image database; wherein, the to-be-tested images in the reference image database are taken as reference images, the reference images are to-be-tested images including the to-be-tested device, and to-be-tested images with the same device category and the same target position data form a group; a target image determination module, which is used to obtain reference timestamps of the reference images, and take at least two reference images adjacent in reference timestamp as target images; a motion state recognition module, which is used to obtain image pixel values of the target images, and determine a motion result of the to-be-tested device according to the image pixel values; wherein, the motion result is used to indicate that the to-be-tested device is in a motion state or a non-motion state.

9. An electronic device, comprising: comprise: at least one memory; at least one processor; at least one computer program; the computer program is stored in the memory, and the processor executes the at least one computer program to implement the motion state recognition method of the mechanical device according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to execute the motion state recognition method of the mechanical equipment according to any one of claims 1 to 7.

Images