Working state detection method, training method, equipment and device of control device

Abstract

This application discloses a method, training method, device, and apparatus for detecting the operating status of a control device. The method comprises: acquiring a video frame of a target to be detected, wherein the target to be detected includes a first object and a second object, the first object being controlled by a control device and acting on the second object; performing operating status detection on the first object and the second object in the video frame, respectively, to obtain a first state of the first object and a second state of the second object; and determining the operating status of the control device based on the first and second states. This approach can improve the accuracy of operating status detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for detecting the working state of a control device, a method for training a working state detection model, a computer device, and a storage device. Background Art

[0002] With the advancement of technology, more and more industries are becoming mechanized. For example, in areas such as transportation, material transport, engineering construction, and manufacturing, equipment is increasingly being used to replace manual labor, enabling more efficient tasks. Currently, equipment operating conditions can be abnormal. Failure to accurately monitor the operating status of equipment can affect its normal operation and result in wasted resources and energy. Summary of the Invention

[0003] The main technical problem solved by this application is to provide a working status detection method, training method, equipment and device for a control device, which can improve the accuracy of working status detection.

[0004] In order to solve the above problems, the first aspect of the present application provides a method for detecting the working status of a control device, the method comprising: obtaining a video frame of a target to be detected, wherein the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object; performing working status detection on the first object and the second object in the video frame respectively to obtain a first state of the first object and a second state of the second object; and determining the working status of the control device based on the first state and the second state.

[0005] In order to solve the above problems, the second aspect of the present application provides a training method for a working status detection model, which includes: taking a video frame of a preset video of a target to be detected as a sample image, wherein the target to be detected includes a first object and a second object, the first object is controlled by the target to be detected, and the first object acts on the second object; performing data enhancement processing on the sample image to obtain an enhanced sample image; inputting the enhanced sample image into the working status detection model for processing to obtain a first state corresponding to the first object and a second state corresponding to the second object; adjusting the parameters of the working status detection model based on the first state and the second state to obtain a trained working status detection model.

[0006] In order to solve the above problems, the third aspect of the present application provides a computer device, which includes a memory and a processor coupled to each other, wherein program data is stored in the memory, and the processor is used to execute the program data to implement any step of the working status detection method of the above-mentioned control device and / or the training method of the working status detection model.

[0007] In order to solve the above problems, the fourth aspect of the present application provides a storage device, which stores program data that can be run by a processor, and the program data is used to implement any step of the working status detection method of the above-mentioned control device and / or the training method of the working status detection model.

[0008] The above scheme obtains a video frame of a target to be detected, where the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object; working status detection is performed on the first object and the second object in the video frame respectively to obtain a first state of the first object and a second state of the second object; based on the first state and the second state, the working state of the control device is determined. Since the control device is detected by using the states of the first object and the second object that have an operational relationship with the control device, the accuracy of the working status detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of this application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0010] Figure 1 This is a flow chart of a first embodiment of a method for detecting the working state of a control device of the present application;

[0011] Figure 2 This is a schematic diagram of an application scenario of an embodiment of a method for detecting the working state of a control device of the present application;

[0012] Figure 3 This application Figure 1 A flow chart of an embodiment of step S12;

[0013] Figure 4 This application Figure 3 A flow chart of an embodiment of step S121;

[0014] Figure 5 This is an example schematic diagram of an embodiment of a video frame of the present application;

[0015] Figure 6 This is an example schematic diagram of an embodiment of a detection area in a video frame of the present application;

[0016] Figure 7 This is an exemplary schematic diagram of an embodiment of the detection area of ​​the present application;

[0017] Figure 8 This is an exemplary schematic diagram of an embodiment of the first target area and the second target area in the detection area of ​​the present application;

[0018] Figure 9 This is an exemplary schematic diagram of another embodiment of the first target area and the second target area in the detection area of ​​the present application;

[0019] Figure 10 This is an exemplary schematic diagram of an embodiment of the first target area of ​​the present application;

[0020] Figure 11 This is an exemplary schematic diagram of an embodiment of the second target area of ​​the present application;

[0021] Figure 12 This is a flow chart of a second embodiment of the method for detecting the working state of a control device of the present application;

[0022] Figure 13 This is a flow chart of an embodiment of a training method for a working status detection model of the present application;

[0023] Figure 14 This is a structural diagram of an embodiment of the working status detection device of the present application;

[0024] Figure 15 This is a schematic structural diagram of an embodiment of a computer device of the present application;

[0025] Figure 16 It is a structural diagram of an embodiment of the storage device of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The terms "first" and "second" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0028] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] This application provides the following embodiments, and each embodiment is described in detail below.

[0030] See also Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for detecting the working state of the control device of the present application. The method may include the following steps:

[0031] S11: Acquire a video frame of a target to be detected, wherein the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object.

[0032] When the control device is operating, the control device can control the use of the first object so as to use the first object to act on the second object to process the second object.

[0033] In some embodiments, when the control device is in an on state, the first object can act on the second object. When the control device is in an off state, the second object can be prevented from acting on the second object.

[0034] When detecting the working state of the control device, a video frame of the target to be detected can be obtained, wherein the target to be detected includes a first object and a second object, that is, a video frame containing the first object and / or the second object can be obtained.

[0035] In some embodiments, a camera device may be used to capture the target in real time to obtain a target video, and a video frame may be obtained from the target video in real time.

[0036] In some implementations, each video frame of the target video may be used as a video frame of the target to be detected, so that the working status can be detected in real time through the video frames of the target video.

[0037] In some embodiments, the control device includes a dust suppression device, which can be used to spray the first object to suppress dust on the second object.

[0038] In some embodiments, the control device includes a spraying device, which can be used to spray a first object to water a second object, etc. For example, in agricultural and landscaping applications, the spraying device can spray a first object (e.g., water) to water a second object (e.g., a plant).

[0039] The embodiments of the present application are described using an example of a control device including a dust suppression device. Of course, the control device of the present application can also be used in other operation scenarios where a first object processes a second object, and the control device, first object, and second object of the present application are not limited thereto.

[0040] See also Figure 2 Taking the transportation of a second item as an example, the second item can be loaded onto a transport vehicle and transported. If the second item includes items that are prone to generating dust, such as coal, sand, and gravel, the second item will generate a large amount of dust during transportation, which not only seriously pollutes the environment but also seriously affects the health of nearby people. To address dust pollution, spraying the first item can be used to achieve a dust reduction effect.

[0041] A control device (such as a dust suppression device) 103 can be installed on the transport vehicle. The dust suppression device can be used to spray the first object, which can include dust suppressants such as dust suppressant, water, and dust-proof covering agent. The first object can be sprayed on the second object for dust suppression.

[0042] The transport vehicle's carriage 104 can be loaded with a second object (not shown). A camera 101 can be installed on the transport vehicle's carriage 104 to capture the entire spraying state of the dust suppression device, thereby positioning the dust suppression device's spraying area in the center of the image. The spraying area can be the spraying range of the dust suppression device for the first object, or it can be the area where the second object is placed. This application does not impose any restrictions on this.

[0043] The camera 101's shooting area 102 (or lens orientation, shooting angle) is opposite to the transport vehicle's forward direction. Alternatively, the camera 101's lens orientation may be at a predetermined angle relative to the transport vehicle's forward direction. The camera 101's shooting area 102 can cover the spraying area of ​​the control device 103, thereby capturing a target video of the first and second objects. A video frame is obtained from the target video, where the video frame includes the area containing the first and second objects.

[0044] In some embodiments, the lens of the camera device 101 may be fixed to capture the entire spraying area of ​​the control device 103 , that is, to capture the first object and the second object in the spraying area.

[0045] In some embodiments, the lens of the camera device 101 may be variable, for example, by moving the lens parallely or rotating the lens, and the shooting angle of the camera device 101 may capture the spraying area or part of the spraying area of ​​the control device 103 .

[0046] In some embodiments, the lens of the camera 101 can be variable, and the control device 103 can change the spraying angle, spraying amount, etc. on the first object. The camera 101 can capture each area of ​​the second object to detect the degree of effect (dust suppression treatment degree) of the first object on the second object in each area. Therefore, the dust suppression device can change the spraying angle, spraying amount, etc. on the first object to perform dust suppression treatment on the second object in areas where dust suppression treatment is insufficient.

[0047] S12: Performing working status detection on the first object and the second object in the video frame respectively to obtain a first status of the first object and a second status of the second object.

[0048] Since the video frame includes the areas where the first object and the second object belong, the video frame can be used to detect the working states of the first object and the second object in the video frame to obtain the first state of the first object and the second state of the second object.

[0049] In some embodiments, when performing working status detection on a first object and a second object in a video frame, respectively, the image frames may be cropped to obtain cropped video frames for detecting the working status of the first object and cropped video frames for detecting the working status of the second object, and the cropped video frames may be used to perform working status detection on the first object and the second object in the video frame, respectively, to obtain a first state of the first object and a second state of the second object.

[0050] In some embodiments, for example, the control device includes a dust suppression device that can be used to spray a first object to suppress dust from a second object. A first state classification corresponding to the first object can include spraying or not spraying; and a second state classification corresponding to the second object can include the presence of the second object or the absence of the second object.

[0051] S13: Determine the operating state of the control device based on the first state and the second state.

[0052] Since the first object is controlled by the control device, the first state of the first object is affected by the working state of the control device. Therefore, the working state of the control device can be judged by the first state of the first object.

[0053] Taking the control device including a dust suppression device as an example, if the first state of the first object is spraying, the working state of the control device is open; if the first state of the first object is not spraying, it can be determined that the working state of the control device is closed.

[0054] Since the first object acts on the second object, it is possible to determine whether the first object needs to act on the second object according to the working state of the second object, that is, whether the first object needs to perform dust suppression on the second object.

[0055] Whether the working state of the control device is normal can be determined through the first state of the first object and the second state of the second object.

[0056] In this embodiment, a video frame of a target to be detected is obtained, where the target to be detected includes a first object and a second object, the first object being controlled by a control device, and the first object acting on the second object; working status detection is performed on the first object and the second object in the video frame respectively to obtain a first state of the first object and a second state of the second object; and the working state of the control device is determined based on the first state and the second state. Since the control device is detected by utilizing the states of the first object and the second object that have an operational relationship with the control device, the accuracy of the working state detection can be improved.

[0057] In addition, the present application uses a camera to shoot the target in real time to obtain a target video. Video frames are obtained from the target video in real time to detect the working status of the first object and the second object. The device has few components and is easy to install, which can reduce the cost of detecting the working status.

[0058] In some embodiments, see Figure 3 The above step S12, detecting the working status of the first object and the second object in the video frame respectively to obtain the first state of the first object and the second state of the second object, may include the following steps:

[0059] S121: Acquire a first target area corresponding to a first object and a second target area corresponding to a second object in a video frame.

[0060] A video frame includes an area where a first object and a second object are located. A first target area corresponding to the first object and a second target area corresponding to the second object in the video frame can be obtained. The first target area and the second target area refer to images obtained by cropping the video frame. The first target area includes the first object, and the second target area includes the second object.

[0061] In some embodiments, to more accurately determine the first state of the first object and the second state of the second object and reduce the impact of background noise in the video frame on detection, the area within the video frame where the first and second objects are located can be extracted. In other words, a detection area corresponding to the target to be detected can be determined. This detection area can be set based on the camera's capture area (e.g., the center of the image) or determined based on a preset algorithm. This detection area includes the first and second objects. The video frame can be cropped to obtain the detection area.

[0062] The detection area may be an area of ​​high interest in a video frame during user or work status detection. The detection area may also be referred to as a ROI (Region Of Interest).

[0063] In some embodiments, to prevent image distortion of the video frame from affecting the working status detection effect, a portion of the central area of ​​the video frame can be cropped as the detection area, and the detection area maintains the image ratio. The image ratio of the detection area can be consistent with or similar to that of the video frame, that is, the aspect ratio of the detection area can be the same or similar to the aspect ratio of the video frame.

[0064] The detection area is divided into a first target area and a second target area, wherein the first target area and the second target area can be obtained by cropping the detection area. To avoid image distortion, the first target area and the second target area can be obtained by cropping the middle area of ​​the detection area.

[0065] In some embodiments, to avoid image distortion, at least one of the first target area and the second target area may have the same image ratio as the detection area. For example, the first target area and the second target area may have the same image ratio as the detection area, that is, the aspect ratio of the first target area and the second target area may be the same as the aspect ratio of the detection area.

[0066] In some embodiments, see Figure 4 The above step S121 may include the following steps:

[0067] S1211: Acquire the coordinates of several area positioning points input by the user, or determine the coordinates of several area positioning points of the detection area using a preset algorithm.

[0068] In some embodiments, the coordinates of several area positioning points input by the user can be obtained. The coordinates of the several area positioning points can be the coordinates of the several area positioning points externally configured by the user for the detection area. Alternatively, the coordinates of the several area positioning points generated by the user clicking on the detection area screen in the video frame can be obtained. This application does not impose any restrictions on this.

[0069] In some embodiments, the coordinates of several area positioning points of the detection area may be determined using a preset algorithm. For example, the preset algorithm may calculate the coordinates of several area positioning points of the detection area based on the shooting area or shooting angle of the camera device. For example, the preset algorithm may obtain the coordinates of several area positioning points of the detection area from the video frame based on the size of the detection area input by the user.

[0070] See also Figures 5 and 6 As an example, a video frame may include a first object and / or a second object. In some application scenarios, the first object and the second object have a layered feature in the video frame.

[0071] The preset shape can be a rectangle. Obtain the coordinates of the four or two regional positioning points configured for the detection area (ROI area) input by the user. The coordinates of the four regional positioning points of the detection area may include: coordinate 1 (x0, y0), coordinate 2 (x1, y1), coordinate 3 (x2, y2), and coordinate 4 (x3, y3). If the coordinates of two regional positioning points are configured, the coordinates of the two regional positioning points of the detection area may include: coordinate 1 (x0, y0) and coordinate 4 (x3, y3). Among them, coordinate 1 and coordinate 4 are the diagonal coordinates of the rectangle.

[0072] S1212: Based on the coordinates of the plurality of regional positioning points, determine a detection area that covers the plurality of regional positioning points and has a preset shape.

[0073] After obtaining the coordinates of several area positioning points in the detection area, a detection area of ​​a predetermined shape covering the area positioning points can be determined based on the coordinates of the area positioning points. For example, the coordinates of the area positioning points can be used to form a minimum circumscribed polygon of the area positioning point coordinates, and this polygon can be used as the detection area.

[0074] See also Figure 7 As an example, we can use the coordinates of four regional positioning points: coordinate 1 (x0, y0), coordinate 2 (x1, y1), coordinate 3 (x2, y2), and coordinate 4 (x3, y3). We can find the minimum bounding rectangle of the four regional positioning points and use this minimum bounding rectangle to crop the video frame to obtain the detection area.

[0075] The coordinates of the first vertex of the detection area are (x min ,y min ), the coordinates of the second vertex are (x max ,y max ). The first vertex coordinates may include the upper left corner coordinates, and the second vertex coordinates may include the lower right corner coordinates. The present application is not limited thereto.

[0076] In this embodiment, by acquiring the detection area from the video frame, the influence of image distortion can be avoided, and the accuracy of state classification performed by the subsequent working state classification model can be improved.

[0077] S1213: Using the coordinates of the first vertex and the second vertex of the detection area, respectively obtain the coordinates of the first vertex and the second vertex of the first target area and the coordinates of the first vertex and the second vertex of the second target area.

[0078] In some embodiments, within the inspection area, the inspection area can be cropped to generate a first target area and a second target area based on the upper and lower layer characteristics of the second object. To avoid image distortion, only the center area of ​​the inspection area can be cropped. Furthermore, the image ratios of the first target area, the second target area, and the inspection area are the same.

[0079] In some embodiments, the coordinates of the first vertex and the second vertex of the detection area can be used to obtain the coordinates of the first vertex and the second vertex of the first target area, and the coordinates of the first vertex and the second vertex of the second target area, respectively. The first vertex coordinates may include the coordinates of the upper left corner, and the second vertex coordinates may include the coordinates of the lower right corner. The present application is not limited to this.

[0080] In some embodiments, see Figure 8 The first target area and the second target area may not overlap in the detection area. This approach allows the first target area to retain more information about the first object, while the second target area to retain more information about the second object.

[0081] In some embodiments, see Figure 9 The first target area and the second target area may have overlapping areas in the detection area. This method can obtain more image information of the first target area and the second target area.

[0082] Taking the example that the first target area and the second target area do not have overlapping areas in the detection area, the coordinates of the first vertex of the detection area (x min ,y min ) and the second vertex coordinate (x max ,y max ), get the first vertex coordinate (x spray_0 ,y spray_0 ) and the coordinates of the second vertex (x spray_1 ,y spray_1 ), wherein the first vertex coordinates may include the upper left corner coordinates, and the second vertex coordinates may include the lower right corner coordinates. The obtained first vertex coordinates and second vertex coordinates can be expressed as:

[0083]

[0084] In the above formula (1), A is the first parameter of the first target, and A is a constant. For example, A can be B is the second parameter of the first target, B is a constant, for example, B can be 1; C is the third parameter of the first target, C is a constant, for example, C can be D is the fourth parameter of the first target, and D is a constant. For example, D can be

[0085] In some embodiments, in the above formula (1), (1-A), (1-C), and (1-D) can also be replaced by other parameters, and this application does not impose any limitation on this.

[0086] The coordinates of the first vertex of the detection area (x min ,y min ) and the second vertex coordinate (x max ,y max ), get the coordinates of the first vertex of the second target area (x coal_0 ,y coal_0 ) and the coordinates of the second vertex (x coal_1 ,y coal_1 ), the obtained coordinates of the first vertex and the second vertex can be expressed as:

[0087]

[0088] In the above formula (2), E is the first parameter of the second target, and E is a constant. For example, E can be F is the second parameter of the second target, and F is a constant. For example, F can be G is the third parameter of the second objective, and G is a constant. For example, G can be G is the fourth parameter of the second target, and H is a constant, for example, H can be 1.

[0089] In some embodiments, in the above formula (1), (1-E), (1-F), and (1-G) may be replaced by other parameters, and this application does not impose any limitation on this.

[0090] See also Figures 10 and 11 Based on the obtained coordinates of the first vertex and the second vertex of the first target area, the first target area in the detection area can be determined, and the first target area can be clipped from the detection area. Based on the coordinates of the first vertex and the second vertex of the second target area, the second target area can be determined, and the second target area can be clipped from the detection area.

[0091] S122: Using the first target area and the second target area, respectively, the working status of the first object and the second object are detected to obtain a first status and a second status.

[0092] In some implementations, before step S122 , the first target area and / or the second target area may be pre-processed.

[0093] The preprocessing may include adjusting the sizes of the first target area and the second target area to be consistent with the size of the detection area. For example, the images of the first target area and the second target area may be enlarged to reach the same size as the pre-detection area.

[0094] The pixels of the first target region and the second target region may also be normalized, for example, each channel (eg, red pixel, blue pixel, green pixel) of the first target region and the second target region may be divided by 255 to obtain pre-processed first target region and second target region.

[0095] Thus, the working state of the first object is detected by using the first target area to obtain the first state, and the working state of the second object is detected by using the second target area to obtain the second state.

[0096] In some implementations, the trained working state detection model may be used to process the first target area and the second target area respectively to obtain probability values ​​corresponding to the state classifications of the first state and the second state, respectively.

[0097] The trained working state detection model may be a deep learning network, for example, a ResNet-18 network. The first target area and the second target area are input into the ResNet-18 network to detect the states of the first and second objects, and output probability values ​​corresponding to the state classifications of the first and second states.

[0098] In this embodiment, by using a deep learning network (such as a ResNet-18 network) to detect the first state of the first object and the second state of the second object, the accuracy of detecting the first state and the second state can be improved.

[0099] Detection thresholds can be pre-set for the first state and the second state, with the first state corresponding to a first preset threshold and the second state corresponding to a second preset threshold. The first preset threshold can be the same as or different from the second preset threshold. For example, the first preset threshold and the second preset threshold are both 0.5.

[0100] If the probability value of the first state belonging to the first state classification is greater than the first preset threshold, the first state classification is represented by the first character; otherwise, it is represented by the second character. The first state classification includes spraying or not spraying. This application uses the first state classification of spraying as an example for explanation. If the probability value of the first state belonging to spraying is greater than the first preset threshold, the state classification of the first state is represented by the first character "1", otherwise, it is represented by the second character "0". This application is not limited to this.

[0101] If the probability value of the second state belonging to the second state classification is greater than the second preset threshold, the second state classification is represented by the third character; otherwise, it is represented by the fourth character. The second state classification includes the presence of the second item or the absence of the second item. This application uses the second state classification of the presence of the second item as an example for explanation. If the probability value of the second state belonging to the presence of the second item is greater than the second preset threshold, the state classification of the second state is represented by the third character "a"; otherwise, it is represented by the fourth character "b". This application is not limited to this.

[0102] In some embodiments, the first state and the second state can use the same character to record the state. If the probability value of the second state belonging to the second state classification is greater than the second preset threshold, the second state classification is represented by the first character, otherwise it is represented by the second character. The second state classification includes the presence of the second item or the absence of the second item. This application uses the second state classification of the second item as an example for explanation. If the probability value of the second state belonging to the presence of the second item is greater than the second preset threshold, the state classification of the second state is represented by the first character "1", otherwise, it is represented by the second character "0". The following is explained using this example.

[0103] The first state or the second state can be expressed by the following formula:

[0104]

[0105] In the above formula (3), thresh represents the first preset threshold or the second preset threshold, state can represent the first state spray_state or the second state coal_state, α i Indicates the probability value of the first state classification or the probability value of the second state classification.

[0106] In this embodiment, by setting classification detection thresholds (first preset thresholds or second preset thresholds) for the state classification of the first state and the state classification of the second state respectively, false detections of the first state and the second state can be reduced.

[0107] In some embodiments, the above step S13, determining the working state of the control device based on the first state and the second state, may include: determining whether the working state of the control device is normal or abnormal based on the characters used in the state classification of the first state and the state classification of the second state.

[0108] Specifically, the first state and the second state can be represented by the following table:

[0109] Table 1: State classification table of the first state and the second state

[0110]

[0111] In Table 1 above, 11 indicates spraying the first object in the presence of the second object, 01 indicates not spraying the first object in the presence of the second object, 10 indicates spraying the first object in the absence of the second object, and 00 indicates not spraying the first object in the absence of the second object.

[0112] From the above four states, we can see that 11 and 00 indicate that the control device is in normal working state, while 10 and 01 indicate abnormal working state. Based on the characters used in the state classification of the first state and the state classification of the second state, the XOR operation of mathematical logic can be used to determine whether the working state of the control device is normal or abnormal. The normal or abnormal working state of the control device can be expressed as:

[0113] alarm_state=spray_state⊕coal_state (4)

[0114] In formula (4), alarm_state represents the operating state of the control device, indicating whether it is normal or abnormal. An output of 0 indicates normal operation, while an output of 1 indicates abnormal operation. spray_state represents the first state of the first object, and coal_state represents the second state of the second object. ⊕ represents the exclusive OR operation in mathematical logic. If the values ​​of spray_state and coal_state are different, the exclusive OR of alarm_state is 1. If the values ​​of spray_state and coal_state are the same, the exclusive OR of alarm_state is 0.

[0115] The characters of the first state of the first object and the second state of the second object can be used to determine whether the operating state of the control device is abnormal. If it is determined to be abnormal, the abnormality of the control device can be handled. For details, please refer to the following embodiments.

[0116] See also Figure 12 , Figure 12This is a flow chart of the second embodiment of the method for detecting the working state of the control device of the present application. The method may include the following steps:

[0117] S21: Acquire a video frame of a target to be detected, wherein the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object.

[0118] S22: Performing working status detection on the first object and the second object in the video frame respectively to obtain a first status of the first object and a second status of the second object.

[0119] S23: Based on the first state and the second state, determine whether the working state of the control device is normal.

[0120] If it is determined that the working state of the control device is abnormal, step S24 is executed, and then step S21 is continued to obtain the next video frame of the target to be detected, and the working state of the control device is detected.

[0121] The specific implementation process of steps S21 to S23 in this embodiment can refer to the implementation process of steps S11 to S13 in the above embodiment, and this application will not repeat them here.

[0122] S24: Count the abnormal duration of the abnormal working status.

[0123] After determining that the working state of the control device is abnormal, the abnormal duration of the working state may be counted, and it may be determined whether the abnormal duration exceeds a preset duration.

[0124] If the accumulated abnormal duration exceeds the preset duration, step S25 is executed.

[0125] If the accumulated abnormal duration does not exceed the preset duration, step S21 may be continued.

[0126] S25: In response to the abnormality duration exceeding the first preset duration, issuing a prompt message; and / or, in response to the abnormality duration exceeding the second preset duration, adjusting the working state of the control device.

[0127] If the abnormality duration exceeds the first preset duration, a prompt message may be issued in response to the abnormality duration exceeding the first preset duration, for example, an alarm sound, an alarm message, etc. may be issued.

[0128] If the abnormality duration exceeds the first preset duration, the working state of the control device may be adjusted in response to the abnormality duration exceeding the second preset duration.

[0129] The first preset time length and the second preset time length may be the same. In some application scenarios, in response to the prompt information, the working state of the control device is adjusted so that the user can receive the prompt information and check whether the working state of the control device is adjusted.

[0130] The first preset duration and the second preset duration may be different, so that different adjustment methods can be set for different abnormal situations. In response to the abnormal duration exceeding the first preset duration, a prompt message is issued, and in response to the abnormal duration exceeding the first preset duration, the working state of the control device is adjusted.

[0131] For example, the control device includes a dust suppression device configured to spray a first object to suppress dust from a second object. The first state corresponding to the first object includes spraying and not spraying, and the second state corresponding to the second object includes the presence of the second object and the absence of the second object.

[0132] If the first state is spraying and the second state is no second object, it can be determined that the working state of the control device is abnormal, and the working state of the control device is adjusted to off.

[0133] If the first state is no spraying and the second state is the presence of the second object, it can be determined that the working state of the control device is abnormal, and the working state of the control device is adjusted to open.

[0134] In some implementations, after the working state of the control device is adjusted, the statistical abnormal duration of the abnormal working state may be cleared, and the working state may continue to be detected.

[0135] In this embodiment, when the working state of the control device is abnormal and the abnormal duration reaches a preset duration, the operation of the control device can be adjusted, which can reduce misjudgment of abnormalities of the control device and avoid frequent prompt messages.

[0136] See also Figure 13 , Figure 13 This is a flow chart of an embodiment of a training method for a working status detection model of the present application. The method may include the following steps:

[0137] S31: Using a video frame of a preset video of a target to be detected as a sample image, wherein the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object.

[0138] The target to be detected can be photographed using a camera to obtain a preset video; and video frames are obtained from the target video as sample images.

[0139] This step can refer to the specific implementation process in the above embodiment, and this application will not go into details here.

[0140] S32: Perform data enhancement processing on the sample image to obtain an enhanced sample image.

[0141] In some embodiments, data enhancement processing may be performed on the sample image to obtain an enhanced sample image, wherein the data enhancement processing may include mirror processing, histogram transformation, etc.

[0142] In some embodiments, before step S32, a detection region (ROI) corresponding to the target to be detected can be determined. Specifically, this can include obtaining the coordinates of several region positioning points input by the user, or using a preset algorithm to determine the coordinates of several region positioning points of the detection area; based on the coordinates of the several region positioning points, a detection region of a preset shape covering the several region positioning points is determined. In other words, the detection region can be cropped from the sample image. The detection region has the same aspect ratio as the sample image.

[0143] Data enhancement processing is performed on the detection area to obtain enhanced sample images. Data enhancement processing can include mirroring, histogram transformation, etc. For example, the mirroring process can be left-right mirroring, thereby converting one detection area into two detection areas. This method can increase the number of sample images and enhance the correlation between sample images.

[0144] S33: Inputting the enhanced sample image into the working state detection model for processing to obtain a first state corresponding to the first object and a second state corresponding to the second object.

[0145] Before step S33, that is, before the enhanced sample image is input into the working state detection model for processing, a first target area and a second target area can be divided from the enhanced sample image, wherein at least one of the first target area and the second target area has the same image ratio as the enhanced sample image.

[0146] In some embodiments, the coordinates of the first and second vertices of the enhanced sample image may be used to respectively obtain the coordinates of the first and second vertices of the first target region and the coordinates of the first and second vertices of the second target region.

[0147] The sizes of the first target area and the second target area are adjusted to be consistent with the size of the detection area, and the pixels of the first target area and the second area are normalized.

[0148] Then, the first target area and the second target area are input into the working state detection model for processing to obtain the first state corresponding to the first item and the second state corresponding to the second item. The working state detection model includes a ResNet-18 network, which is used to process the first target area and the second target area respectively to obtain the probability values ​​corresponding to the state classifications of the first state and the second state respectively. If the probability value of the first state belonging to the first state classification is greater than the first preset threshold, the first state classification is represented by the first character, otherwise it is represented by the second character; if the probability value of the second state belonging to the second state classification is greater than the second preset threshold, the second state classification is represented by the third character, otherwise it is represented by the fourth character.

[0149] S34: Based on the first state and the second state, adjust the parameters of the working state detection model to obtain a trained working state detection model.

[0150] Based on the characters used in the state classification of the first state and the state classification of the second state, it is determined whether the working state of the control device is normal or abnormal, and the parameters of the working state detection model are adjusted to obtain a trained working state detection model.

[0151] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0152] For the above embodiment, this application proposes a working status detection device. Figure 14 , Figure 14 1 is a schematic diagram of the structure of an embodiment of the working state detection device of the present application. The working state detection device 40 includes an acquisition module 41, a detection module 42 and a determination module 43, wherein the acquisition module 41, the detection module 42 and the determination module 43 are connected.

[0153] The acquisition module 41 is used to acquire a video frame of a target to be detected, wherein the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object.

[0154] The detection module 42 is configured to detect the working states of the first object and the second object in the video frame respectively, and obtain a first state of the first object and a second state of the second object.

[0155] The determination module 43 is configured to determine the operating state of the control device based on the first state and the second state.

[0156] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0157] For the above embodiment, this application provides a computer device, see Figure 15 , Figure 15 1 is a schematic diagram of the structure of an embodiment of a computer device of the present application. The computer device 50 includes a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled to each other, the memory 51 stores program data, and the processor 52 is configured to execute the program data to implement the steps of any of the embodiments of the aforementioned control device operating state detection method and / or operating state detection model training method.

[0158] In this embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor 52 may be any conventional processor.

[0159] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0160] The method of the above embodiment can be implemented in the form of a computer program, so this application proposes a storage device, see Figure 16 , Figure 16 Schematic diagram of the structure of an embodiment of a storage device of the present application. The storage device 60 stores program data 61 that can be executed by a processor. The program data 61 can be executed by the processor to implement the steps of any embodiment of the above-mentioned control device working state detection method and / or working state detection model training method.

[0161] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0162] The storage device 60 of this embodiment can be a medium that can store program data 61, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the program data 61. The server can send the stored program data 61 to other devices for execution, or it can also execute the stored program data 61 by itself.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0164] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage device, which is a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application.

[0167] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device. They can be concentrated on a single computing device or distributed across a network consisting of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0168] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for detecting the working state of a control device, characterized in that: The method comprises: Acquire a video frame of an object to be detected, wherein the object to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object; Performing working state detection on a first object and a second object in the video frame respectively to obtain a first state of the first object and a second state of the second object; determining an operating state of the control device based on the first state and the second state; If it is determined that the working state of the control device is abnormal, then counting the abnormal duration of the working state being abnormal; In response to the abnormality duration exceeding a second preset duration, adjusting the working state of the control device; The control device includes a dust suppression device, which is used to spray the first object to suppress dust on the second object; and adjusting the working state of the control device includes: If the first state is spraying and the second state is no second object, the working state of the control device is adjusted to off; or if the first state is no spraying and the second state is the presence of the second object, the working state of the control device is adjusted to on.

2. The method according to claim 1, characterized in that The detecting the working states of the first object and the second object in the video frame respectively to obtain a first state of the first object and a second state of the second object includes: Acquire a first target area corresponding to the first object and a second target area corresponding to the second object in the video frame; The first target area and the second target area are used to detect the working status of the first object and the second object respectively, so as to obtain the first status and the second status.

3. The method according to claim 2, characterized in that The acquiring of a first target area corresponding to the first object and a second target area corresponding to the second object in the video frame includes: Determining a corresponding detection area for the target to be detected; The first target area and the second target area are divided from the detection area, wherein at least one of the first target area and the second target area has the same image ratio as the detection area.

4. The method according to claim 3, characterized in that Determining a corresponding detection area for the target to be detected includes: Obtaining coordinates of several area positioning points input by a user, or determining the coordinates of the several area positioning points of the detection area using a preset algorithm; Determining, based on the coordinates of the plurality of area positioning points, the detection area covering the plurality of area positioning points and having a preset shape; The dividing the first target area and the second target area from the detection area includes: The coordinates of the first vertex and the second vertex of the first target area and the coordinates of the first vertex and the second vertex of the second target area are respectively acquired by using the coordinates of the first vertex and the second vertex of the detection area.

5. The method according to claim 2, characterized in that The detecting the working status of the first object and the second object using the first target area and the second target area respectively to obtain the first status and the second status includes: Using the trained working state detection model to process the first target area and the second target area respectively, and obtain probability values ​​corresponding to the state classifications of the first state and the second state respectively; If the probability value of the first state belonging to the first state category is greater than a first preset threshold, the first state category is represented by the first character, otherwise it is represented by the second character; If the probability value that the second state belongs to the second state category is greater than a second preset threshold, the second state category is represented by the third character; otherwise, the second state category is represented by the fourth character.

6. The method according to claim 5, characterized in that Before processing the first target area and the second target area respectively using the trained working state detection model, the method further includes: Adjusting the sizes of the first target area and the second target area to be consistent with the size of the detection area; Normalization is performed on pixels of the first target area and the second area.

7. The method according to claim 5, characterized in that The determining the operating state of the control device based on the first state and the second state includes: Based on the characters used in the status classification of the first state and the status classification of the second state, it is determined whether the working state of the control device is normal.

8. The method according to claim 1, characterized in that Also includes: In response to the abnormal duration exceeding the first preset duration, a prompt message is issued.

9. The method according to claim 8, characterized in that The state classification of the first state corresponding to the first object includes spraying and not spraying; the state classification of the second state corresponding to the second object includes the presence of the second object and the absence of the second object.

10. A training method for a working state detection model, characterized in that: The method comprises: Using a video frame of a preset video of a target to be detected as a sample image, wherein the target to be detected includes a first object and a second object, the first object is controlled by a control device, and the first object acts on the second object; performing data enhancement processing on the sample image to obtain an enhanced sample image; Inputting the enhanced sample image into a working state detection model for processing to obtain a first state corresponding to the first object and a second state corresponding to the second object; Based on the first state and the second state, adjusting parameters of the working state detection model to obtain a trained working state detection model; wherein, the trained working status detection model is used to realize working status detection of the first object and the second object in the video frame respectively, and obtain the first state of the first object and the second state of the second object; so that the working state of the control device is determined based on the first state and the second state; if the working state of the control device is determined to be abnormal, the abnormal duration of the abnormal working state is counted; in response to the abnormal duration exceeding the second preset duration, the working state of the control device is adjusted; wherein, the control device includes a dust suppression device, and the dust suppression device is used to spray the first object to suppress dust on the second object; the adjustment of the working state of the control device includes: if the first state is spraying and the second state is no second object, the working state of the control device is adjusted to closed; or, if the first state is no spraying and the second state is with the second object, the working state of the control device is adjusted to open.

11. A computer device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement the steps of the method according to any one of claims 1 to 10.

12. A storage device, characterized in that: Program data that can be executed by a processor is stored, and the program data is used to implement the steps of the method according to any one of claims 1 to 10.

Images