A terminal device status detection method, model generation method and device

Abstract

The embodiments of the present invention provide a terminal device status detection method, a model generation method and an apparatus. The terminal device status detection method includes extracting a screen area image of at least one terminal device from a collected monitoring image; inputting the screen area image of the terminal device into a preset first status detection model; and determining the operating status of the terminal device according to the first classification information output by the first status detection model; wherein the first classification information includes a normal screen status and an abnormal screen status. Thus, the first status detection model can be used to detect whether the screen of the terminal device in the monitoring image is displayed normally, so as to determine whether the terminal device is operating normally. The monitoring image is used to automatically monitor the terminal device to ensure the normal operation of the terminal device.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a terminal device state detection method, a terminal device state detection model generation method, a terminal device state detection device, and a terminal device state detection model generation device. Background Art

[0002] At present, several terminals are usually installed in public spaces such as business halls, shopping malls, and transportation hubs of various enterprises. Customers can use the terminals to handle business by themselves, which can improve the efficiency of business handling to a certain extent. However, since the areas where the terminals are installed are usually unsupervised, it is usually difficult to detect the damage of the terminal in time, which affects the customer's use of the terminal. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a terminal device status detection method, a terminal device status detection model generation method, a terminal device status detection device, and a terminal device status detection model generation device that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problem, an embodiment of the present invention discloses a terminal device status detection method, including:

[0005] Extracting a screen area image of at least one terminal device from the collected monitoring image;

[0006] Inputting the screen area image of the terminal device into a preset first state detection model;

[0007] The operating state of the terminal device is determined according to the first classification information output by the first state detection model; wherein the first classification information includes a normal screen state and an abnormal screen state.

[0008] Optionally, the step of determining the operating state of the terminal device according to the first classification information output by the first state detection model includes:

[0009] When the first classification information output by the first state detection model is an abnormal screen state, extracting a device environment image of the terminal device from the monitoring image;

[0010] Inputting the device environment image of the terminal device into a preset second state detection model;

[0011] The operating state of the terminal device is determined according to the second classification information output by the second state detection model; wherein the second classification information includes a person-blocked state and a person-unblocked state.

[0012] Optionally, when the first classification information output by the first state detection model is an abnormal screen state, the step of extracting a device environment image of the terminal device in the monitoring image includes:

[0013] When the first classification information output by the first state detection model is that the screen is in an abnormal state, determining the size of the device environment image based on a preset expansion ratio and the size of the screen area image;

[0014] The device environment image is extracted from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

[0015] Optionally, the method further comprises:

[0016] Using a preset frame extraction frequency, extract at least one frame of surveillance image from the collected surveillance video.

[0017] Optionally, the operating state of the terminal device includes a normal operating state and an abnormal operating state;

[0018] The method further comprises:

[0019] When it is determined in a preset number of consecutive monitoring images that the operating status of the terminal devices is abnormal, an alarm message is issued.

[0020] The embodiment of the present invention also discloses a method for generating a terminal device status detection model, comprising:

[0021] Acquire a screen area sample of a terminal device; wherein the screen area sample corresponds to first annotation information, and the first annotation information includes a normal screen state and an abnormal screen state;

[0022] The screen area samples and the first annotation information corresponding to the screen area samples are used to train a preset first model to be trained to generate a first state detection model.

[0023] Optionally, the method further comprises:

[0024] Acquire a device environment sample of the terminal device; wherein the device environment sample corresponds to second annotation information, and the second annotation information includes a person blocking state and a person not blocking state;

[0025] The device environment image sample and the second annotation information corresponding to the device environment image sample are used to train a preset second model to be trained to generate a second state detection model.

[0026] The embodiment of the present invention further discloses a terminal device status detection device, comprising:

[0027] A screen extraction module, used to extract a screen area image of at least one terminal device from the collected monitoring image;

[0028] A screen detection module, used for inputting a screen area image of the terminal device into a preset first state detection model;

[0029] A state determination module is used to determine the operating state of the terminal device according to the first classification information output by the first state detection model; wherein the first classification information includes a normal screen state and an abnormal screen state.

[0030] Optionally, the state determination module includes:

[0031] an environment extraction submodule, configured to extract a device environment image of the terminal device from the monitoring image when the first classification information output by the first state detection model is an abnormal screen state;

[0032] An environment detection submodule, used for inputting a device environment image of the terminal device into a preset second state detection model;

[0033] The state determination submodule is used to determine the operating state of the terminal device according to the second classification information output by the second state detection model; wherein the second classification information includes a person-blocked state and a person-unblocked state.

[0034] Optionally, the environment extraction submodule includes:

[0035] a size determination unit, configured to determine the size of the device environment image based on a preset expansion ratio and the size of the screen area image when the first classification information output by the first state detection model is an abnormal screen state;

[0036] The environment extraction unit is used to extract the device environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

[0037] Optionally, the device further comprises:

[0038] The image extraction module is used to extract at least one frame of monitoring image from the collected monitoring video using a preset frame extraction frequency.

[0039] Optionally, the operating state of the terminal device includes a normal operating state and an abnormal operating state;

[0040] The device also includes:

[0041] The alarm module is used to issue an alarm message when it is determined that the operating status of the terminal equipment is abnormal in a continuous preset number of monitoring images.

[0042] The embodiment of the present invention further discloses a device for generating a terminal device status detection model, comprising:

[0043] A first sample acquisition module, used to acquire a screen area sample of a terminal device; wherein the screen area sample corresponds to first annotation information, and the first annotation information includes a normal screen state and an abnormal screen state;

[0044] The first training module is used to train a preset first model to be trained by adopting the screen area samples and the first annotation information corresponding to the screen area samples to generate a first state detection model.

[0045] Optionally, the device further comprises:

[0046] A first sample acquisition module is used to acquire a device environment sample of a terminal device; wherein the device environment sample corresponds to second annotation information, and the second annotation information includes a person blocking state and a person not blocking state;

[0047] The second training module is used to train a preset second model to be trained by adopting the device environment image sample and the second annotation information corresponding to the device environment image sample to generate a second state detection model.

[0048] The embodiment of the present invention further discloses a device, including:

[0049] one or more processors; and

[0050] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the device to perform one or more methods as described in the embodiments of the present invention.

[0051] The embodiment of the present invention further discloses one or more machine-readable media on which instructions are stored, and when executed by one or more processors, the processors execute one or more methods as described in the embodiment of the present invention.

[0052] The embodiments of the present invention include the following advantages:

[0053] According to the terminal device status detection method of the embodiment of the present invention, the screen area image of at least one terminal device is extracted from the collected monitoring image, and the screen area image of the terminal device is input into a preset first status detection model, and the operation status of the terminal device is determined according to the first classification information output by the first status detection model. Therefore, the first status detection model can be used to detect whether the screen of the terminal device in the monitoring image is displayed normally, so as to determine whether the terminal device is operating normally. The monitoring image is used to automatically monitor the terminal device to ensure the normal operation of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of a method for detecting a terminal device status according to an embodiment of the present invention;

[0055] Figure 2 is a flowchart of another method for detecting the state of a terminal device according to an embodiment of the present invention;

[0056] Figure 3 is a flowchart of a method for generating a terminal device state detection model according to an embodiment of the present invention;

[0057] Figure 4 is a structural block diagram of a terminal device status detection device according to an embodiment of the present invention;

[0058] Figure 5 It is a structural block diagram of a device for generating a terminal device status detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Reference Figure 1 , shows a flowchart of a terminal device status detection method embodiment of the present invention, which may specifically include the following steps:

[0061] Step 101, extracting a screen area image of at least one terminal device from the acquired monitoring image;

[0062] In the embodiment of the present invention, the terminal device may be a terminal device installed in a public space and used to provide business processing functions for personnel, for example, an ATM installed in a bank, a self-service payment machine installed in a hospital, a self-service reporting machine installed in an airport, a self-service ticket purchasing machine installed in a railway station, etc.

[0063] In an embodiment of the present invention, the terminal device may be generally provided with a screen so as to display business-related information to personnel. Generally speaking, in the event of an abnormality in the terminal device, the screen of the terminal device generally cannot display information normally. For example, in the event of a power outage in the terminal device, the screen of the terminal device may be a black screen. In the event of a system abnormality in the terminal device, the screen of the terminal device may display abnormal prompt information with a blue background.

[0064] Therefore, a preset monitoring device can be used to monitor the area where the terminal device is installed, and the monitoring device can be set at a position where the screen position of the terminal device can be captured, so that it can be determined whether the terminal device is in a normal state based on the monitoring image collected by the monitoring device.

[0065] The monitoring image may include at least one terminal device, and an image of a screen area of ​​the terminal device in the monitoring image may be extracted to obtain an image of a screen area of ​​at least one terminal device in the monitoring image.

[0066] In a specific implementation, the screen area of ​​the terminal device in the monitoring image can be identified by target recognition using a pre-trained target recognition model, thereby extracting the screen area image in the monitoring image.

[0067] In a specific implementation, since the position of the monitoring device can be unchanged, the area that can be photographed by the monitoring device can also be unchanged. Therefore, it is also possible to record the coordinate information of the screen area of ​​the terminal device in the monitoring image in advance, and after acquiring the monitoring image, extract the image at the coordinate information in the monitoring image to obtain the screen area image of the terminal device.

[0068] Step 102, inputting the screen area image of the terminal device into a preset first state detection model;

[0069] In the embodiment of the present invention, the screen area image of the terminal device may be input into a preset first state detection model to detect the screen display state of the terminal device.

[0070] Among them, the first state detection model can be a pre-trained model, which can be used to classify the screen area image to distinguish the screen display state of the terminal device. Specifically, the first state detection model can classify the screen area image into a normal screen state in which the screen display of the terminal device is normal, and an abnormal screen state in which the screen display of the terminal device is abnormal. If the first state detection model classifies the screen area image as a normal screen state, it can be considered that the screen display of the terminal device is normal. If the first state detection model classifies the screen area image as an abnormal screen state, it can be considered that the screen display of the terminal device is abnormal.

[0071] In a specific implementation, a screen area sample with first annotation information may be used to train a preset first model to be trained to generate a first state detection model. The first annotation information may include a normal state of the screen and an abnormal state of the screen. The preset first model to be trained may be a convolutional neural network model, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, etc., and the present invention does not limit this.

[0072] Step 103: Determine the operating state of the terminal device according to the first classification information output by the first state detection model; wherein the first classification information includes a normal screen state and an abnormal screen state.

[0073] In an embodiment of the present invention, the first state detection model may output first classification information. The first classification information may include a normal screen state and an abnormal screen state. According to the first classification information output by the first state detection model, it can be known whether the screen display state of the terminal device is normal, and according to the screen display state, it can be determined whether the operating state of the terminal device is normal.

[0074] In a specific implementation, when the first classification information output by the first state detection model is a normal screen state, it can be considered that the screen display of the terminal device is normal, and the terminal device can be in a normal operating state.

[0075] When the first classification information output by the first state detection model is an abnormal screen state, it can be considered that the screen display of the terminal device is abnormal, and the terminal device may be in an abnormal operation state. At this time, the terminal device may fail, resulting in the screen being unable to display normally.

[0076] According to the terminal device status detection method of the embodiment of the present invention, the screen area image of at least one terminal device is extracted from the collected monitoring image, and the screen area image of the terminal device is input into a preset first status detection model, and the operation status of the terminal device is determined according to the first classification information output by the first status detection model. Therefore, the first status detection model can be used to detect whether the screen of the terminal device in the monitoring image is displayed normally, so as to determine whether the terminal device is operating normally. The monitoring image is used to automatically monitor the terminal device to ensure the normal operation of the terminal device.

[0077] Reference Figure 2 , shows a flowchart of a terminal device status detection method embodiment of the present invention, which may specifically include the following steps:

[0078] Step 201, extracting a screen area image of at least one terminal device from the collected monitoring image;

[0079] In an embodiment of the present invention, the terminal device may be a terminal device that is arranged in a public space and is used to provide business processing functions for personnel. The terminal device may usually be provided with a screen so as to display business-related information to personnel. Generally speaking, in the event of an abnormality in the terminal device, the screen of the terminal device usually cannot display information normally. For example, in the event of a power outage in the terminal device, the screen of the terminal device may correspond to a black screen. In the event of a system abnormality in the terminal device, the screen of the terminal device may correspond to displaying abnormal prompt information with a blue background.

[0080] Therefore, a preset monitoring device can be used to monitor the area where the terminal device is installed, and the monitoring device can be set at a position where the screen position of the terminal device can be captured, so that it can be determined whether the terminal device is in a normal state based on the monitoring image collected by the monitoring device.

[0081] The monitoring image may include at least one terminal device, and an image of a screen area of ​​the terminal device in the monitoring image may be extracted to obtain an image of a screen area of ​​at least one terminal device in the monitoring image.

[0082] In a specific implementation, the screen area of ​​the terminal device in the monitoring image can be identified by target recognition using a pre-trained target recognition model, thereby extracting the screen area image in the monitoring image.

[0083] In a specific implementation, since the position of the monitoring device can be unchanged, the area that can be photographed by the monitoring device can also be unchanged. Therefore, it is also possible to record the coordinate information of the screen area of ​​the terminal device in the monitoring image in advance, and after acquiring the monitoring image, extract the image at the coordinate information in the monitoring image to obtain the screen area image of the terminal device.

[0084] In one embodiment of the present invention, the method further comprises:

[0085] S11, using a preset frame extraction frequency, extracting at least one frame of monitoring image from the collected monitoring video.

[0086] In the embodiment of the present invention, the monitoring device may be a video acquisition device. The monitoring device may acquire a video of the area of ​​the terminal device to obtain a monitoring video. The monitoring video may be composed of multiple frames of monitoring images.

[0087] For the platform for managing monitoring devices, since the monitoring video contains a large number of monitoring images, it takes a lot of computing resources to detect the operating status of the terminal device for all monitoring images. In addition to monitoring the terminal device, the platform usually needs to handle other monitoring tasks, so the platform may be occupied by other monitoring tasks. Therefore, based on the processing power and idle resources of the platform for managing monitoring devices, a frame extraction frequency can be determined, and the preset frame extraction frequency can be used to extract at least one frame of monitoring image from the collected monitoring video for detecting the operating status of the terminal device, so that the real-time monitoring of the terminal device can be ensured to a certain extent, while reasonably controlling the computing resources consumed by detecting the operating status of the terminal device.

[0088] Among them, the frame extraction frequency can be determined according to actual needs, for example, extracting 4 frames every 1 second, extracting one frame of monitoring image every 5 seconds, extracting one frame of monitoring image every 1 minute, etc., and the present invention does not impose any limitation on this.

[0089] Step 202, inputting the screen area image of the terminal device into a preset first state detection model;

[0090] In the embodiment of the present invention, the screen area image of the terminal device may be input into a preset first state detection model to detect the screen display state of the terminal device.

[0091] Among them, the first state detection model can be a pre-trained model, which is used to classify the screen area image to distinguish the screen display state of the terminal device. Specifically, the first state detection model can classify the screen area image into a normal screen state in which the screen display of the terminal device is normal, and an abnormal screen state in which the screen display of the terminal device is abnormal. If the first state detection model classifies the screen area image as a normal screen state, it can be considered that the screen display of the terminal device is normal. If the first state detection model classifies the screen area image as an abnormal screen state, it can be considered that the screen display of the terminal device is abnormal.

[0092] In a specific implementation, a screen area sample with annotation information may be used to train a preset first model to be trained to generate a first state detection model. The annotation information may include a normal state of the screen and an abnormal state of the screen. The preset classification model may be a convolutional neural network model, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, etc., and the present invention does not limit this.

[0093] Step 203: when the first classification information output by the first state detection model is that the screen is in an abnormal state, extracting a device environment image of the terminal device from the monitoring image;

[0094] In an embodiment of the present invention, the first state detection model may output first classification information. The first classification information may include a normal screen state and an abnormal screen state. Whether the screen display state of the terminal device is normal can be known based on the first classification information output by the first state detection model.

[0095] In a specific implementation, when the first classification information output by the first state detection model is a normal screen state, it can be considered that the screen display of the terminal device is normal, and the terminal device can be in a normal operating state.

[0096] When the first classification information output by the first state detection model is an abnormal screen state, it can be considered that the screen display of the terminal device is abnormal, and the terminal device may be in an abnormal operation state. At this time, the terminal device may fail, resulting in the screen being unable to display normally.

[0097] In an embodiment of the present invention, when the first classification information output by the first state detection model is an abnormal screen state, it is possible that the screen of the terminal device does not display abnormalities, but is simply blocked by a person in the monitoring image due to a person operating the terminal device, thereby causing the classification information output by the first state detection model to be an abnormal screen state.

[0098] In order to avoid the situation where the first state detection model incorrectly judges the operating state of the terminal device, when the classified information output by the first state detection model is an abnormal screen state, it can be further determined whether the screen of the terminal device actually has a display abnormality.

[0099] Therefore, when the first classification information output by the first state detection model is an abnormal screen state, a device environment image including the terminal device itself and the surrounding environment of the terminal device can be extracted from the monitoring image, so as to use the device environment image to determine whether the terminal device is obscured by a person.

[0100] In an embodiment of the present invention, when the first classification information output by the first state detection model is an abnormal screen state, the step of extracting the device environment image of the terminal device in the monitoring image includes:

[0101] S21, when the first classification information output by the first state detection model is that the screen is in an abnormal state, determining the size of the device environment image based on a preset external expansion ratio and the size of the screen area image;

[0102] In an embodiment of the present invention, when the first classification information output by the first state detection model is an abnormal screen state, the size of the device environment image that may include the terminal device itself and the surrounding environment of the terminal device can be determined based on a preset expansion ratio and the size of the screen area image.

[0103] The outward expansion ratio can be determined according to actual needs to determine a ratio that allows the terminal device itself and the surrounding environment of the terminal device to be included in the device environment image, and the present invention does not impose any limitation on this.

[0104] S22: extracting the device environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

[0105] In an embodiment of the present invention, the device environment image of the terminal device can be extracted from the monitoring image based on the position of the screen area image in the monitoring image and the size of the device environment image, so as to use the device environment image to further determine whether the terminal device is blocked by a person.

[0106] In a specific implementation, the center coordinates of the screen area image can be used as the center coordinates of the device environment image, and based on the size of the device environment image, the device environment image can be extracted from the monitoring image, so that the obtained device environment image can include the terminal device itself and the surrounding environment of the terminal device.

[0107] Step 204, inputting the device environment image of the terminal device into a preset second state detection model;

[0108] In the embodiment of the present invention, the device environment image of the terminal device may be input into a preset second state detection model to detect whether the terminal device is blocked by a person.

[0109] Among them, the second state detection model can be a pre-trained model, which can be used to classify the device environment image to distinguish the surrounding environment state of the terminal device. Specifically, the second state detection model can classify the device environment image into a person-blocking state where a person blocks the terminal device, or a person-unblocking state where no person blocks the device, but there is an abnormality in the terminal device environment or the terminal device itself. If the second state detection model classifies the device environment image as a person-blocking state, it can be considered that the screen of the terminal device is blocked by a person, and the terminal device itself displays normally. If the second state detection model classifies the device image as a person-unblocking state, it can be considered that the screen of the terminal device is not blocked by a person, but there is an abnormality in the surrounding environment of the terminal device, or the screen display of the terminal device is abnormal.

[0110] In a specific implementation, a device environment sample with second annotation information can be used to train the preset second model to be trained to generate a second state detection model. The second annotation information may include a personnel occlusion state and a personnel unocclusion state. The preset second model to be trained may be a convolutional neural network model, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, etc., and the present invention does not limit this.

[0111] Step 205: Determine the operating state of the terminal device according to the second classification information output by the second state detection model; wherein the second classification information includes a person-blocked state and a person-unblocked state.

[0112] In an embodiment of the present invention, the second state detection model can output second classification information, and the second classification information can include a person blocking state and a person not blocking state. According to the second classification information output by the second state model, it can be known whether the terminal device is blocked by a person, thereby determining whether the terminal device is indeed in an abnormal operating state.

[0113] In a specific implementation, if the second classification information output by the second state detection model is a person-blocked state, it can be considered that the terminal device is blocked by a person. In this case, the screen of the terminal device may display normally, but it is classified as an abnormal screen state by the first state detection model only because the terminal device is blocked by a person. At this time, it can be considered that the terminal device is still in a normal operating state.

[0114] If the second classification information output by the second state detection model is a state where a person is not blocking the terminal device, it can be considered that the terminal device is not blocked by a person. In this case, the terminal device may be classified as a screen abnormality state by the first detection classification model due to abnormal screen display. At this time, it can be considered that the terminal device cannot display the screen normally due to a fault, and the terminal device may be in an abnormal operation state.

[0115] If the second classification information output by the second state detection model is a state in which a person is not blocking the terminal device, the terminal device may also be classified as an abnormal screen state by the first detection classification model due to an abnormal surrounding environment. At this time, the screen of the terminal device may be in a normal screen state or an abnormal screen state, but because the surrounding environment of the terminal device may be abnormal, causing the person to be unable to use the terminal device normally, it can be considered that the terminal device is also in an abnormal operating state. For example, if the terminal device is blocked by other objects other than people, causing the person to be unable to use the terminal device, it can be considered that the surrounding environment of the terminal device is abnormal at this time, and the terminal device is in an abnormal operating state.

[0116] In one embodiment of the present invention, the method further comprises:

[0117] S31, when it is determined in a preset number of consecutive monitoring images that the operating status of the terminal devices is abnormal, an alarm message is issued.

[0118] In an embodiment of the present invention, the operation state of the terminal device may include a normal operation state and an abnormal operation state. Wherein, when the screen of the terminal device is in a normal screen state, or when the terminal device is in a state of being blocked by a person, the terminal device may be considered to be in a normal operation state. When the screen of the terminal device is in an abnormal screen state, or when the screen of the terminal device is in an abnormal screen state and the terminal device is in a state of not being blocked by a person, the terminal device may be considered to be in an abnormal operation state.

[0119] In an embodiment of the present invention, in a preset number of surveillance images that are continuous in time sequence, if it is determined that the operating status of the terminal devices are all abnormal operating statuses, it can be considered that there is a high possibility that the terminal devices have indeed failed, so that an alarm message can be sent to the maintenance personnel, prompting the maintenance personnel to maintain the terminal devices in a timely manner.

[0120] The preset number may be determined according to actual needs, for example, 3 consecutive surveillance images, 5 consecutive surveillance images, 10 consecutive surveillance images, etc., and the present invention does not impose any limitation on this.

[0121] According to the terminal device status detection method of the embodiment of the present invention, by extracting the screen area image of at least one terminal device from the collected monitoring image, and inputting the screen area image of the terminal device into a preset first status detection model, when the first classification information output by the first status detection model is the abnormal screen status, extracting the device environment image of the terminal device from the monitoring image, inputting the device environment image of the terminal device into a preset second status detection model, and determining the operating status of the terminal device according to the second classification information output by the second status detection model. Thus, the first status detection model can be used to detect whether the screen of the terminal device in the monitoring image is displayed normally, so as to determine whether the terminal device is operating normally. When the first status detection model detects that the screen of the terminal device is abnormally displayed, the second status detection model can be further used to determine whether the terminal device is blocked by a person, so as to further determine whether the terminal device is operating normally, and further improve the accuracy of detecting the abnormal operating status of the terminal device. The monitoring image is used to automatically monitor the terminal device to ensure the normal operation of the terminal device.

[0122] Reference Figure 3 , shows a flowchart of a method for generating a terminal device state detection model according to an embodiment of the present invention, which may specifically include the following steps:

[0123] Step 301, obtaining a screen area sample of a terminal device; wherein the screen area sample corresponds to first annotation information, and the first annotation information includes a normal screen state and an abnormal screen state;

[0124] In an embodiment of the present invention, in order to train a preset first model to be trained, an image of a screen area of ​​a terminal device may be obtained as a screen area sample. The screen area sample may have manually annotated first annotation information. The first annotation information may include a normal screen state and an abnormal screen state.

[0125] The first annotation information is a sample of the screen area in a normal state, which can be an image of the screen area of ​​the terminal device when the terminal device is operating normally. The first annotation information is a sample of the screen area in an abnormal state, which can be an image of the screen area of ​​the terminal device when the terminal device is operating abnormally.

[0126] Optionally, since there may be a situation where the screen area is displayed normally, but in the screen area sample, the screen of the terminal device is partially blocked by a person, a certain number of samples where the screen area is normal but the screen of the terminal device is partially blocked by a person can be appropriately added to the screen area samples whose first annotation information is the screen normal state. This can improve the classification accuracy to a certain extent.

[0127] Step 302: Use the screen area samples and the first annotation information corresponding to the screen area samples to train a preset first model to be trained to generate a first state detection model.

[0128] In the embodiment of the present invention, the screen area samples and the first annotation information corresponding to the screen area samples may be used to train a preset first model to be trained to generate a first state detection model.

[0129] Among them, the first model to be trained can be a convolutional neural network model such as an EfficientNet network, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, etc., and the present invention is not limited to this.

[0130] In a specific implementation, a preset number of screen area samples whose first annotation information is a normal screen state and a preset number of screen area samples whose first annotation information is an abnormal screen state can be collected. The screen area samples are normalized into images with the same size and the same number of channels. For example, the screen area samples are normalized into a color image with a size of 300×300 and a channel number of 3.

[0131] Optionally, the screen area samples may be adjusted in brightness and definition to obtain more screen area samples under different circumstances.

[0132] Thereafter, the screen area sample may be used as the input of the model, and the first annotation information may be used as the output of the model to train the preset first model to be trained, thereby generating a first state detection model.

[0133] As an example of the present invention, when the first model to be trained is a convolutional neural network model, a deep learning framework such as TensorFlow framework can be used to construct the first model to be trained, and the screen area sample is used as the input of the model, and the first annotation information is used as the output of the model to train the first model to be trained. During the training process, the initial learning rate can be set to 0.001, and the learning rate is decayed by one tenth after each preset round, and the adaptive moment estimation (Adam) optimizer is used to adjust the model parameters to make the model converge, and finally the first model to be trained is obtained as the first state detection model.

[0134] In one embodiment of the present invention, the method further comprises:

[0135] S41, obtaining a device environment sample of a terminal device; wherein the device environment sample corresponds to second annotation information, and the second annotation information includes a person blocking state and a person not blocking state;

[0136] In an embodiment of the present invention, when the first classification information output by the first state detection model is an abnormal screen state, it is possible that the screen of the terminal device does not display abnormalities, but is simply blocked by a person in the monitoring image due to a person operating the terminal device, thereby causing the classification information output by the first state detection model to be an abnormal screen state.

[0137] In order to avoid the situation where the first state detection model incorrectly judges the operating state of the terminal device, a second state detection model can be trained. When the first classification information output by the first state detection model is an abnormal screen state, the second state detection model can be used to detect the device environment image to identify whether there is a person partially blocking the terminal device or a person completely blocking the terminal device.

[0138] Thus, an image including the terminal device itself and the surrounding environment of the terminal device is obtained as a device environment sample of the terminal device to train the second model to be trained and obtain a second state detection model.

[0139] In an embodiment of the present invention, the device environment sample may have manually annotated second annotation information. The second annotation information may include a person blocking state and a person not blocking state.

[0140] The second annotation information is a device environment sample in a state of being blocked by a person, which may be a device environment image in which the terminal device is operating normally and the terminal device is partially blocked by a person or completely blocked by a person. The second annotation information is a device environment sample in a state of being not blocked by a person, which may be a device environment image in which the terminal device is operating abnormally, and a device environment image in which the terminal device is operating abnormally and the terminal device is partially blocked or completely blocked by other objects other than a person.

[0141] S42: Use the device environment image sample and the second annotation information corresponding to the device environment image sample to train a preset second model to be trained to generate a second state detection model.

[0142] In the embodiment of the present invention, the device environment image sample and the second annotation information corresponding to the device environment image sample may be used to train a preset second model to be trained to generate a second state detection model.

[0143] Among them, the second model to be trained can be a convolutional neural network model such as an EfficientNet network, a perceptron model, a random forest model, a support vector machine model, a K-nearest neighbor algorithm model, etc., and the present invention is not limited to this.

[0144] In a specific implementation, a preset number of device environment samples whose second annotation information is a state of being blocked by a person and a preset number of device environment samples whose second annotation information is a state of being not blocked by a person may be collected. The device environment samples are normalized into images of the same size and the same number of channels. For example, the device environment samples are normalized into a color image of a size of 300×300 and a number of channels of 3.

[0145] Optionally, the device environment samples may be adjusted in brightness and definition to obtain more device environment samples under different circumstances.

[0146] Thereafter, the device environment sample may be used as the input of the model, and the second annotation information may be used as the output of the model to train the preset second model to be trained, thereby generating a second state detection model.

[0147] As an example of the present invention, when the second model to be trained is a convolutional neural network model, a deep learning framework such as TensorFlow framework can be used to construct the second model to be trained, and the device environment sample is used as the input of the model, and the second annotation information is used as the output of the model to train the second model to be trained. During the training process, the initial learning rate can be set to 0.001, and the learning rate decays by one tenth after each preset round, and the adaptive moment estimation (Adam) optimizer is used to adjust the model parameters to make the model converge, and finally the trained second model to be trained is obtained as the second state detection model.

[0148] Through the method for generating a terminal device state detection model according to an embodiment of the present invention, the screen area sample and the first annotation information corresponding to the screen area sample are used to train a preset first model to be trained, and a first state detection model is generated. Thus, the first state detection model can be used to detect whether the screen of the terminal device is displayed normally, and determine whether the terminal device is operating normally according to the display state of the device screen. At the same time, the device environment image sample and the second annotation information corresponding to the device environment image sample are used to train a preset second model to be trained, and a second state detection model is generated. Thus, when the first state detection model detects that the terminal device is in an abnormal screen state, the second state detection model can be used to detect whether the terminal device is blocked by a person, thereby further improving the accuracy of detecting whether the terminal device is operating normally.

[0149] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0150] Reference Figure 4 , shows a structural block diagram of an embodiment of a terminal device status detection device of the present invention, which may specifically include the following modules:

[0151] The screen extraction module 401 is used to extract a screen area image of at least one terminal device from the collected monitoring image;

[0152] The screen detection module 402 is used to input the screen area image of the terminal device into a preset first state detection model;

[0153] The state determination module 403 is used to determine the operating state of the terminal device according to the first classification information output by the first state detection model; wherein the first classification information includes a normal screen state and an abnormal screen state.

[0154] Optionally, the state determination module 403 includes:

[0155] an environment extraction submodule, configured to extract a device environment image of the terminal device from the monitoring image when the first classification information output by the first state detection model is an abnormal screen state;

[0156] An environment detection submodule, used for inputting a device environment image of the terminal device into a preset second state detection model;

[0157] The state determination submodule is used to determine the operating state of the terminal device according to the second classification information output by the second state detection model; wherein the second classification information includes a person-blocked state and a person-unblocked state.

[0158] In one embodiment of the present invention, the environment extraction submodule includes:

[0159] a size determination unit, configured to determine the size of the device environment image based on a preset expansion ratio and the size of the screen area image when the first classification information output by the first state detection model is an abnormal screen state;

[0160] The environment extraction unit is used to extract the device environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

[0161] In one embodiment of the present invention, the device further comprises:

[0162] The image extraction module is used to extract at least one frame of monitoring image from the collected monitoring video using a preset frame extraction frequency.

[0163] In one embodiment of the present invention, the operating state of the terminal device includes a normal operating state and an abnormal operating state;

[0164] The device also includes:

[0165] The alarm module is used to issue an alarm message when it is determined that the operating status of the terminal equipment is abnormal in a continuous preset number of monitoring images.

[0166] Reference Figure 5 , shows a structural block diagram of an embodiment of a device for generating a terminal device state detection model of the present invention, which may specifically include the following modules:

[0167] A first sample acquisition module 501 is used to acquire a screen area sample of a terminal device; wherein the screen area sample corresponds to first annotation information, and the first annotation information includes a normal screen state and an abnormal screen state;

[0168] The first training module 502 is used to train a preset first to-be-trained model using the screen area samples and the first annotation information corresponding to the screen area samples to generate a first state detection model.

[0169] In one embodiment of the present invention, the device further comprises:

[0170] A first sample acquisition module is used to acquire a device environment sample of a terminal device; wherein the device environment sample corresponds to second annotation information, and the second annotation information includes a person blocking state and a person not blocking state;

[0171] The second training module is used to train a preset second model to be trained by adopting the device environment image sample and the second annotation information corresponding to the device environment image sample to generate a second state detection model.

[0172] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] The embodiment of the present invention further provides a device, including:

[0174] one or more processors; and

[0175] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the apparatus to perform the method described in the embodiment of the present invention.

[0176] The embodiment of the present invention further provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the method described in the embodiment of the present invention.

[0177] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0178] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0179] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0182] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0183] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0184] The above is a detailed introduction to a terminal device status detection method, a terminal device status detection model generation method, a terminal device status detection device, and a terminal device status detection model generation device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A terminal device status detection method, characterized in that: include: Extracting a screen area image of at least one terminal device from the collected monitoring image; Inputting the screen area image of the terminal device into a preset first state detection model; Determine the operating state of the terminal device according to the first classification information output by the first state detection model; wherein the first classification information includes a normal screen state and an abnormal screen state; The step of determining the operating state of the terminal device according to the first classification information output by the first state detection model comprises: When the first classification information output by the first state detection model is an abnormal screen state, extracting a device environment image of the terminal device from the monitoring image; When the first classification information output by the first state detection model is an abnormal screen state, the step of extracting the device environment image of the terminal device in the monitoring image comprises: When the first classification information output by the first state detection model is that the screen is in an abnormal state, determining the size of the device environment image based on a preset expansion ratio and the size of the screen area image; Extracting the device environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image; Inputting the device environment image of the terminal device into a preset second state detection model; The operating state of the terminal device is determined according to the second classification information output by the second state detection model; wherein the second classification information includes a person-blocked state and a person-unblocked state.

2. The method according to claim 1, characterized in that The method further comprises: Using a preset frame extraction frequency, extract at least one frame of surveillance image from the collected surveillance video.

3. The method according to any one of claims 1 to 2, characterized in that The operating state of the terminal device includes a normal operating state and an abnormal operating state; The method further comprises: When it is determined in a preset number of consecutive monitoring images that the operating status of the terminal devices is abnormal, an alarm message is issued.

4. A method for generating a terminal device status detection model, characterized in that: include: Acquire a screen area image sample of a terminal device; wherein the screen area image sample corresponds to first annotation information, and the first annotation information includes a normal screen state and an abnormal screen state; Using the screen area image samples and the first annotation information corresponding to the screen image area samples, training a preset first to-be-trained model to generate a first state detection model; Acquire a device environment image sample of the terminal device; wherein the device environment image sample corresponds to second annotation information, and the second annotation information includes a person blocking state and a person not blocking state; Using the device environment image sample and the second annotation information corresponding to the device environment image sample, training a preset second to-be-trained model to generate a second state detection model; The step of obtaining a device environment image sample of a terminal device includes: When the first classification information output by the first state detection model is that the screen is in an abnormal state, determining the size of the device environment image based on a preset expansion ratio and the size of the screen area image; The device environment image is extracted from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

5. A terminal device status detection device, characterized in that: include: A screen extraction module, used to extract a screen area image of at least one terminal device from the collected monitoring image; A screen detection module, used for inputting a screen area image of the terminal device into a preset first state detection model; A state determination module, configured to determine the operating state of the terminal device according to first classification information output by the first state detection model; wherein the first classification information includes a normal screen state and an abnormal screen state; The state determination module comprises: an environment extraction submodule, configured to extract a device environment image of the terminal device from the monitoring image when the first classification information output by the first state detection model is an abnormal screen state; An environment detection submodule, used for inputting a device environment image of the terminal device into a preset second state detection model; A state determination submodule, configured to determine the operating state of the terminal device according to the second classification information output by the second state detection model; wherein the second classification information includes a person-blocked state and a person-unblocked state; The environment extraction submodule includes: a size determination unit, configured to determine the size of the device environment image based on a preset expansion ratio and the size of the screen area image when the first classification information output by the first state detection model is an abnormal screen state; The environment extraction unit is used to extract the device environment image from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

6. A device for generating a terminal device status detection model, characterized in that: include: A first sample acquisition module, configured to acquire a screen area image sample of a terminal device; wherein the screen area image sample corresponds to first annotation information, and the first annotation information includes a normal screen state and an abnormal screen state; A first training module, used to train a preset first to-be-trained model using the screen area image sample and the first annotation information corresponding to the screen area image sample to generate a first state detection model; A first sample acquisition module is used to acquire a device environment image sample of a terminal device; wherein the device environment image sample corresponds to second annotation information, and the second annotation information includes a person blocking state and a person unblocking state; A second training module, used to train a preset second to-be-trained model using the device environment image sample and the second annotation information corresponding to the device environment image sample to generate a second state detection model; The step of obtaining a device environment image sample of a terminal device includes: When the first classification information output by the first state detection model is that the screen is in an abnormal state, determining the size of the device environment image based on a preset expansion ratio and the size of the screen area image; The device environment image is extracted from the monitoring image according to the position of the screen area image in the monitoring image and the size of the device environment image.

7. A device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the apparatus to perform any method as claimed in claim 1 or 4.

8. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the processors to perform any of the methods of claims 1-4.

Images