State detection method, device, electronic device and storage medium

Abstract

The present invention provides a state detection method, device, electronic device, and storage medium, wherein the method comprises: determining an image to be detected; adaptively locating a state-related region in the image to be detected through spatial transformation based on a state detection model, and performing person state detection on the image to be detected using the state-related region; the state detection model is trained based on sample images and person state category labels of the sample images. The method, device, electronic device, and storage medium provided by the present invention can adaptively locate a state-related region in the image to be detected by spatially transforming features of the image to be detected, and then perform person state detection on the image to be detected using the state-related region. This achieves the goal of using the state-related region as a detection target and obtaining a state-related region in the image to be detected, thereby reducing the problem of erroneous subsequent state category detection results caused by fixed area detection and improving the accuracy of state category detection.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a state detection method, device, electronic equipment and storage medium. Background Art

[0002] Driver status monitoring is a crucial component of assisted driving systems. It aims to alert the driver to potential dangers and mitigate them. Driver status monitoring primarily detects driver distractions, such as smoking, talking on the phone, or fatigue. When these conditions are detected, the assisted driving system issues a warning via voice or warning lights.

[0003] Existing driver status detection primarily relies on identifying fixed ROIs (regions of interest) within an image, such as the eyes, mouth, and hands, and then using local features within those ROIs to detect the driver's status. However, using a keypoint detection mechanism that is unrelated to the driver's status can lead to erroneous ROI recognition and result in incorrect driver status classification. It can also result in correct ROI recognition but still lead to incorrect driver status classification due to a lack of ROI features. Summary of the Invention

[0004] The present invention provides a state detection method, device, electronic device and storage medium, which are used to solve the defect in the prior art that personnel state category detection through ROI easily leads to erroneous detection results.

[0005] The present invention provides a state detection method, comprising:

[0006] Determine an image to be detected;

[0007] Based on the state detection model, adaptively locating the state-related area in the image to be detected through spatial transformation, and performing person state detection on the image to be detected through the state-related area;

[0008] The state detection model is trained based on sample images and the person state category labels of the sample images.

[0009] According to a state detection method provided by the present invention, based on the state detection model, the state-related area in the image to be detected is adaptively located through spatial transformation, and the state detection of the person in the image to be detected is performed using the state-related area, including:

[0010] Based on the state localization network in the state detection model, adaptively locate the state-related area in the image to be detected through spatial transformation to obtain a state localization feature, and determine the image feature of the state-related area through the state localization feature and the convolution feature of the image to be detected, wherein the state localization feature is used to indicate the position of the state-related area in the image to be detected;

[0011] Based on the classification network in the state detection model, the image features of the state-related area are applied to perform personnel state detection on the image to be detected.

[0012] According to a state detection method provided by the present invention, the state localization network in the state detection model adaptively locates the state-related area in the image to be detected through spatial transformation to obtain a state localization feature, and determines the image feature of the state-related area through the state localization feature and the convolution feature of the image to be detected, including:

[0013] Based on the multi-layer convolutional network in the state localization network, feature extraction is performed on the image to be detected to obtain convolution features of each convolution output in the multi-layer convolutional network;

[0014] Based on the spatial transformation network in the state localization network, the convolution features of the current layer convolution output and the spatial transformation features obtained by the spatial transformation of the previous layer are applied to perform spatial transformation to obtain the spatial transformation features of the current layer spatial transformation until the spatial transformation features of the last layer are obtained, and the spatial transformation features of the last layer are determined as the state localization features. The image features of the state-related area are determined by the state localization features and the convolution features output by the last layer convolution.

[0015] According to a state detection method provided by the present invention, the convolution feature output by the current layer convolution and the spatial transformation feature obtained by the spatial transformation of the previous layer are applied to perform spatial transformation to obtain the spatial transformation feature of the current layer spatial transformation, including:

[0016] The convolution features of the current layer convolution output are fused with the state positioning features obtained by the spatial transformation of the previous layer to obtain the fused features corresponding to the current layer convolution, and the fused features corresponding to the current layer convolution are spatially transformed to obtain the state positioning features of the current layer spatial transformation.

[0017] According to a state detection method provided by the present invention, based on the classification network in the state detection model, the image features of the state-related area are applied to perform person state detection on the image to be detected, including:

[0018] Based on the illumination perception network in the classification network, performing illumination intensity equalization on the image features of the state-related area to obtain equalized features of the state-related area;

[0019] Based on the state classification network in the classification network, the balanced features of the state-related area are applied to perform personnel state detection on the image to be detected.

[0020] According to a state detection method provided by the present invention, the light perception network in the classification network performs light intensity equalization on the image features of the state-related area to obtain the equalized features of the state-related area, including:

[0021] Based on the strong light perception network branch and the weak light perception network branch in the light perception network, respectively extracting light features from the image features of the state-related area to obtain strong light features of the state-related area and weak light features of the state-related area;

[0022] Based on the weight fusion branch in the light perception network, the image features of the state-related area are predicted to obtain a light intensity weight value; and based on the light intensity weight value, the strong light perception feature and the weak light perception feature are weighted to obtain a balanced feature of the state-related area.

[0023] According to a state detection method provided by the present invention, the state detection model is trained based on the following steps:

[0024] Determine an initial detection model; the initial detection model includes an initial state positioning network, an initial light perception network and an initial state classification network;

[0025] Training an initial detection model based on the sample images and the person status category labels of the sample images to obtain the status positioning network and the light perception network;

[0026] Determining an intermediate detection model; the intermediate detection model includes the state positioning network, the light perception network, and the initial state classification network;

[0027] The parameters of the state localization network and the light perception network are fixed, and the intermediate detection model is trained based on the sample image and the person state category label of the sample image to obtain the state classification network.

[0028] According to a state detection method provided by the present invention, the parameters of the state localization network and the light sensing network are fixed, and the intermediate detection model is trained based on the sample image and the person state category label of the sample image to obtain the state classification network, including:

[0029] Determining the weight of each personnel status category based on the sample data volume of the sample images under each personnel status category;

[0030] Loss calculation is performed based on the state classification result output by the intermediate detection model based on the sample image, the personnel state category label of the sample image and the weight of each personnel state category, and the parameters of the initial classification network are updated based on the loss calculation result until the loss calculation result converges to obtain the state classification network.

[0031] The present invention also provides a state detection device, comprising:

[0032] A determination module, used for determining an image to be detected;

[0033] a detection module, configured to adaptively locate a state-related region in the image to be detected through spatial transformation based on a state detection model, and perform person state detection on the image to be detected based on the state-related region;

[0034] The state detection model is trained based on sample images and the person state category labels of the sample images.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-described state detection methods when executing the program.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above-mentioned state detection methods when executed by a processor.

[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned state detection methods are implemented.

[0038] The state detection method, device, electronic device and storage medium provided by the present invention perform spatial transformation on the features of the image to be detected, adaptively locate the area related to the state in the image to be detected, and then perform personnel state detection on the image to be detected by using the state-related area. This realizes the use of the state-related area as the detection target and obtains the area related to the state in the image to be detected, reduces the problem of erroneous results in subsequent personnel state category detection caused by fixed area detection, and improves the accuracy of personnel state category detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is one of the flow charts of the state detection method provided by the present invention;

[0041] Figure 2 It is a schematic diagram of the process of detecting the state of the state detection model provided by the present invention;

[0042] Figure 3 It is a flow chart of the state positioning feature acquisition method provided by the present invention;

[0043] Figure 4 This is the second flow chart of the state detection method provided by the present invention;

[0044] Figure 5 1 is a flow chart of the method for obtaining the equalization feature provided by the present invention;

[0045] Figure 6 It is a flow chart of the state detection model training method provided by the present invention;

[0046] Figure 7 It is a flow chart of the training method of the state classification network provided by the present invention;

[0047] Figure 8 It is a network framework diagram of the state detection model provided by the present invention;

[0048] Figure 9 It is the STN network structure diagram provided by the present invention;

[0049] Figure 10 This is a network framework diagram of the light perception network provided by the present invention;

[0050] Figure 11 It is a structural schematic diagram of the status detection device provided by the present invention;

[0051] Figure 12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] Currently, driver status detection primarily relies on face detection or designated area detection, followed by driver status detection within the detected ROI region. For example, this involves detecting the eye and mouth ROI regions within the face area of ​​the image being detected, or detecting the hand ROI region. However, this state detection method doesn't directly target the driver's status, and errors in ROI detection can lead to erroneous state detection results. Even if the ROI detection is correct, a lack of features within the ROI region can still result in a high error rate in the state detection result.

[0054] Therefore, how to directly use status as the detection target to improve the accuracy of personnel status category detection results is a technical problem that needs to be solved urgently in this field.

[0055] In response to the above technical problems, an embodiment of the present invention provides a state detection method. Figure 1 This is one of the flow charts of the state detection method provided by the present invention. Figure 1 As shown, the method can be applied to the driver status detection scenario. The following embodiments are all described in the driver status detection scenario. In addition, the method can also be applied to similar scenarios such as personnel status detection, such as: student listening status detection scenario, assembly line worker working status detection scenario, etc. The method includes:

[0056] Step 110: Determine the image to be detected.

[0057] Specifically, the image to be detected may be a current image containing the driver captured in real time by a camera, or may be a frame of image in a video image containing the driver, which is not limited in the embodiment of the present invention.

[0058] Step 120: Based on the state detection model, adaptively locate the state-related area in the image to be detected through spatial transformation, and perform person state detection on the image to be detected using the state-related area;

[0059] The state detection model is trained based on sample images and the person state category labels of the sample images.

[0060] In order to avoid a series of problems that may arise from collecting fixed areas of interest for personnel status detection, in an embodiment of the present invention, the status-related areas are adaptively located according to the characteristics of the image to be detected itself, thereby performing personnel status detection. Here, the status-related areas are image areas that are related to the personnel status for the image to be detected itself. For example, in an image to be detected where a person is dozing off, the hand area may not be the focus, while the eye area is the status-related area that needs to be located. For example, in the eye-rubbing action, the area where the hands and eyes overlap is the status-related area that needs to be located. That is, the status-related areas in different images may be different. By adaptively locating the status-related areas in the image to be detected, that is, locating the status-related areas with status detection as the goal guide and performing personnel status detection accordingly, the status-related areas can be accurately located.

[0061] To achieve adaptive positioning of state-related areas, embodiments of the present invention introduce spatial transformation technology in the process of applying a state detection model to perform personnel state detection. Specifically, in the state detection model, spatial transformation technology can be implemented in the form of a spatial transformation network. In the state detection model obtained by training sample images and the personnel state category labels of the sample images, the spatial transformation network can learn how to locate areas useful for personnel state detection from the input image. In other words, the spatial transformation network has the ability to adaptively locate state-related image areas. Therefore, personnel state detection can be performed on the relevant areas of the image to be detected that are adaptively located by spatial transformation.

[0062] The personnel status detection result thus obtained may be the probability of each personnel status category, or may directly output the personnel status category, and determine whether to issue an early warning reminder based on the status detection result. This embodiment of the present invention does not impose any limitation on this.

[0063] Before executing step 120, it is necessary to pre-train a state detection model. Specifically, when training the state detection model, the sample images can be input into the model being trained respectively, so as to obtain the personnel state detection results output by the model for the sample images. On this basis, the personnel state detection results are compared with the sample images and the personnel state category labels of the sample images to obtain the loss value of the model training. The model parameters are iteratively updated based on the loss value. In this process, the model can learn the correspondence between the sample images and the personnel state detection results, so that the trained state detection model can have the ability to perform personnel state detection on the relevant areas of the image to be detected through adaptive positioning of spatial transformation.

[0064] It should be noted that the spatial transformation in the state detection model can be one or more times, and the embodiments of the present invention do not limit this. Among them, the located state-related area can be one or more areas, and the person state detection of the image to be detected through the state-related area can be performed directly on the local area features of the state-related area. It can also be performed based on the local features of the state-related area and the image features determined by the convolution features of the image to be detected. The embodiments of the present invention do not limit this.

[0065] The state detection method provided by the embodiment of the present invention performs spatial transformation on the features of the image to be detected, adaptively locates the area related to the state in the image to be detected, and then performs personnel state detection on the image to be detected by using the state-related area. This realizes using the state-related area as the detection target and obtaining the area related to the state in the image to be detected, thereby reducing the problem of erroneous results in subsequent personnel state category detection caused by fixed area detection and improving the accuracy of personnel state category detection.

[0066] Based on the above embodiments, Figure 2 This is a flow chart of the state detection model provided by the present invention. Figure 2 As shown, step 120 includes:

[0067] Step 121, based on the state localization network in the state detection model, adaptively locate the state-related area in the image to be detected through spatial transformation to obtain a state localization feature, and determine the image feature of the state-related area through the state localization feature and the convolution feature of the image to be detected. The state localization feature is used to indicate the position of the state-related area in the image to be detected.

[0068] Step 122 : Based on the classification network in the state detection model, the image features of the state-related area are used to perform person state detection on the image to be detected.

[0069] Specifically, the image to be detected is input into the state detection model. The state localization network in the state detection model detects the state-related area in the image to be detected through spatial transformation, and obtains the state localization feature used to indicate the position of the state-related area in the image to be detected. Considering that the subsequent classification network can perform state detection from the perspective of global features, and thus can obtain the state detection result more accurately, the state localization feature and the convolution feature of the image to be detected are fused here to obtain the image feature of the state-related area. Then, through the classification network in the state detection model, the image feature of the state-related area is applied to perform personnel state detection on the image to be detected, and the personnel state detection result can be obtained. Among them, the convolution feature of the image to be detected refers to the feature that can reflect the overall image to be detected that is finally obtained by the convolution network.

[0070] Based on the above embodiments, Figure 3 FIG. 1 is a flow chart of the state positioning feature acquisition method provided by the present invention. Figure 3 As shown, step 121 includes:

[0071] Step 310: Based on the multi-layer convolutional network in the state localization network, feature extraction is performed on the image to be detected to obtain convolution features of each convolution output layer in the multi-layer convolutional network;

[0072] Step 320, based on the spatial transformation network in the state localization network, applies the convolution features of the current layer convolution output and the spatial transformation features obtained by the spatial transformation of the previous layer to perform spatial transformation, obtains the spatial transformation features of the current layer spatial transformation, until the spatial transformation features of the last layer are obtained, and the spatial transformation features of the last layer are determined as the state localization features, and the image features of the state-related area are determined by the state localization features and the convolution features output by the last layer convolution.

[0073] Considering that as the convolutional layers of a convolutional network gradually deepen, the resolution of deep convolutional features decreases, resulting in the loss of subtle information in the image, while shallow convolutional features, while containing rich details, lack contextual mutual information, i.e., have low semantics, the embodiments of the present invention obtain state localization features in a step-by-step localization manner through multiple spatial transformations of convolutional layers of different depths.

[0074] Specifically, in step 310, in the multi-layer convolutional network in the state localization network, each convolutional layer performs feature extraction based on the convolutional features output by the previous convolutional layer to obtain the convolutional features corresponding to each convolutional layer. Among them, the first convolutional layer in the multi-layer convolutional network performs feature extraction on the image to be detected to obtain the convolutional features of the first convolutional layer. The convolutional layer here is a virtual convolutional layer, and each convolutional layer can contain one or more physical convolutional layers. The number of physical convolutional layers contained in each layer of the virtual convolutional layer can be the same or different, and this is not limited in the embodiment of the present invention.

[0075] In step 320, the spatial transformation network in the state localization network performs spatial transformation on each convolutional layer from shallow to deep layers based on the convolutional features of each convolutional layer obtained in step 310, applies the convolutional features output by the current convolutional layer and the spatial transformation features obtained by the spatial transformation of the previous layer to perform spatial transformation to obtain the spatial transformation features of the current convolutional layer, and then uses the next convolutional layer as the current layer to perform the above operation until the spatial transformation features of the last convolutional layer are obtained, and the spatial transformation operation is terminated. The spatial transformation features of the last convolutional layer are used as the state localization features obtained by the state detection model, and the state localization features are fused with the convolutional features output by the last convolutional layer to obtain the image features of the state-related area. Among them, the shallow and deep layers of the convolutional layers in the convolutional network are determined by the order in which the features pass through the convolutional layers. The convolutional layer whose features are input first is the shallow layer of the convolutional layer input later, and vice versa.

[0076] It should be noted that the spatial transformation features of the current layer can be obtained by fusion of the convolution features output by the current convolution layer and the spatial transformation features obtained by the spatial transformation of the previous layer, and spatial transformation of the obtained fusion features. In addition, when the current convolution layer is the first convolution layer, that is, there is no spatial transformation feature obtained by the transformation of the previous convolution layer, then the convolution features output by the first convolution layer are directly spatially transformed to obtain the spatial transformation features corresponding to the first convolution layer.

[0077] Based on the above embodiment, in step 320, the convolution feature output by the convolution of the current layer and the spatial transformation feature obtained by the spatial transformation of the previous layer are applied to perform spatial transformation to obtain the spatial transformation feature of the spatial transformation of the current layer, including:

[0078] The convolution features of the current layer convolution output are fused with the state positioning features obtained by the spatial transformation of the previous layer to obtain the fused features corresponding to the current layer convolution, and the fused features corresponding to the current layer convolution are spatially transformed to obtain the state positioning features of the current layer spatial transformation.

[0079] Considering that shallow features contain more detailed features, while deep features contain more semantic features, fusing the two can effectively improve the performance of the model. Therefore, the embodiment of the present invention fuses the convolutional features output by the current convolutional layer with the state positioning features obtained by spatial transformation of the previous layer, and then spatially transforms the fused features obtained by feature fusion to obtain the state positioning features of the spatial transformation of the current convolutional layer. The state positioning features corresponding to each layer are obtained layer by layer from shallow to deep layers until the state positioning features of the spatial transformation of the last convolutional layer are obtained, thus ending the spatial transformation operation.

[0080] It should be noted that when the current convolutional layer is the first convolutional layer, the convolutional features of the first convolutional layer can be directly used to perform spatial transformation to obtain state positioning features.

[0081] The state detection method provided by the embodiment of the present invention obtains state positioning features by performing layer-by-layer spatial transformation on the convolution features corresponding to each convolution layer of the multi-layer convolutional network, thereby realizing the gradual fusion of the state positioning features obtained in the shallow layer and the convolution features in the deep layer, so as to gradually locate the state-related areas, so that the state positioning features of the image to be detected obtained by adaptive positioning through spatial transformation are more accurate, thereby further improving the accuracy of state detection.

[0082] Based on the above embodiments, Figure 4 This is the second flow chart of the state detection method provided by the present invention. Figure 4 As shown, step 122 includes:

[0083] Step 410: Based on the illumination perception network in the classification network, perform illumination intensity equalization on the image features of the state-related region to obtain equalized features of the state-related region;

[0084] Step 420 : Based on the state classification network in the classification network, the balanced features of the state-related region are applied to perform person state detection on the image to be detected.

[0085] Considering the complex external environment during driving, such as daytime, nighttime, sunny, or cloudy, the image to be detected can experience significant variations in illumination intensity due to these environmental factors. Therefore, to provide a more stable feature input for the state classification network within the classification network, the present invention performs illumination intensity balancing on the image features before inputting them into the state classification network for classification.

[0086] Specifically, a light perception network is built within the classification network to perform light intensity equalization on the image features of the state-related areas output by the state localization network. The image features of the state-related areas output by the state localization network are input into the light perception network for light intensity equalization, obtaining the equalized features of the state-related areas. These equalized features are then input into the classification network for occupant status detection.

[0087] It should be noted that the illumination perception network can perform illumination restoration on image features through an illumination restoration model, so that the illumination intensity of the image features can be balanced, that is, balanced features are obtained. It can also be possible to extract strong light features through a strong light perception network, extract weak light features through a weak light perception network, calculate the illumination intensity weight value through a weight fusion network, and then weight them according to the strong light features, weak light features and illumination intensity weight value to obtain balanced features. The embodiments of the present invention are not limited to this.

[0088] The state detection method provided by the embodiment of the present invention achieves light intensity balancing of image features in state-related areas by adding a light perception network to the state detection model, making the features input into the classification network more stable, further improving the accuracy of state detection.

[0089] Based on the above embodiments, Figure 5 FIG. 1 is a flow chart of the method for obtaining the equalization feature provided by the present invention. Figure 5 As shown, step 410 includes:

[0090] Step 411 , based on the strong light perception network branch and the weak light perception network branch in the light perception network, respectively extract light features from the image features of the state-related area to obtain strong light features of the state-related area and weak light features of the state-related area.

[0091] Taking into account the situation that the image to be detected may have large differences in light intensity due to environmental factors, driver state detection involves driving installation and requires high real-time performance. The existing light restoration model mainly converts complex light changes into slight / moderate light changes through linear iteration, which requires a high processing time. At the same time, while ensuring real-time performance, it also needs to be able to adapt to sudden changes in the environment, such as entering a low-light tunnel from a high-light road. Therefore, the embodiment of the present invention uses a strong-light perception network branch, a low-light perception network branch and a weight fusion branch to jointly balance the image features of the state-related area to obtain balanced features of the state-related area.

[0092] Specifically, the image features of the state-related area are input into the light perception network, and the strong light perception network branch and the weak light perception network branch in the light perception network extract light features in parallel to obtain the strong light features of the state-related area and the weak light features of the state-related area, respectively.

[0093] It should be noted that the network structure of the high-light perception network branch and the low-light perception network branch is identical, including two different types of convolutions at the same latitude to improve the network's adaptability to objects of different scales. The high-light perception network branch and the low-light perception network branch were trained separately using different sample sets.

[0094] In step 412, based on the weight fusion branch in the light perception network, the image features of the state-related area are predicted to obtain the light intensity weight value, and based on the light intensity weight value, the strong light perception features and the weak light perception features are weighted to obtain the balanced features of the state-related area.

[0095] As mentioned above, in order to adapt to sudden changes in the environment, it is necessary to be able to adaptively adjust the weight ratio of the strong light features of the state-related area and the weak light features of the state-related area.

[0096] Specifically, the light perception network also includes a weight fusion branch that predicts the weights of image features in state-related areas to obtain light intensity weights. Based on these light intensity weights, the strong light perception features and weak light perception features are weighted to obtain balanced features in the state-related areas.

[0097] It should be noted that, in order to further improve the execution efficiency of the light perception network, the operation of predicting the light intensity weight value in step 412 can be processed in parallel with step 411. The weighting of the strong light perception features and the weak light perception features in step 412 to obtain the balanced features of the state-related areas needs to wait until step 411 is completed and the operation of predicting the light intensity weight value is completed before processing.

[0098] The state detection method provided by the embodiment of the present invention obtains the balanced characteristics of the state-related area through parallel processing of the strong light perception network branch, the weak light perception network branch and the weight fusion branch, and realizes the balanced characteristics of the light balance output by adaptively sensing the light intensity in a parallel manner, thereby improving the feature extraction capability of the light perception network under complex lighting conditions and improving the execution efficiency of the network.

[0099] Based on the above embodiments, Figure 6 FIG. 1 is a flow chart of the state detection model training method provided by the present invention. Figure 6 As shown in Figure 1, the state detection model is trained based on the following steps:

[0100] Step 610: Determine an initial detection model; the initial detection model includes an initial state positioning network, an initial light perception network, and an initial state classification network;

[0101] Step 620: Train the initial detection model based on the sample images and the person status category labels of the sample images to obtain a status positioning network and a light perception network.

[0102] Considering that the detection of personnel status is mainly divided into image feature extraction of status-related areas and detection of the extracted image features of the status-related areas, in order to make the status detection results more accurate, the embodiment of the present invention trains the image feature extraction part network and the classification network of the status-related areas in a stage-by-stage training manner to improve the image feature extraction ability and classification ability of the status-related areas of the status detection model.

[0103] Specifically, in the first phase, corresponding to steps 610 and 620, an initial detection model is constructed using the initial state localization network, the initial light perception network, and the initial state classification network. Sample images and the person state category labels of the sample images are used to train the initial detection model until training is complete. The initial state localization network and the initial light perception network in the trained initial detection model serve as the state localization network and the light perception network for use in the second phase of training. The initial state localization network and the initial light perception network have different network learning rates. For example, the parameters of the initial light perception network are one-fifth or one-tenth of the learning rate of the initial state localization network, but this is not a limitation in this embodiment of the present invention.

[0104] Step 630: Determine an intermediate detection model; the intermediate detection model includes a state positioning network, a light perception network, and an initial state classification network;

[0105] Step 640 , fixing the parameters of the state positioning network and the light sensing network, and training the intermediate detection model based on the sample images and the person state category labels of the sample images to obtain a state classification network.

[0106] Specifically, in the second stage, corresponding to steps 630 and 640, an intermediate detection model is constructed by the state localization network, the light perception network, and the initial state classification network, and the parameters of the state localization network and the light perception network are fixed. The initial state classification network in the intermediate detection model is trained with sample images and the personnel state category labels of the sample images until the training is completed to obtain the state classification network. At this time, the state localization network and the light perception network trained in the first stage, and the state classification network trained in the second stage constitute the final state detection model.

[0107] Based on the above embodiments, Figure 7 FIG. 1 is a flow chart of the training method of the state classification network provided by the present invention. Figure 7 As shown, step 640 includes:

[0108] Step 641, determining the weight of each personnel status category based on the sample data volume of the sample images under each personnel status category;

[0109] Step 642, based on the state classification results output by the intermediate detection model based on the sample image, the personnel state category labels of the sample image and the weights of each personnel state category, loss calculation is performed, and the parameters of the initial classification network are updated based on the loss calculation results until the loss calculation results converge to obtain a state classification network.

[0110] Considering that the amount of image samples for each person status category varies, and that some categories are particularly difficult to obtain, such as drivers smoking or talking on the phone, the trained model may suffer from a long-tail problem, resulting in inaccurate detection results for categories with small amounts of sample data. Therefore, the present embodiment trains the classification network using category weights.

[0111] Specifically, the weight of the personnel status category is first determined by the sample data volume of the sample images under each personnel status category, and then the loss is calculated based on the status classification results corresponding to the sample images output by the intermediate detection model, the personnel status category labels of the sample images and the weights of each personnel status category. Then, the parameters of the initial classification network are updated according to the loss calculation results. When the loss calculation results reach convergence, the training is completed and the status classification network is obtained.

[0112] It should be noted that the weight of a person's status category can be obtained based on the ratio of the sample data volume of the person's status category to the sample data weight of all person's status categories, or based on the mapping relationship between the sample data volume and the weight, and the embodiment of the present invention does not limit this. In addition, the above-mentioned loss calculation based on the status classification result corresponding to the sample image output by the intermediate detection model, the person's status category label of the sample image, and the weight of each person's status category can be calculated as follows:

[0113] First, the weight value w = [w1, w2 ... w m ], where m is the number of categories of personnel status.

[0114] Then, the weight value of the personnel status category w=[w1,w2……w m ], the state prediction result of the sample image output by the initial state classification network is p=[p1,p2……p m ], and the person status category label of the sample image is y=[y1,y2……y m ] to calculate the loss, and then update the parameters of the initial classification network according to the loss calculation results. When the loss calculation results reach convergence, the training is completed and the state classification network is obtained. Among them, the loss function is specifically expressed as:

[0115]

[0116] In the formula, γ is a hyperparameter, m is the number of categories of personnel status, and w i is the weight value of the i-th personnel status category, y i is the label value of the i-th person status category of the sample image, p i is the status prediction result of the i-th person status category of the sample image.

[0117] The state detection method provided by an embodiment of the present invention calculates the weight of each personnel state category based on the number of sample images of each personnel state category during the training of the state detection model, and performs loss calculation based on the weight of each personnel state category, thereby increasing the weight of the tail samples, solving the problem of uneven sample distribution, and improving the recognition accuracy of the state detection model for personnel state categories with a small sample size.

[0118] Based on the above embodiments, Figure 8 This is a network framework diagram of the state detection model provided by the present invention. In the figure, Indicates connection, represents multiplication, Indicates addition. Figure 8 As shown in the figure, the execution process of the model is as follows:

[0119] Step 810: After pre-processing the driver image to be detected, the state localization network in the state detection model is used to extract state localization features. The specific operation details are as follows:

[0120] In step 811, a multi-layer convolutional network is used as a feature extractor. The multi-layer convolutional network can be any CNN network. In this embodiment of the present invention, the CNN network of resnet50 is used as an example. Res1-2 extracts features from the image of the driver to be detected to obtain bottom-level features (convolutional features) F1. Res3 extracts features from bottom-level features F1 to obtain middle-level features (convolutional features). Res4 extracts features from middle-level features F2 to obtain deep-level features (convolutional features) F3. First, an STN (spatial transformer network) is used on the bottom-level features F1 to locate the state-related region, obtaining the state location feature LF1.

[0121] in, Figure 9 This is a diagram of the STN network structure provided by the present invention. In the diagram, Indicates multiplication. Figure 9 As shown, STN can perform spatial transformations on image features, such as cropping, translation, and scaling, and adaptively discover distinguishing regions in different states. First, the parameters of the spatial transformation θ = [θ1, θ2, θ3, θ4] are generated through the fully connected layer FC, where θ1 and θ2 are scaling parameters, and θ3 and θ4 are translation parameters. The values ​​of the parameters are constrained to be in the range of (0, 1) or (-1, 1) through the sigmoid or tanh function. A bounding box is obtained through these four parameters, and the new pixel coordinates are obtained through the following expression:

[0122]

[0123] Where, Represents the original coordinates of a pixel in the image. The coordinates after transformation.

[0124] In step 812, the state positioning feature LF1 and the middle-layer feature F2 are connected, and the connected features are passed through STN again to obtain the middle-layer state positioning feature LF2.

[0125] In step 813, the state location feature LF2 and the deep feature F3 are connected. The connected features are passed through the STN again to obtain the bottom-level state location feature LF3. The bottom-level state location feature LF3 and F3 are fused to obtain the image features of the state-related area. The deep features of CNN have a coarser resolution and may have some subtle loss. In contrast, the shallow features contain richer details but lack contextual information. It can be seen that low-level details and high-level semantics are complementary. Therefore, the embodiment of the present invention gradually and accurately locates the state-related area through progressive positioning.

[0126] In step 820, the image features of the state-related area obtained in step 810 may come from images of the driver under strong or weak light conditions. Therefore, this embodiment of the present invention constructs a light perception network. The light perception network integrates the effective features of the two branches and can process images of any style. In this way, the problem of light differences can be effectively solved. The specific operation details are as follows:

[0127] Step 821: The light sensing network includes: a strong light sensing network branch and a weak light sensing network branch, wherein the strong light sensing network branch and the weak light sensing network branch are both CNN networks. Figure 10 This is the network framework diagram of the light perception network provided by the present invention. In the figure, Conv represents convolution and Concat represents connection. Figure 10 As shown in the figure, different illumination sample image data are used for training to learn feature representations under specific illumination conditions. In each branch, the first 1×1 convolution is used to capture illumination-specific feature representations. Then, two additional 1×1 convolution layers with half channels are used to reduce the dimensionality of the input features, which are divided into two streams and fed into two types of 3×3 convolutions to improve the network's adaptability to targets of different scales. The outputs of the strong light perception network branch and the weak light perception network branch are connected together as illumination-specific feature representations. The image features of the state-related areas pass through the strong light perception network branch and the weak light perception network branch respectively to obtain the strong light feature F a and low-light feature F b .

[0128] Step 822, in the actual detection process, the input image is often just one lighting scene. In order to be able to adaptively obtain the balanced features of the image, a weight fusion branch is constructed in the lighting perception network. Given a modal input, the features of the two branch outputs are adaptively integrated. In this way, no matter which modality is input, effective features can be obtained. The embodiment of the present invention utilizes the strong light features and weak light features output in parallel by the strong light perception network branch and the weak light perception network branch, and fuses them with normalized weights to achieve adaptive fusion of the features of the two branches. Specifically, a sigmoid-based weight fusion branch is used to predict an adaptive weight for modality selection to solve this problem.

[0129] The weight fusion branch consists of a global average pooling (GAP) layer and two fully connected layers (FC), followed by a sigmoid function with learnable parameters. In this way, a normalized selection weight can be predicted and soft selection can be performed as shown below:

[0130] F=α*F a +(1-α)*F b

[0131]

[0132] Where α is the selection weight, x is the output feature of the image features of the state-related area after GAP and two FC layers, and k is a parameter adjusted based on training to adapt to changes in different lighting conditions. Through this method, more appropriate selection weights can be predicted in complex scenarios, adapting to changes in input cross-model data and maintaining feature recognition capabilities during modal switching.

[0133] Step 830: Perform personnel status detection on the balanced features of the status-related areas outputted in step 820. The specific operation details are as follows:

[0134] In step 831, the balanced features of the state-related regions are first passed through the global average pooling layer in the state classification network, and then m 1x1 convolutional layers are added, where m is the number of state categories.

[0135] Step 832: For each feature that has undergone 1x1 convolution, use the linear layer in the state classification network to obtain the category prediction result of the state, and then splice all the predicted results together to obtain the final driver state prediction result p = [p1, p2 ... p m ], and determine whether to issue an early warning reminder based on the status detection result, where m is the number of categories of personnel status.

[0136] The state detection device provided by the present invention is described below. The state detection device described below and the state detection method described above can be referenced to each other.

[0137] Figure 11 Schematic diagram of the structure of the state detection device provided by the present invention. Figure 11 As shown, the device includes: a determination module 1110 and a detection module 1120.

[0138] in,

[0139] A determination module 1110 is configured to determine an image to be detected;

[0140] A detection module 1120 is configured to adaptively locate state-related regions in the image to be detected through spatial transformation based on the state detection model, and perform person state detection on the image to be detected based on the state-related regions;

[0141] The state detection model is trained based on sample images and the person state category labels of the sample images.

[0142] In an embodiment of the present invention, a determination module 1110 is used to determine an image to be detected; a detection module 1120 is used to adaptively locate a state-related area in the image to be detected through spatial transformation based on a state detection model, and perform personnel state detection on the image to be detected based on the state-related area; the state detection model is obtained by training based on sample images and personnel state category labels of the sample images, and realizes the use of state-related areas as detection targets to obtain areas related to the state in the image to be detected, thereby reducing the problem of erroneous subsequent personnel state category detection results caused by fixed area detection, and improving the accuracy of personnel state category detection.

[0143] Based on any of the above embodiments, the detection module 1120 includes:

[0144] The state localization submodule is used to adaptively locate the state-related area in the image to be detected through spatial transformation based on the state localization network in the state detection model, obtain the state localization feature, and determine the image feature of the state-related area through the state localization feature and the convolution feature of the image to be detected. The state localization feature is used to indicate the position of the state-related area in the image to be detected;

[0145] The state classification submodule is used to detect the state of people in the image to be detected by applying the image features of the state-related areas based on the classification network in the state detection model.

[0146] Based on any of the above embodiments, the state positioning submodule includes:

[0147] The feature extraction submodule is used to extract features of the image to be detected based on the multi-layer convolutional network in the state localization network, and obtain the convolution features of each convolution output in the multi-layer convolutional network;

[0148] The spatial transformation submodule is used to perform spatial transformation based on the spatial transformation network in the state localization network, apply the convolution features of the current layer convolution output and the spatial transformation features obtained by the spatial transformation of the previous layer, obtain the spatial transformation features of the current layer spatial transformation, until the spatial transformation features of the last layer are obtained, and the spatial transformation features of the last layer are determined as the state localization features, and the image features of the state-related area are determined by the state localization features and the convolution features output by the last layer convolution.

[0149] Based on any of the above embodiments, the spatial transformation submodule is specifically configured to:

[0150] It is used to fuse the convolution features output by the current convolution layer with the state positioning features obtained by the spatial transformation of the previous layer to obtain the fused features corresponding to the convolution of the current layer, and perform spatial transformation on the fused features corresponding to the convolution of the current layer to obtain the state positioning features of the spatial transformation of the current layer.

[0151] Based on any of the above embodiments, the status classification submodule includes:

[0152] The balanced feature extraction submodule is used to balance the light intensity of the image features in the state-related area based on the light perception network in the classification network to obtain the balanced features of the state-related area;

[0153] The state detection submodule is used to detect the state of the person in the image to be detected by applying the balanced features of the state-related area based on the state classification network in the classification network.

[0154] Based on any of the above embodiments, the equalization feature extraction submodule includes:

[0155] An illumination feature extraction module is used to extract illumination features from the image features of the state-related area based on the strong light perception network branch and the weak light perception network branch in the illumination perception network, thereby obtaining strong light features of the state-related area and weak light features of the state-related area.

[0156] The weight fusion submodule is used to predict the image features of the state-related area based on the weight fusion branch in the light perception network, obtain the light intensity weight value, and weight the strong light perception features and weak light perception features based on the light intensity weight value to obtain the balanced features of the state-related area.

[0157] Based on any of the above embodiments, the state detection device further includes: a training module, the training module including:

[0158] Construct an initial model submodule to determine the initial detection model; the initial detection model includes an initial state positioning network, an initial light perception network, and an initial state classification network;

[0159] The first-stage training submodule is used to train the initial detection model based on sample images and the person status category labels of the sample images to obtain the state positioning network and light perception network;

[0160] Construct an intermediate model submodule to determine the intermediate detection model; the intermediate detection model includes a state positioning network, a light perception network, and an initial state classification network;

[0161] The second stage training submodule is used to fix the parameters of the state positioning network and the light perception network, and train the intermediate detection model based on the sample images and the person state category labels of the sample images to obtain the state classification network.

[0162] Based on any of the above embodiments, the second-stage training submodule includes:

[0163] A status category weight calculation submodule is used to determine the weight of each personnel status category based on the sample data volume of the sample images under each personnel status category;

[0164] The state classification network training submodule is used to perform loss calculation based on the state classification results output by the intermediate detection model based on the sample image, the person state category label of the sample image, and the weight of each person state category, and update the parameters of the initial classification network based on the loss calculation results until the loss calculation results converge to obtain the state classification network.

[0165] Figure 12 An example of a physical structure diagram of an electronic device is shown below. Figure 12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 may call logic instructions in the memory 1230 to execute a state detection method, which includes: determining an image to be detected; based on a state detection model, adaptively locating state-related regions in the image to be detected through spatial transformation, and performing person state detection on the image to be detected using the state-related regions; the state detection model is trained based on sample images and the person state category labels of the sample images.

[0166] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the state detection method provided by the above methods, which includes: determining the image to be detected; based on the state detection model, adaptively locating the state-related area in the image to be detected through spatial transformation, and performing personnel state detection on the image to be detected through the state-related area; the state detection model is obtained by training based on the sample image and the personnel state category label of the sample image.

[0168] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the state detection method provided by the above-mentioned methods, the method comprising: determining an image to be detected; based on a state detection model, adaptively locating state-related areas in the image to be detected through spatial transformation, and performing state detection on the image to be detected through the state-related areas; the state detection model is obtained by training based on sample images and personnel state category labels of the sample images.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0170] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A state detection method, characterized in that: include: Determine an image to be detected; Based on the state localization network in the state detection model, the state-related area in the image to be detected is adaptively located through spatial transformation to obtain a state localization feature, and the image feature of the state-related area is determined by the state localization feature and the convolution feature of the image to be detected, wherein the state localization feature is used to indicate the position of the state-related area in the image to be detected; Based on the classification network in the state detection model, applying the image features of the state-related area to perform person state detection on the image to be detected; The state detection model is trained based on sample images and person state category labels of the sample images; The state localization network in the state detection model adaptively locates the state-related area in the image to be detected through spatial transformation to obtain a state localization feature, and determines the image feature of the state-related area through the state localization feature and the convolution feature of the image to be detected, including: Based on the multi-layer convolutional network in the state localization network, feature extraction is performed on the image to be detected to obtain convolution features of each convolution output in the multi-layer convolutional network; Based on the spatial transformation network in the state localization network, the convolution features of the current layer convolution output and the spatial transformation features obtained by the spatial transformation of the previous layer are applied to perform spatial transformation to obtain the spatial transformation features of the current layer spatial transformation until the spatial transformation features of the last layer are obtained, and the spatial transformation features of the last layer are determined as the state localization features. The image features of the state-related area are determined by the state localization features and the convolution features output by the last layer convolution.

2. The state detection method according to claim 1, characterized in that: The step of applying the convolution features output by the convolution of the current layer and the spatial transformation features obtained by the spatial transformation of the previous layer to perform spatial transformation to obtain the spatial transformation features of the spatial transformation of the current layer includes: The convolution features of the current layer convolution output are fused with the state positioning features obtained by the spatial transformation of the previous layer to obtain the fused features corresponding to the current layer convolution, and the fused features corresponding to the current layer convolution are spatially transformed to obtain the state positioning features of the current layer spatial transformation.

3. The state detection method according to claim 1, characterized in that: The method of performing personnel status detection on the image to be detected by applying the image features of the status-related area based on the classification network in the status detection model includes: Based on the illumination perception network in the classification network, performing illumination intensity equalization on the image features of the state-related area to obtain equalized features of the state-related area; Based on the state classification network in the classification network, the balanced features of the state-related area are applied to perform personnel state detection on the image to be detected.

4. The state detection method according to claim 3, characterized in that: The light perception network in the classification network is used to perform light intensity balancing on the image features of the state-related area to obtain balanced features of the state-related area, including: Based on the strong light perception network branch and the weak light perception network branch in the light perception network, respectively extracting light features from the image features of the state-related area to obtain strong light features of the state-related area and weak light features of the state-related area; Based on the weight fusion branch in the light perception network, the image features of the state-related area are predicted to obtain a light intensity weight value, and based on the light intensity weight value, the strong light feature and the weak light feature are weighted to obtain a balanced feature of the state-related area.

5. The state detection method according to claim 3, characterized in that: The state detection model is trained based on the following steps: Determine an initial detection model; the initial detection model includes an initial state positioning network, an initial light perception network and an initial state classification network; Training an initial detection model based on the sample images and the person status category labels of the sample images to obtain the status positioning network and the light perception network; Determining an intermediate detection model; the intermediate detection model includes the state positioning network, the light perception network, and the initial state classification network; The parameters of the state localization network and the light perception network are fixed, and the intermediate detection model is trained based on the sample image and the person state category label of the sample image to obtain the state classification network.

6. The state detection method according to claim 5, characterized in that: The method of fixing the parameters of the state localization network and the light sensing network, and training the intermediate detection model based on the sample images and the person state category labels of the sample images to obtain the state classification network includes: Determining the weight of each personnel status category based on the sample data volume of the sample images under each personnel status category; Loss calculation is performed based on the state classification result output by the intermediate detection model based on the sample image, the personnel state category label of the sample image and the weight of each personnel state category, and the parameters of the initial state classification network are updated based on the loss calculation result until the loss calculation result converges to obtain the state classification network.

7. A state detection device, characterized in that: include: A determination module, used for determining an image to be detected; a detection module, configured to adaptively locate a state-related region in the image to be detected by spatial transformation based on a state localization network in a state detection model, obtain a state localization feature, and determine an image feature of the state-related region based on the state localization feature and a convolution feature of the image to be detected, wherein the state localization feature is used to indicate a position of the state-related region in the image to be detected; The detection module is further configured to perform personnel status detection on the image to be detected by applying the image features of the status-related area based on the classification network in the status detection model; The state detection model is trained based on sample images and person state category labels of the sample images; The detection module is specifically used for: Based on the multi-layer convolutional network in the state localization network, feature extraction is performed on the image to be detected to obtain convolution features of each convolution output in the multi-layer convolutional network; Based on the spatial transformation network in the state localization network, the convolution features of the current layer convolution output and the spatial transformation features obtained by the spatial transformation of the previous layer are applied to perform spatial transformation to obtain the spatial transformation features of the current layer spatial transformation until the spatial transformation features of the last layer are obtained, and the spatial transformation features of the last layer are determined as the state localization features. The image features of the state-related area are determined by the state localization features and the convolution features output by the last layer convolution.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the state detection method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the state detection method according to any one of claims 1 to 6 are implemented.

Images