Method and system for safety equipment detection

Abstract

A method for safety equipment detection is provided. The method includes obtaining an input image containing an occupant, inputting the input image into a pre-trained instance segmentation model to obtain instance segmentation result output by the instance segmentation model, calculating attribute data associated with a safety equipment based on the instance segmentation result, and determining whether the occupant is equipped with the safety equipment based on the attribute data.

Description

[Technical Field]

[0001] This invention relates to image analysis technology, and in particular to a method and system for detecting security devices. [Previous Technology]

[0002] Passive safety equipment is used in various vehicles (such as cars, locomotives, trains, airplanes, and yachts), construction vehicles (such as forklifts, excavators, and cranes), and amusement facilities (such as roller coasters, pirate ships, and freefall rides). Examples include seat belts, helmets, life jackets, and goggles. Their primary purpose is to reduce injuries to occupants in the event of an accident. Taking the automotive industry as an example, the application of artificial intelligence technology in Advanced Driving Assistant Systems (ADAS) has led to its increasing popularity. One of the key focuses of ADAS is the detection of whether occupants (especially drivers) are wearing safety equipment. The aim is to quickly and accurately detect the absence of safety equipment to assist in issuing timely warnings. It has been proven that the harm caused by occupants not wearing safety equipment in various accidents is not to be underestimated. In various accidents, the lack of safety equipment can lead to more serious or even fatal injuries to occupants. Therefore, the research and development of safety equipment detection has extremely important practical significance.

[0003] Currently, unsafe behavior detection of drivers and passengers typically uses object detection technology, such as detecting the location of a mobile phone in an image during phone calls. However, known object detection technologies cannot meet the practical needs of safety device detection because object detection outputs a rectangular bounding box that surrounds the object, but the attributes of some safety devices are not suitable for description using only a rectangular bounding box. Taking seat belts as an example, because the seat belt occupies too small a proportion of the rectangular bounding box and there is too much background information, the false detection rate is too high, making it difficult to meet the accuracy requirements in practice.

[0004] On the other hand, instance segmentation technology involves segmenting regions in an image at the pixel level and identifying the instance category of each region to determine the location and extent of a specific object in the image. Existing instance segmentation models, such as the SparseInst network, may enjoy real-time computation efficiency and relatively high accuracy on personal computers or server computers. However, due to limitations in computing resources, instance segmentation models are difficult to deploy in ADAS or other similar embedded systems without sacrificing computational efficiency and accuracy. In addition to computational capabilities, the board support package (BSP) used by some embedded systems also has limited support for neural networks, making it difficult to support or not supporting certain special types of neural network layers, such as downsampling layers.

[0005] Therefore, there is a need for a detection method and system for security devices that can overcome the above-mentioned technical challenges. [Summary of the Invention]

[0006] One embodiment of this disclosure provides a method for detecting a safety device implemented by a computer system, which includes acquiring an input image containing a driver or passenger, inputting the input image into a pre-trained instance segmentation model to obtain an instance segmentation result output by the instance segmentation model, calculating attribute data associated with the safety device based on the instance segmentation result, and determining whether the driver or passenger is equipped with a safety device based on the attribute data.

[0007] One embodiment of this disclosure further provides a safety device detection system, which includes a processing device that performs the steps of acquiring an input image containing a driver or passenger, inputting the input image into a pre-trained instance segmentation model to obtain an instance segmentation result output by the instance segmentation model, calculating attribute data associated with the safety device based on the instance segmentation result, and determining whether the driver or passenger is equipped with a safety device based on the attribute data.

Implementation Method

[0008] The following description lists various embodiments of the present invention, but is not intended to limit the scope of the invention. The actual scope of the invention is defined by the claims.

[0009] In the embodiments listed below, the same or similar elements or components will be represented by the same reference numerals.

[0010] The serial numbers in this specification and in the claims, such as "first", "second", etc., are for convenience only and there is no sequential relationship between them.

[0011] The following description of embodiments of the apparatus or system also applies to embodiments of the method, and vice versa.

[0012] In order to better describe the embodiments disclosed herein, some of the most critical terms involved are defined as follows.

[0013] Safety equipment: can be any equipment applied to various means of transportation (e.g., automobiles, locomotives, trains, airplanes and yachts), engineering vehicles (e.g., forklifts, excavators and cranes) and amusement facilities (e.g., roller coasters, pirate ships and freefall facilities) that aims to reduce the injury to occupants in the event of an accident, such as seat belts, helmets, life jackets and goggles.

[0014] Instance segmentation: A technique in the field of computer vision that involves segmenting regions in an image at the pixel level and identifying instance categories of each region in order to identify the location and extent of a specific object in the image.

[0015] Attribute data: Associated with security equipment, used to describe the visible physical properties and / or characteristics of security equipment, such as the location, size, shape, orientation, color and other related information of security equipment in the image.

[0016] Figure 1 is a flowchart of a detection method 10 for a security device according to one embodiment of the present disclosure. As shown in Figure 1, method 10 may include steps S101-S104.

[0017] In step S101, an input image containing the driver and passengers is obtained. For example, the input image may be an image obtained by taking a picture of the driver's seat inside a car and then performing certain preprocessing, such as geometric transformation, normalization, smoothing, image enhancement, and / or cropping. However, the various embodiments disclosed herein do not limit the application to the detection of car driving, nor do they limit the preprocessing operations performed on the captured images. In some embodiments, the input image may originate from various vehicles (e.g., cars, locomotives, trains, airplanes, and yachts), construction vehicles (e.g., forklifts, excavators, and cranes), or amusement facilities (e.g., roller coasters, pirate ships, and freefall facilities).

[0018] In step S102, the input image is input into a pre-trained instance segmentation model to obtain the instance segmentation result output by the instance segmentation model. The instance segmentation model involves segmenting regions in the image at the pixel level and identifying the instance category of each region to identify the location and range of a specific object in the image. In the instance segmentation result, the security device is regarded as an instance object, and the pixels in its region are marked with category labels, such as "1" to represent the region of the security device and "0" to represent the background region, so as to further identify the security device in the input image.

[0019] In step S103, based on the instance segmentation results, attribute data associated with the safety device is calculated. For example, the attribute data may include information related to the safety device's position, size, shape, orientation, and color in the image, but this disclosure is not limited to this. Safety devices can be any equipment applied to various means of transportation (e.g., cars, locomotives, trains, airplanes, and yachts), construction vehicles (e.g., forklifts, excavators, and cranes), and amusement facilities (e.g., roller coasters, pirate ships, and freefall facilities) that aims to reduce injuries to occupants in the event of an accident, such as seat belts, helmets, life jackets, and goggles.

[0020] In step S104, based on the attribute data, it is determined whether the driver or passenger is equipped with safety equipment. Conceptually, if the driver or passenger is indeed equipped with safety equipment, the safety equipment will have a reasonable position and range configuration in the input image. Therefore, it can be determined whether the driver or passenger is equipped with safety equipment by analyzing the degree of consistency between the relevant information in the attribute data and the expected position and range.

[0021] In one embodiment, the above instance segmentation model is implemented using an improved SparseInst network. Typical SparseInst networks are characterized by large scale, complex structure, numerous parameters, and special network types, making them unsuitable for deployment on computer systems with limited computing resources and limited support for network types, such as embedded systems. To address these issues, certain further embodiments of this disclosure relate to improvements to the SparseInst network to expand its application scope. The various possible improvements to the SparseInst network proposed in this disclosure will be described below with reference to Figures 2 and 3.

[0022] Figure 2 illustrates a typical architecture of a SparseInst network 20. As shown in Figure 2, the SparseInst network 20 includes a backbone module 21, an encoder module 22, and a decoder module 23, which will be described in sequence below.

[0023] The main function of the backbone module 21 is to extract image features. As shown in Figure 2, the backbone module 21 takes an input image 201 with a batch size of B, a channel size of 3 (or "3D"), a width of W, and a height of H (marked as (B,3,W,H) in the figure) as input, and uses the backbone 202 to extract multi-scale features 203 from the input image 201, including feature maps of different levels such as C5, C4, and C3. These levels correspond to feature extraction layers of different depths. The C3 feature map contains a lower-level feature representation, which usually has higher spatial information and less semantic information. The C4 feature map contains a medium-level feature representation, which has lower spatial information than the C3 feature map, but contains richer semantic information. The C5 feature map contains a higher-level feature representation, which has the least spatial information, but the richest semantic information. In the example in Figure 2, the dimensions (width and height) of feature maps C5, C4, and C3 are 1 / 32, 1 / 16, and 1 / 8 of the input image 201, respectively, with 256, 1024, and 512 channels. A typical backbone 202 is implemented using a residual neural network (ResNet), which has a large number of parameters, requiring considerable storage and computational resources for both training and inference phases.

[0024] The main function of encoder module 22 is to enhance contextual information and fuse multi-scale features 203. As shown in Figure 2, encoder module 22 takes the multi-scale features 203 output by backbone module 21 as input, where feature maps C5, C4, and C3 are processed by pyramid pooling module (PPM) 204, convolution 205, and convolution 206, respectively. The main function of pyramid pooling module 204 is to expand the receptive field and fuse features of different scales to enhance SparseInst's ability to perceive global information. The result of C5 after the pyramid pooling module is processed by convolution 207 to output a feature map whose size is increased by a factor of 4 (i.e., from 1 / 32 to 1 / 8 of the input image 201). It is also increased by a factor of 2 and added to the result of convolution 205 on the C4 feature map (represented by ⊕ in the diagram). This added result is then processed by convolution 208 to output a feature map whose size is increased by a factor of 2 (i.e., from 1 / 16 to 1 / 8 of the input image 201). It is then added to the result of convolution 206 on the C3 feature map, and finally processed by convolution 209 to output a feature map whose size is 1 / 8 of the input image 201. Next, the outputs of convolutions 207, 208, and 209 are combined and fused to obtain feature map 210 with batch size B, number of channels 256, width W / 8, and height H / 8. Feature map 210 is then processed by convolution 211 to obtain feature map 212. Feature map 212 is concatenated (concat) 214 with 2D coordinate information of the same size (i.e., x and y coordinates) to obtain a feature map with batch number B, channel number 258, width W / 8, and height H / 8, i.e., encoded features 215, which serves as the output of encoder module 22. It is worth noting that, unlike the illustration in Figure 2, some literature lists feature map 212 as the output of encoder module 22, while listing the concatenation 214 of feature map 212 with coordinate information 213 as an operation of decoder module 23. However, it is certain that a typical feature map 212 involves coordinate concatenation operations, whether in encoder module 22 or decoder module 23. Since the positions of different instances are also different, the introduction of coordinate information 213 helps the model distinguish between instances.

[0025] The decoder module 23 mainly utilizes the sparse set of instance activation maps to highlight distinguishable instance pixels while suppressing useless pixels, thus facilitating the classification and segmentation of foreground objects. As shown in Figure 2, the decoder module 23 includes a mask branch 216 and an instance branch 217. Both branches take the encoded features 215 output by the encoder module 22 as input and output mask features 218 and instance information 219, respectively. The mask branch 216 mainly consists of a sequence of convolutions to derive the mask features 218 from the encoded features 215. The mask features 218 are sometimes abbreviated as "Mask" in some literature. They indicate the probability that a pixel belongs to a particular instance, and can therefore be used to generate instance segmentation results, i.e., to determine whether a pixel belongs to a certain instance. The instance branch 217 also mainly consists of a sequence of convolutions to derive the instance information 219 from the encoded features 215. Instance information 219 includes core features 220, predicted class confidence 221, and predicted object confidence 222. Core features 220, often abbreviated as "Kernel," contain the feature representation of the instance, i.e., a 128-dimensional vector of num_mask potential instances. Predicted class confidence 221, sometimes abbreviated as "pred_logits," refers to a tensor where each element represents the model's prediction probability score for each pixel belonging to a certain instance; therefore, it contains the confidence scores for num_mask potential instances corresponding to num_class categories. Predicted object confidence 222, sometimes abbreviated as "pred_objectness," represents the probability score of whether num_mask potential instances exist.

[0026] It should be noted that the mask feature 218 output by the mask branch 216 is not the final predicted mask of the model. It still needs to be combined with the instance information 219 provided by the instance branch 217 to obtain the instance segmentation result. Specifically, the instance segmentation result includes a prediction mask and its corresponding confidence score. The prediction mask is obtained by performing matrix multiplication on the mask feature 218 and the core feature 220 and taking the square root. The confidence score is obtained by multiplying the prediction class confidence score 221 and the prediction object confidence score 222.

[0027] The training phase of the SparseInst network 20 involves optimizing the SparseInst network 20 using a training dataset and a loss function. The training dataset contains multiple training data sets, each containing a training image and its corresponding labeled data. The training images can be collected by the developer and preprocessed, or they can be collected from open-source datasets such as Pascal VOC or CoCo (Common Objects in Context). The labeled data can be, for example, the results of using labelme or other similar annotation tools to depict the edge contours of object instances in the training images and label their categories on the user interface provided by the tool, and then parsed and converted into files in a specific format (e.g., JSON, XML, or CSV) required for training. During training, the training images are input into the SparseInst network 20 to obtain the instance segmentation prediction results output by the SparseInst network 20 corresponding to the training images. The loss function evaluates the difference between the predictions output by the SparseInst network during training and the labeled data serving as the ground truth. This difference determines the direction of parameter updates for the SparseInst network, typically achieved through gradient descent or its variants, such as stochastic gradient descent (SGD), Momentum, AdaGrad, or Adam, combined with backpropagation. After the loss converges, a test dataset different from the training dataset is used to allow the SparseInst network to perform inferences to evaluate its generalization ability, ensuring it can accurately analyze diverse data. The test results help determine if the model is overfitting and provide a basis for further optimization.

[0028] The loss function used to evaluate and optimize the SparseInst network 20 involves focal loss, mask loss, and binary cross-entropy loss, as shown in the following <Formula 1>: <Formula 1> Where represents the loss function, represents the focus loss for object classification, represents the mask loss between the predicted mask and the baseline fact mask, and represents the binary cross-entropy loss for the Intersection over Union (IoU). The coefficients of and are hyperparameters used to mitigate the imbalance between foreground and background. To further overcome this imbalance, the mask loss is a combination of Dice loss and pixel-wise binary cross-entropy loss, as shown in the following <Formula 2>: <Formula 2> Where and represent the Dice loss and pixel-wise binary cross-entropy loss between the predicted mask and the baseline fact mask, respectively, and and are the corresponding coefficients. More specifically, the Dice loss is calculated as follows <Formula 3>: <Formula 3> Where and represent the values ​​of the prediction mask and the baseline fact mask at the (x,y) position, respectively.

[0029] Figure 3 is an architecture diagram of an improved SparseInst network 30 according to one embodiment of the present disclosure. Similar to the typical SparseInst network 20, the improved SparseInst network 30 also includes a backbone module 31, an encoder module 32, and a decoder module 33, and the main functions of each module are similar to those of the aforementioned backbone module 21, encoder module 22, and decoder module 23. Specifically, the backbone module 31 involves extracting multi-scale features 303 of the input image 301 using a backbone 301. The encoder module 32 involves performing a series of feature fusion operations on the multi-scale features 303, including convolutions 305-309 and 311, and addition operations on the convolution results during the process to obtain encoded features 315. The decoder module 33 includes a mask branch 316 and an instance branch 317. The mask branch 316 involves deriving mask features 318 from the encoded features 315, and the instance branch 317 involves deriving instance information 319 from the encoded features 315, including core features 320, predicted class confidence 321, and predicted object confidence 322. The instance segmentation model further involves deriving instance segmentation results from the mask features 318 and instance information 319, as previously described in detail, and will not be repeated here. The following will explain the possible improvements of the improved SparseInst network 30 compared to the typical SparseInst network 20. It should be understood that although Figure 3 illustrates all possible improvements, this disclosure does not limit the application of all possible improvements. In various embodiments, some improvements may be selected for application based on practical needs, such as available system computing resources and / or support for specific network types.

[0030] In one embodiment, the backbone 302 used by the backbone module 31 is implemented using a MobileNetV2 network, replacing the ResNet used in the typical backbone 202. MobileNetV2 is a lightweight convolutional neural network with a much smaller number of parameters than ResNet. Therefore, replacing ResNet with MobileNetV2 as the backbone 302 can significantly reduce the computational load. Furthermore, the multi-scale features 303 extracted by the backbone 302 include C5, C4, and C3 feature maps, each with a dimension of 128, lower than the 256, 1024, and 512 dimensions of the C5, C4, and C3 feature maps of the multi-scale features 203, respectively. Subsequently, the feature map 310 obtained by summing the outputs of convolutions 307, 308, and 309 also has a dimension of 128, lower than the 256 dimensions of the feature map 210 in the second figure. This reduces the computational load of the encoder module 32 and the decoder module 33, and improves the inference speed.

[0031] In one embodiment, the encoder module 32 does not involve a pyramid pooling module. Specifically, the C5 feature map is directly processed by convolution 307 and then directly magnified by 2 and added to the result of convolution 305 of the C4 feature map. This process no longer involves the pyramid pooling module 204 shown in Figure 2. The removal of the pyramid pooling module can further reduce the amount of computation and allow the improved SparseInst network 30 to be deployed in embedded systems with relatively monotonous neural network support.

[0032] In one embodiment, neither the encoder module 32 nor the decoder module 33 involves coordinate concatenation. Specifically, the feature map obtained after convolution 311 of the feature map 310 is directly used as the input to the encoded feature 315, namely the mask branch 316 and the instance branch 317, without being concatenated with coordinate information. This allows the improved SparseInst network 30 to be deployed in embedded systems that do not support coordinate concatenation. Furthermore, training the instance segmentation model involves optimizing the improved SparseInst network 30 using an improved loss function. To compensate for the loss of coordinate information caused by the removal of coordinate concatenation, the improved loss function adds a heatmap loss to the original loss function (as described in Equation 1 above), as shown in Equation 4 below: <Formula 4> Wherein, represents the improved loss function, represents the heatmap loss between the predicted class confidence and the baseline fact class confidence, and is a hyperparameter. More specifically, the heatmap loss is calculated as follows <Formula 5>: <Formula 5> Where, represents the total number of categories (which can be set to 1 if used only for identifying security devices), c represents the category label, represents the predicted category confidence value at position (x,y) and category c, and represents the baseline fact category confidence value at position (x,y) and category c. The baseline fact category confidence is calculated as follows <Formula 6>: <Formula 6> Where represents the confidence level of the baseline fact category, and represents the value of the baseline fact mask at position (x,y). The sum is 1 when (x,y) is the coordinate of the object's center point, and 0 otherwise.

[0033] In one embodiment, instance branch 317 involves multiple convolutional layers but no downsampling layers. In other words, typical instance branch 217 involves downsampling layers, while in instance branch 317, downsampling layers are replaced by convolutional layers. This allows the improved SparseInst network 30 to be deployed in embedded systems that do not support downsampling layers. Furthermore, the core features 220, predicted class confidence 221, and predicted object confidence 222 derived from typical instance branch 217 have a fixed size of num_mask, i.e., the number of potential instance masks, for example, 20*20. As for the core features 320, predicted class confidence 321, and predicted object confidence 322 derived from instance branch 317, their sizes depend on the size of the input image 301. In the example of Figure 3, the sizes of core features 320, predicted class confidence 321, and predicted object confidence 322 are 1 / 32 of the input image 301. For example, in a typical SparseInst network 20, for input images of 640*640 and 256*256, the core feature 220, predicted class confidence 221, and predicted object confidence 222 derived from instance branch 217 are all 20*20 in size; while in the improved SparseInst network 30, the core feature 320, predicted class confidence 321, and predicted object confidence 322 derived from instance branch 317 will be 20*20 and 8*8, respectively. This demonstrates that for instance branch 317, simply specifying a lower input image size can significantly reduce the required computation.

[0034] Figure 4A is a flowchart of more detailed steps S401-S403 of step S101 in one embodiment. Corresponding to Figure 4A, Figure 4B is a schematic diagram of steps S401-S403. Please refer to Figures 4A and 4B together for a better understanding of this embodiment.

[0035] In step S401, an original image 411 is captured using a photographic device. The scene in which the original image 411 is captured may be, for example, the seat of various vehicles (e.g., cars, locomotives, trains, airplanes, and yachts), construction vehicles (e.g., forklifts, excavators, and cranes), or amusement facilities (e.g., roller coasters, pirate ships, and freefall facilities), but this disclosure is not limited to this.

[0036] In step S402, the occupant 410 in the original image 411 is detected, and a local image 413 containing the occupant 410 is cropped from the original image 411. The detection of the occupant 410 can be achieved through a machine learning-based object detection model, or through traditional object detection algorithms such as the Haar cascade detector, template matching, and Hough transform; however, this disclosure is not limited to these methods. The detection result is a rectangular bounding box 412 surrounding the occupant 410. Cropping the original image 411 along the bounding box 412 yields the local image 413 containing the occupant 410.

[0037] In step S403, the size of the local image 413 is adjusted to the specified input size to obtain the input image 414. The specified input size can be the size required to match the instance segmentation model input, such as 256*256.

[0038] During the execution of steps S401-S403, certain additional processing may be performed on the original image 411, the local image 413 and the input image 414, such as geometric transformation, normalization, smoothing and / or image enhancement, but this disclosure is not limited to this.

[0039] In one embodiment, the attribute data related to the safety device calculated in step S403 may include the height, width, and rotation angle of the minimum bounding rectangle (MBR) of the safety device, as well as the pixel area of ​​the safety device. Please refer to Figure 5, which shows the minimum bounding rectangle 500 as an example and the pixel area A of the seat belt, wherein the minimum bounding rectangle has a height h, a width w, and a rotation angle. The rotation angle may be defined as the angle between a certain dimension (height or width) of the minimum bounding rectangle and a horizontal or vertical line, but this disclosure is not limited thereto. The pixel area A may be defined as the number of pixels contained in the area where the safety device is located, or as the area ratio of the area where the safety device is located relative to the minimum bounding rectangle 500, but this disclosure is not limited thereto. In one implementation example, the calculation of the minimum bounding rectangle and the pixel area may be implemented using functions provided by computer vision libraries such as OpenCV, PyTorch, TensorFlow, scikit-image, or MATLAB. For example, OpenCV's `cv2.minAreaRect()` takes a contour as input and returns a rotated rectangle that exactly encloses the entire contour. OpenCV's `cv2.contourArea()` takes a contour as input and returns the pixel area of ​​the region enclosed by that contour.

[0040] Furthermore, step S401 further includes checking whether the height h, width w, rotation angle, and pixel area A of the minimum bounding rectangle 500 are all within their respective specified ranges. The setting of the specified range can be determined according to the actual application scenario, and this disclosure does not limit it. For example, the judgment conditions can be set as follows: the height h falls between 50 and 400 pixels in length, the width w falls between 100 and 300 pixels in length, the rotation angle falls between 30 and 100 degrees, and the pixel area A occupies more than 70% of the area of ​​the minimum bounding rectangle 500. If all the above conditions are met, it is determined that the driver and passengers are equipped with safety equipment; if any condition is not met, it is determined that the driver and passengers are not equipped with safety equipment.

[0041] Figure 6 is a system block diagram of a system 60 for implementing the method 10 described above, according to one embodiment of the present disclosure. System 60 can be any computer system with computing power, such as a personal computer (e.g., a desktop or laptop computer) or a server computer, or a mobile device such as a tablet or smartphone, or an embedded system designed to handle specific tasks with relatively limited computing resources; however, the present disclosure is not limited to this. In one embodiment, system 60 is an embedded system using a board support package (BSP) that supports a relatively limited range of neural network types.

[0042] System 60 includes at least a processing device 602 to perform the steps S101-S104, S402-S403 and their various embodiments described above. The processing device 602 may include any one or more general-purpose or special-purpose processors or combinations thereof for executing instructions, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a system on a chip (SoC), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or combinations thereof, but this disclosure is not limited thereto.

[0043] In one embodiment, the system 60 further includes a photographic device 601 to perform the above-described step S401, i.e., capturing an original image. The photographic device 601 may include a lens and a conversion element. The lens may include one or more lenses, such as a zoom lens for magnifying or reducing the size of a target object, and a focusing lens for adjusting the focal length of the target object. The conversion element may be, for example, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) for receiving light signals from the lens and converting the light signals into electrical signals. In addition, the photographic device 601 may communicate with the processing device 602 through various wired or wireless communication interfaces, such as Universal Serial Bus (USB), Wi-Fi, Ethernet, Bluetooth, or Near Field Communication (NFC), to transmit the captured image data to the processing device 602 for subsequent processing and analysis.

[0044] In one embodiment, system 60 further includes an output device 603 to issue a warning message when it detects that a driver or passenger is not wearing safety equipment. The warning message can be in various forms, including but not limited to text, sound, vibration, or light. The recipient of the warning message can be the driver or passenger themselves or relevant supervisory personnel. The output device 603 can be a device that communicates to the driver or passenger or supervisory personnel through visual, auditory, or tactile means, such as a display that outputs text messages, a sound alarm that outputs sound messages, a vibration alarm that outputs vibration messages, or a warning light that outputs light signals. The output device 603 can communicate with the processing device through various communication interfaces, such as High Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), Wi-Fi, or Bluetooth. In response to determining that a driver or passenger is not wearing safety equipment, the processing device 602 causes the output device 603 to issue a warning message.

[0045] The preceding paragraphs use multiple models for description. Clearly, the teachings herein can be implemented in various ways, and any particular architecture or functionality disclosed in the examples is merely representative. Based on the teachings herein, it should be understood in the art that the individual models disclosed herein can be implemented independently, or that two or more models can be implemented in combination.

[0046] Although the present disclosure has been described above with reference to embodiments, it is not intended to limit the present disclosure. Anyone skilled in the art may make some modifications and refinements without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the invention shall be determined by the claims defined in the appended claims. [Simplified Explanation of the Diagram]

[0047] This disclosure will be better understood from the following description of exemplary embodiments in conjunction with the accompanying drawings. Furthermore, it should be understood that the execution order of the blocks in the flowcharts of this disclosure may be changed, and / or certain blocks may be modified, deleted, or merged. Figure 1 is a flowchart of a method for detecting a security device according to one embodiment of this disclosure. Figure 2 illustrates an architecture diagram of a typical SparseInst network. Figure 3 is an architecture diagram of an improved SparseInst network according to one embodiment of this disclosure. Figure 4A is a flowchart of more detailed steps for obtaining an input image containing occupants in one embodiment. Figure 4B is a schematic diagram of more detailed steps for obtaining an input image containing occupants in one embodiment. Figure 5 shows, as an example, a minimum bounding rectangle 500 and the pixel area A of a seatbelt, wherein the minimum bounding rectangle has a height h, a width w, and a rotation angle. Figure 6 is a system block diagram of a system for implementing a method for detecting a security device according to one embodiment of this disclosure.

Claims

1. A method for detecting a security device, implemented by a computer system, the method comprising: Obtain an input image containing a driver and passenger; The input image is input into a pre-trained instance segmentation model to obtain the instance segmentation result output by the instance segmentation model; based on the instance segmentation result, attribute data associated with a safety device is calculated; and based on the attribute data, it is determined whether the occupant is equipped with the safety device; wherein the instance segmentation model is implemented using an improved SparseInst network, the improved SparseInst network includes: a backbone module, which is involved in extracting multi-scale features of the input image using a backbone; an encoder module, which is involved in performing a series of feature fusion operations on these multi-scale features to obtain an encoded feature; and a decoder module, including: a mask branch, which is involved in deriving a mask feature from the encoded feature; and an instance branch, which is involved in deriving instance information from the encoded feature, the instance information including a core feature, a predicted class confidence, and a predicted object confidence; wherein the instance segmentation model further involves deriving the instance segmentation result from the mask feature and the instance information; Neither the encoder module nor the decoder module involves a coordinate concatenation operation; and the training of the instance segmentation model involves optimizing an improved SparseInst network using an improved loss function, which includes a focal loss, a mask loss, a binary cross entropy loss, and a heatmap loss.

2. The method as described in claim 1, wherein the backbone is implemented using a MobileNetV2 network, and the multi-scale features extracted from the backbone have a dimensionality of 128 dimensions.

3. The method as described in Request 1, wherein the encoder module does not involve a pyramid pooling module (PPM).

4. The method as described in Request 1, wherein the instance branch involves multiple convolutional layers but no downsampling layers, and the number of core features, predicted class confidences, and predicted object confidences derived by the instance branch depends on the size of the input image.

5. The method as described in claim 1, wherein obtaining the input image of the driver or passenger includes: A photographic device is used to capture an original image; the occupant in the original image is detected, and a partial image containing the occupant is cropped from the original image; and the size of the partial image is adjusted to a specified input size to obtain the input image.

6. The method as described in claim 1, wherein the attribute data includes a height, a width, and a rotation angle of a minimum bounding rectangle (MBR), and a pixel area; and wherein determining whether the occupant is equipped with the safety device based on the attribute data includes: Check whether the height, width, rotation angle, and pixel area of ​​the minimum bounding rectangle are all within their respective specified ranges.

7. The method as described in claim 1, further comprising: In response to the determination that the occupant is not equipped with the safety device, a warning message is issued.

8. The method as described in claim 1, wherein the computer system is an embedded system using a board support package (BSP).

9. A detection system for a safety device, comprising a processing unit, performing the following steps: acquiring an input image containing a driver or passenger; inputting the input image into a pre-trained instance segmentation model to obtain an instance segmentation result output by the instance segmentation model; calculating attribute data associated with a safety device based on the instance segmentation result; and determining whether the driver or passenger is equipped with the safety device based on the attribute data; wherein the instance segmentation model is implemented using an improved SparseInst network, the improved SparseInst network comprising: A backbone module is used to extract multi-scale features from the input image using a backbone; an encoder module is used to perform a series of feature fusion operations on these multi-scale features to obtain encoded features. And a decoder module, including: a mask branch, which involves deriving a mask feature from the encoded feature; And an instance branch, which involves deriving instance information from the encoded features, the instance information including a core feature, a predicted class confidence, and a predicted object confidence; wherein the instance segmentation model further involves deriving the instance segmentation result from the mask features and the instance information; wherein neither the encoder module nor the decoder module involves a coordinate concatenation operation; and wherein the training of the instance segmentation model involves optimizing the improved SparseInst network using an improved loss function, the improved loss function involving a focal loss, a mask loss, a binary cross entropy loss, and a heatmap loss.

10. The system as described in claim 9, wherein the backbone is implemented using a MobileNetV2 network, and the multi-scale features extracted from the backbone have a dimension of 128.

11. The system as described in claim 9, wherein the encoder module does not involve a pyramid pooling module (PPM).

12. The system as described in claim 9, wherein the instance branch involves multiple convolutional layers but no downsampling layers, and the number of core features, predicted class confidences, and predicted object confidences derived by the instance branch depends on the size of the input image.

13. The system as described in claim 9, further comprising: A photographic device that can communicate with the processing device to capture a raw image; wherein the processing device further detects the occupant in the raw image to crop a partial image containing the occupant from the raw image; and wherein the processing device further adjusts the size of the partial image to a specified input size to obtain the input image.

14. The system as described in claim 9, wherein the attribute data includes a height, a width, and a rotation angle of a minimum bounding rectangle (MBR), and a pixel area; and wherein the processing device further determines whether the occupant is equipped with the safety device by checking whether the height, the width, the rotation angle, and the pixel area of ​​the minimum bounding rectangle are all within their respective specified ranges.

15. The system as described in claim 9 further includes an output device: wherein, in response to determining that the occupant is not equipped with the safety device, the processing device causes the output device to issue a warning message.

16. The system as described in claim 9, wherein the system is an embedded system using a board support package (BSP).

17. A method for detecting a security device, implemented by a computer system, the method comprising: Obtain an input image containing a driver and passenger; The input image is input into a pre-trained instance segmentation model to obtain the instance segmentation result output by the instance segmentation model; based on the instance segmentation result, attribute data associated with a safety device is calculated; and based on the attribute data, it is determined whether the occupant is equipped with the safety device; wherein the instance segmentation model is implemented using an improved SparseInst network, the improved SparseInst network including: a backbone module, which is involved in extracting multi-scale features of the input image using a backbone; an encoder module, which is involved in performing a series of feature fusion operations on the multi-scale features to obtain an encoded feature; and a decoder module, including: a mask branch, which is involved in deriving a mask feature from the encoded feature; and an instance branch, which is involved in deriving instance information from the encoded feature, the instance information including a core feature, a predicted class confidence, and a predicted object confidence; The instance segmentation model further involves deriving the instance segmentation result from the mask features and the instance information; and the instance branch involves multiple convolution layers but no downsampling layers, and the number of core features, predicted class confidence and predicted object confidence derived from the instance branch depends on the size of the input image.

18. A method for detecting a security device, implemented by a computer system, the method comprising: Obtain an input image containing a driver and passenger; The input image is input into a pre-trained instance segmentation model to obtain the instance segmentation result output by the instance segmentation model; based on the instance segmentation result, attribute data associated with a safety device is calculated; and based on the attribute data, it is determined whether the driver or passenger is equipped with the safety device; wherein the attribute data includes a height, a width, and a rotation angle of a minimum bounding rectangle (MBR), and a pixel area; And based on the attribute data, determining whether the driver or passenger is equipped with the safety device includes: checking whether the height, width, rotation angle, and pixel area of ​​the minimum bounding rectangle are all within their respective specified ranges.

Images