Semantic-based image recognition method and device and image recognition system

Abstract

The invention provides an image recognition method and device based on semantics and an image recognition system. The method comprises the steps of obtaining a target image; extracting inter-class edges of the target object; determining an inter-class edge in the target image and an image region surrounded by the inter-class edge as a target object image, and identifying the target object image to obtain a class identification result of the target object; and sending the category identification result to a safety monitoring system of the financial institution, so that the safety monitoring system performs safety monitoring based on the category identification result. According to the scheme, the problem that in the prior art, the accuracy of identifying the target category of the image is low, so that the accuracy of safety monitoring of a financial institution is low is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a semantics-based image recognition method, a semantics-based image recognition device, a computer program product, and an image recognition system. Background Art

[0002] With the development of the financial market, regulators have put forward higher requirements for the sales process of financial products. "Dual recording", that is, audio and video recording, has become part of financial supervision to ensure transparency and compliance in the sales process.

[0003] The development of digital finance has driven the intelligentization of financial services. New technologies, such as digital imaging and audio, have provided new solutions for double-recording detection. Traditional manual double-recording detection is a thing of the past. Intelligent double-recording detection improves detection efficiency, saves auditors' time, increases efficiency, and reduces manual intervention. However, current solutions have low accuracy in identifying object categories in images, resulting in low security monitoring accuracy for financial institutions. Summary of the Invention

[0004] The main purpose of this application is to provide a semantic-based image recognition method, a semantic-based image recognition device, a computer program product and an image recognition system, so as to at least solve the problem in the prior art of low accuracy in identifying target categories of images, which leads to low accuracy in security monitoring by financial institutions.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a semantic-based image recognition method is provided, including: acquiring a target image, wherein the target image includes an image of a target object of a category to be identified and a background; extracting inter-class edges of the target object, wherein the inter-class edges include edge pixels of the target object and some pixels of an adjacent area of ​​the target object; determining the inter-class edges in the target image and the image area surrounded by the inter-class edges as the target object image, and identifying the target object image to obtain a category recognition result of the target object, wherein the category recognition result is one of a category of a person, an animal, and a category of an object; and sending the category recognition result to a security monitoring system of a financial institution, so that the security monitoring system performs security monitoring based on the category recognition result.

[0006] Optionally, obtaining a target image includes: obtaining a target video; extracting a frame from the target video at an interval of a predetermined number of frames to obtain a plurality of initial images; using a perceptual hash algorithm to calculate the similarity between any two of the plurality of initial images; deleting one of the two initial images whose similarity is greater than or equal to a similarity threshold, until the similarity between any two of the plurality of initial images is less than the similarity threshold, and determining that all the remaining non-deleted initial images are the target images.

[0007] Optionally, extracting the inter-class edge of the target object includes: performing edge detection on the target object to obtain an edge curve, wherein the edge curve is a curve composed of pixel points of the boundary of the target object detected; obtaining the normal of the edge curve, wherein there are multiple normals, one normal corresponding to one pixel point on the edge curve, and the normal is a straight line perpendicular to the tangent of the pixel point on the edge curve; expanding each pixel point on the edge curve by a preset number of pixels along the direction of the corresponding normal to obtain the inter-class edge.

[0008] Optionally, identifying the target object image to obtain a category recognition result of the target object includes: preprocessing the target object image to obtain a preprocessed object image, wherein the preprocessing method includes image segmentation and / or data expansion, wherein the data expansion method includes one or more of image flipping, changing image brightness, changing image contrast, and changing image hue; forming the target object image and / or the preprocessed object image into a training image set, using the training image set and corresponding category labels to train an image recognition model to obtain a category recognition model, wherein the category label is the category of the object in the image of the training image set; inputting the target object image into the category recognition model to obtain the category recognition result corresponding to the target object image.

[0009] Optionally, before using the training image set and the corresponding category labels to train the image recognition model to obtain the category recognition model, the method also includes: obtaining the image recognition model to be trained, which is one of the HRNet model, U-NET model, and GAN model; adding an attention mechanism module before the decoder layer of the image recognition model to be trained, so that the output of the attention mechanism module is used as the input of the decoder layer to obtain the image recognition model.

[0010] Optionally, preprocessing the target object image to obtain a preprocessed object image includes one of the following: performing image segmentation or data expansion on the target object image to obtain the preprocessed object image; performing image segmentation on the target object image and performing data expansion on the target object image after image segmentation to obtain the preprocessed object image; performing data expansion on the target object image and performing image segmentation on the target object image after data expansion to obtain the preprocessed object image.

[0011] Optionally, the target object image is segmented to obtain the preprocessed object image, including: when the target image includes one target object, cutting out a portion of the target object image to obtain the preprocessed object image, so that the preprocessed object image contains the complete inter-class edge; when the target image includes multiple target objects, segmenting the target object image to obtain multiple preprocessed object images, so that each preprocessed object image contains the complete inter-class edge of at least one target object.

[0012] According to another aspect of the present application, a semantic-based image recognition device is provided, comprising: a first acquisition unit for acquiring a target image, wherein the target image comprises an image of a target object of a category to be identified and a background; an extraction unit for extracting inter-class edges of the target object, wherein the inter-class edges comprise edge pixels of the target object and some pixels of an adjacent area of ​​the target object; an identification unit for determining the inter-class edges and the image area surrounded by the inter-class edges in the target image as a target object image, and identifying the target object image to obtain a category recognition result of the target object, wherein the category recognition result is one of a category of a person, an animal, and a category of an object; and a processing unit for sending the category recognition result to a security monitoring system of a financial institution, so that the security monitoring system performs security monitoring based on the category recognition result.

[0013] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any one of the semantic-based image recognition methods.

[0014] According to another aspect of the present application, an image recognition system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include means for executing any one of the semantic-based image recognition methods.

[0015] By applying the technical solution of the present application, after acquiring the target image, it is necessary to identify the boundary area between the target object and the non-target object in the target image. This area is regarded as the edge area, that is, the inter-class edge. This can enhance the edge representation without missing the pixel points of the target object. The image semantics is the meaning expressed by the image content. Therefore, this solution uses image semantic recognition to identify and process the target object image. In addition, since the inter-class edge of the target object is determined in advance in this solution, the edge representation of the target object is enhanced. In this way, when identifying the target object image, the edge of the target object in the target object image will not be ignored, and the recognition can be performed according to the complete pixel area of ​​the target object, thereby improving the recognition accuracy. In addition, the security monitoring system of the financial institution can perform security monitoring more accurately based on the accurate category recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a semantic-based image recognition method provided in an embodiment of the present application is shown;

[0018] Figure 2 A flowchart of a semantic-based image recognition method provided according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of the process of perceptual hash deduplication is shown;

[0020] Figure 4 A schematic diagram of the inter-class edges is shown;

[0021] Figure 5 A schematic diagram showing the feature flow of the category recognition model of this solution;

[0022] Figure 6 A schematic diagram showing the branches of semantic edge detection;

[0023] Figure 7 shows a schematic diagram of the feature pyramid module;

[0024] Figure 8 A schematic diagram of the attention mechanism module is shown;

[0025] Figure 9 A schematic diagram of the attention mechanism module for decoding is shown;

[0026] Figure 10The figure shows a structural block diagram of a semantic-based image recognition device provided according to an embodiment of the present application.

[0027] The above drawings include the following reference numerals:

[0028] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0033] Semantic segmentation: It is an image processing technology that classifies images at the pixel level. It divides the image into blocks with specific semantic meanings and identifies the semantic category of each block.

[0034] Semantic edge detection: is a technology that combines edge detection and classification recognition. It not only focuses on the location of the edges of objects in the image, but also involves the semantic understanding of the edges, that is, identifying the category to which the edges belong.

[0035] Dual recording: specifically refers to audio and video recording during the monitoring process.

[0036] The existing dual-recording video intelligent detection solutions mainly rely on target detection algorithms and posture detection algorithms.

[0037] The intelligent detection solution based on target detection first uses a sliding window or region proposal network to obtain the region of interest where the target may exist in the digital image to reduce the interference of background information; then it uses a deep convolutional network to extract and fuse features, combining low-level and high-level features during prediction; then it performs classification (determining what the target is) and regression (determining the target location) based on the extracted features; finally, the model selects the best bounding box from the multiple predicted bounding boxes through non-maximum suppression.

[0038] The intelligent detection solution based on posture detection first extracts high-level semantic information from the digital image through a convolutional neural network. It then uses a posture estimation algorithm to detect key points in the image. Key points are usually the main joints of the human body. Based on the detected key points, the relative position relationship between each key point is determined through regression analysis to reconstruct the complete human posture. Finally, the predicted posture is optimized by smoothing, correcting the key point positions, and other optimizations to improve the accuracy of the posture.

[0039] A comprehensive analysis of the shortcomings of existing dual-recording video detection solutions is as follows:

[0040] 1) Object detection in dual-recording video detection has problems including difficulty in accurately capturing the target position and boundary information, limited ability to handle target occlusion and scale changes, and susceptibility to background interference, resulting in low detection accuracy.

[0041] 2) Object detection tasks are usually constrained by speed and timeliness, and self-attention mechanisms are not used to capture the spatial relationships between different positions and the feature correlations between different feature channels, which leads to a loss of model performance and thus causes misclassification or non-detection problems.

[0042] 3) Most existing algorithms only consider abstract features related to classification, and rarely consider underlying image features such as edges and textures, which leads to confusion in the model's recognition of object edge areas and difficulty in segmenting similar categories.

[0043] As introduced in the background technology, the accuracy of target category recognition in images in the existing technology is low, resulting in low accuracy of security monitoring by financial institutions. To solve the above problem, the embodiments of the present application provide a semantic-based image recognition method, a semantic-based image recognition device, a computer program product and an image recognition system.

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0045] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure diagram of a mobile terminal for a semantic-based image recognition method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0046] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the device information display method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0047] In this embodiment, a semantic-based image recognition method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Figure 2 FIG. 1 is a flow chart of a semantic-based image recognition method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0049] Step S201, obtaining a target image, wherein the target image includes an image of a target object of a category to be identified and a background;

[0050] Specifically, a camera or image acquisition device captures an internal scene (eg, a scene inside a bank), including customers, employees, a counter, and a background. The target image includes multiple objects, and the target object is one of the multiple objects.

[0051] Step S202: extracting the inter-class edges of the target object, wherein the inter-class edges include edge pixels of the target object and some pixels of an adjacent area of ​​the target object;

[0052] Specifically, the inter-class edge refers to the boundary between the target object and the background or its adjacent area, which includes the target edge and some background pixels. It can enhance the understanding of boundary details and improve the accuracy of recognition.

[0053] Step S203: determining the inter-class edge and the image region enclosed by the inter-class edge in the target image as a target object image, and performing recognition on the target object image to obtain a category recognition result of the target object, wherein the category recognition result is one of a person, an animal, and an object.

[0054] Specifically, focusing on edge information and surrounding areas within a target image and classifying these areas can significantly improve the accuracy of identifying people, animals, or objects. Edges provide precise location information, while semantic segmentation focuses on pixel-level classification. The combination of these two allows the model to more accurately locate and identify objects, maintaining high performance even in complex environments.

[0055] Step S204: sending the category identification result to the security monitoring system of the financial institution, so that the security monitoring system performs security monitoring based on the category identification result.

[0056] Specifically, identification results (such as the categories of people, animals, and specific items) are transmitted via a network interface to the financial institution's security monitoring system. Based on the received category recognition results, the system automatically triggers appropriate security rule checks. If a suspicious object or unauthorized animal is detected, the system will issue an alarm. Through real-time object classification, the monitoring system can promptly detect anomalies such as illegal intrusion or the presence of dangerous items, allowing immediate action to ensure the safety of personnel and assets.

[0057] In the prior art, the edges of objects in the image are often ignored during target category recognition, which results in a decrease in the accuracy of target category recognition of the image. Through this embodiment, after obtaining the target image, the boundary area between the target object and the non-target object in the target image is identified. This area is used as the edge area, that is, the inter-class edge. This can enhance the edge representation without missing the pixel points of the target object. The image semantics is the meaning expressed by the image content. Therefore, this scheme uses image semantic recognition to identify the target object image. In addition, since the inter-class edge of the target object is determined in advance in this scheme, the edge representation of the target object is enhanced. In this way, when identifying the target object image, the edge of the target object in the target object image will not be ignored, and the recognition can be performed according to the complete pixel area of ​​the target object, thereby improving the recognition accuracy. In addition, the security monitoring system of the financial institution can perform security monitoring more accurately based on the accurate category recognition results.

[0058] Specifically, after receiving the category recognition result, the security monitoring system determines whether to take action based on pre-set security rules and the recognition result. For example, if the recognized category is "Unknown Person" or "Unauthorized Animal," the system automatically triggers an alarm and notifies the nearest security team member via the internal communication system for on-site verification. For example, at 10 p.m., a camera in a bank's cash vault captures an unauthorized person, such as a stray cat, entering the business hall during business hours for more than five minutes (a preset threshold). The system immediately triggers an alarm, notifies on-site security personnel to handle the situation, and logs the incident for subsequent investigation. The category recognition result ("Unknown Person") is quickly transmitted to the security monitoring center via the secure network. Based on pre-set security rules, the system identifies this as a high-risk event and immediately triggers an alarm, logging event details including a timestamp, camera ID, and detailed location. The security monitoring center notifies nearby security team members via the internal communication system, requesting them to arrive for on-site verification within five minutes. In high-risk scenarios, such as a cash vault during off-hours, any unidentified person is considered a potential threat and requires immediate response.

[0059] For example, setting the threshold time for not wearing an ID card to 10 minutes allows the system to more accurately distinguish between normal and abnormal behavior, avoiding false alarms or missed alarms due to improper threshold settings. Setting 10 minutes as the threshold time for not wearing an ID card is based on the normal behavior statistics of bank employees at their posts, that is, most employees will put on their ID cards again within 5 minutes or be reminded to do so by colleagues when they are not wearing them. Therefore, if an employee does not wear an ID card for 10 minutes, it is considered abnormal behavior. The setting of this threshold not only takes into account the actual operating habits of employees, but also ensures the accuracy of security monitoring, avoids excessive alarms for short periods of not wearing an ID card, but also can promptly capture abnormal situations where the ID card is not worn for a long time, which plays a positive role in the bank's security management.

[0060] The system uses video stream analysis to identify and categorize items involved in transactions, such as large amounts of cash and important documents. For transactions involving large amounts of cash, a cash threshold of 50,000 yuan is set. If the amount of cash detected by the system exceeds this threshold, the compliance review process is automatically triggered, including verifying the transaction background and confirming the customer's identity. For the signing of important documents, the system can identify whether the customer is wearing glasses (assuming the threshold time is 2 minutes, which is considered an anomaly) and whether the employee has displayed their ID card, ensuring the safety and compliance of the transaction environment.

[0061] Through video surveillance, the system automatically identifies employees entering back-office areas and the types of items they carry, such as toolboxes and personal bags. For employee-carried items, thresholds for unauthorized items are set. For example, the presence of a toolbox in the archives room is considered an anomaly. A threshold for non-employees is also set. For example, the presence of an animal in the vault is considered an emergency. If the number of unauthorized items or non-employees in the identification results exceeds the set threshold, the system immediately notifies the security supervisor for intervention and records the incident details to improve future security management measures.

[0062] Specifically, the solution of the present application mainly includes the following steps: pulling the video stream; extracting frames and deduplicating the video stream; image preprocessing; and inputting the frame images into the category recognition model, which are described below respectively.

[0063] During the specific implementation process, obtaining the target image can be achieved through the following steps: obtaining a target video; extracting a frame from the target video once at an interval of a predetermined number of frames to obtain multiple initial images; using a perceptual hash algorithm to calculate the similarity between any two of the multiple initial images; deleting one of the two initial images whose similarity is greater than or equal to a similarity threshold, until the similarity between any two of the multiple initial images is less than the similarity threshold, and determining that all the remaining initial images that have not been deleted are the target images.

[0064] In this solution, reducing the number of frames of video processing means reducing the amount of data and computing power, thereby speeding up the subsequent similarity calculation and image screening process. When two frames of images are highly similar in structure and content, they will be considered redundant, thereby deleting duplicate information and ensuring that the image set received by the security monitoring system is minimally redundant. The deduplicated image set reduces the demand for computing resources.

[0065] Specifically, VLC or a similar streaming framework can be used to capture real-time surveillance video inside the bank. The video stream retains only the image information and blocks the audio stream, reducing bandwidth and CPU pressure and improving data processing efficiency.

[0066] The captured video stream is processed by extracting frames, for example, 5 frames out of every 25 frames, to generate a series of initial images. By reducing the number of frames to be processed, the computational cost and processing time can be significantly reduced while retaining sufficient information for subsequent image analysis.

[0067] A perceptual hashing algorithm is used to calculate the similarity between any two initial images. For example, if the similarity threshold is set to 0.9, one of the frames will be deleted if the similarity between any two initial images is greater than or equal to 0.9. This process continues until the similarity between all remaining images is less than 0.9. After this similarity screening, the remaining initial images are designated as target images, which are used for subsequent category recognition and security monitoring.

[0068] Specifically, during the video stream pulling process, since the solution of this application does not involve the recording part of the dual-recording detection, the model input only considers image information and does not consider multimodal detection. The streaming media framework for video stream processing selects VLC, and the audio stream is shielded during the stream pulling process to reduce bandwidth consumption and CPU pressure. In order to reduce service delays and release hardware resources, due to the continuity of the action, it is not necessary to process all frames. Frame processing is performed during the stream pulling process. For example, 15 frames, 10 frames or 5 frames of a 25-frame video are taken for processing.

[0069] Evenly extracting frames relieves the computational pressure, but the result may leave the image basically unchanged. There is still redundancy in the task, so the perceptual hashing algorithm is used to remove duplicates, such as Figure 3 As shown, the perceptual hashing algorithm includes the following steps.

[0070] 1) Size reduction: To remove high-frequency information and details from an image and retain only the structural light and dark information, the image needs to be reduced to a certain size, usually 8x8 pixels. This can reduce the amount of computation while retaining the main features of the image.

[0071] 2) Convert to grayscale image: Convert the scaled image to grayscale image and convert the color image to 256-level grayscale image for subsequent processing.

[0072] 3) Calculate the Discrete Cosine Transform (DCT): Perform a discrete cosine transform (DCT) on the grayscale image to decompose the image into a set of frequencies. In the resulting DCT matrix, only the upper left 8x8 portion, representing the main content of the image, is retained.

[0073] 4) Extract low-frequency components: Extract low-frequency components representing the main information of the image from the matrix after DCT transformation. These components contain the basic structure and contour information of the image.

[0074] 5) Generate and compare hash values: Calculate the hash value based on the extracted low-frequency components, and determine the similarity by comparing the hash values ​​of different images.

[0075] The above solution uses the VLC tool to pull the stream, deduplicates the frame images through the perceptual hash deduplication algorithm and uniform frame extraction strategy, and obtains the category to which the image pixels belong through the semantic segmentation algorithm.

[0076] In some embodiments, extracting the inter-class edge of the target object can be achieved specifically by the following steps: performing edge detection on the target object to obtain an edge curve, wherein the edge curve is a curve composed of pixel points of the boundary of the target object detected; obtaining the normal of the edge curve, wherein there are multiple normals, and one pixel point on the edge curve corresponds to one normal, and the normal is a straight line perpendicular to the tangent of the pixel point on the edge curve; expanding each pixel point on the edge curve by a preset number of pixels along the direction of the corresponding normal to obtain the inter-class edge.

[0077] In this scheme, by appropriately expanding the edge curve, it can be ensured that the obtained inter-class edge can capture all key edge information without over-expanding and causing excessive background interference, which can further enhance the representation of edge information.

[0078] Specifically, edge detection algorithms, such as Canny edge detection or deep learning-based edge detection networks, are used to process the image of the target object and extract the target object's edge curve. This edge curve is a collection of pixels that accurately depicts the target object's outline, marking the boundary between the target and the background.

[0079] Specifically, the semantic segmentation task labels each pixel's category, allowing us to find an edge line connecting several pixels, known as an edge curve. However, to avoid overly thin edge curves, this line can be expanded by five pixels along the normal direction. Furthermore, the edge of each category is captured, resulting in overlapping transition zones. The overlapping areas are considered to belong to the edges of multiple overlapping classes. For example, if a person and a tree overlap, the line between them is expanded into an edge band, and the pixels within this edge band are considered to be both the person's and the tree's edges. Ultimately, a loss function is used to determine whether an object belongs to a person or a tree.

[0080] The inter-class edges include not only the clear boundaries of the target objects but also the transition regions near the boundaries, which helps the model distinguish the target objects from the background more accurately, especially in scenarios with small inter-class differences.

[0081] Specifically, in the model prediction process, the semantic edge detection branch is first used to find inter-class edges. The extracted inter-class edges are then fused with binary edges (using methods such as pyramid decomposition and wavelet transform). This is explicitly shared with the downsampling stage of feature extraction in the semantic segmentation branch. This allows the model to focus more on edge regions that are prone to misclassification and enhances the representation of edge information. The feature maps that incorporate edge information are then fused through multiple parallel feature branches, allowing the output to take into account features of different scales and prevent small objects from being buried in deep models.

[0082] In post-processing, the model predicts a multi-channel feature map, where each channel represents the probability of belonging to a specific category. The maximum channel probability across all positions in the frame is taken to determine the category to which a single pixel belongs. Subsequently, based on the pixel distribution, rules are applied to determine whether someone has left the table, whether the manager's badge is displayed, whether the job qualification certificate is presented, and whether the customer's signed receipt is presented. Furthermore, based on the extracted badge and receipt information, downstream visual tasks can be added, such as using optical character recognition (OCR) technology to determine whether the name on the job qualification certificate matches the account manager's.

[0083] Specifically, the edge-guided semantic segmentation model (i.e., image recognition model) involves the following algorithmic implementations: overlapping image segmentation; extraction of image edge transition zones (i.e., inter-class edges); image expansion; extraction of inter-class edges using a semantic edge detection branch; and enhanced edge representation of the segmentation model using edge guidance. Each of these is described below.

[0084] In the process of extracting edges between classes, the number of pixels on the edge of the object's contour differs too much from that of the entire image, which makes model training difficult and the edge prediction results too wide. Even if manual annotation is used to divide the edge area, there will be large errors, which means that the edge labels can only be obtained by morphological processing of semantic segmentation labels. The contour edge of an object is not just a closed contour curve, and the edge between classes is not two or more non-intersecting curves, but a transition zone with an intersection. The pixels in the intersection area of ​​multiple category contours can belong to all categories of the intersection set. At the same time, the use of a transition zone will exponentially expand the number of pixels belonging to the contour edge in the image, which facilitates model training and optimization. In the design of the transition zone, for each edge point of the edge curve, five pixels are selected along the normal direction as the transition zone area to obtain the edge between classes. The range of the pixel points of the edge between classes is as follows: Figure 4 shown.

[0085] In a specific implementation process, the target object image is identified to obtain a category recognition result of the target object, which can be achieved by the following steps: preprocessing the target object image to obtain a preprocessed object image, wherein the preprocessing method includes image segmentation and / or data expansion, wherein the data expansion method includes one or more of image flipping, changing image brightness, changing image contrast, and changing image hue; forming the target object image and / or the preprocessed object image into a training image set, using the training image set and the corresponding category label to train an image recognition model to obtain a category recognition model, wherein the category label is the category of the object in the image of the training image set; inputting the target object image into the category recognition model to obtain the category recognition result corresponding to the target object image.

[0086] In this solution, a more comprehensive and varied training set can be created through image segmentation and data expansion. This way, there is enough training data for model training, which increases the diversity of the training set. During the training process, the model can be exposed to target objects under various transformation conditions. This enables the model to better understand the shape, color, and texture of the object when facing new images, maintain a high classification accuracy, and thus improve the recognition accuracy.

[0087] Specifically, the target object image is first preprocessed, including image segmentation and data augmentation. Image segmentation breaks large images into smaller pieces, facilitating model training and processing. Data augmentation increases the diversity of the training set and improves the model's generalization capabilities by performing operations such as image flipping, changing image brightness, image contrast, and image hue.

[0088] The training set consists of preprocessed and target object images, along with corresponding category labels, for model training. The preprocessing of the images in the training set ensures that the model can learn the characteristics of the target object from different angles and conditions, improving recognition accuracy.

[0089] The training image set is constructed based on preprocessed images, including original images and new images obtained by flipping, adjusting brightness, changing contrast, modifying hue, etc. Class labels correspond to the target category in each image and are used to guide model learning.

[0090] The target object image is input into the trained category recognition model. The model outputs the category probability distribution of each pixel point, selects the category with the highest probability as the category of the pixel, and finally obtains the category recognition result of the target object.

[0091] For example, by varying the image brightness, we can simulate different lighting conditions. This is particularly critical for surveillance systems inside banks, where lighting can fluctuate over time and with the weather. By setting a brightness adjustment range of ±30%, the model learns the characteristics of target objects under varying brightness conditions during training. This allows for robust target recognition in real-world applications, even in low- and high-light environments.

[0092] Specifically, due to the limitation of video memory and in order to avoid the loss of image features due to downsampling during the training process, the original image is segmented using overlapping image segmentation. The choice of overlapping segmentation scheme ensures that small objects such as the manager's ID card and signed documents can be completely retained in the segmented image.

[0093] Specifically, during data augmentation, images are cropped and flipped to change the position of objects within the image, reducing the model's sensitivity to target locations. Parameters such as brightness, contrast, and hue are also modified to reduce the model's sensitivity to color and mitigate the impact of shadows on feature extraction.

[0094] In some embodiments, before using the above-mentioned training image set and the corresponding category labels to train the image recognition model to obtain the category recognition model, the above-mentioned method also includes the following steps: obtaining the image recognition model to be trained, and the above-mentioned image recognition model to be trained is one of the HRNet model, U-NET model, and GAN model; adding an attention mechanism module before the decoder layer of the above-mentioned image recognition model to be trained, so that the output of the above-mentioned attention mechanism module is used as the input of the above-mentioned decoder layer to obtain the above-mentioned image recognition model.

[0095] In this scheme, an attention mechanism module is added before the decoder layer of the selected image recognition model to be trained. The attention mechanism module can enhance the model's attention to key areas in the image, ensuring that the model prioritizes high-weight information during upsampling and decoding, thereby avoiding the loss of edge information and ensuring a high recognition accuracy of the model.

[0096] Specifically, the attention mechanism module calculates the importance of different regions in the image and adjusts the model's weights for these regions, allowing for more efficient processing of image information. By adding the attention mechanism before the decoder layer, the model can focus more on edges and key feature areas, thereby improving recognition and segmentation accuracy when dealing with objects with small inter-class differences.

[0097] In practice, adding an attention mechanism is crucial for improving model performance. For example, the HRNet model was chosen as the base model. HRNet is renowned for its excellent performance in multi-scale feature fusion. Adding an attention mechanism module before its decoder further enhances the model's feature representation capabilities, particularly for edge and small object detection. This attention mechanism module can include a spatial attention module and a channel attention module.

[0098] The spatial attention module captures long-range relationships in image space, which is particularly useful for processing surveillance videos inside banks. For example, when a target subject (such as an account manager's ID card or a briefcase) appears in a video, the spatial attention mechanism can highlight these key areas, ensuring accurate recognition even if they occupy a small proportion of the image.

[0099] The channel attention module focuses on the channel dimension in the feature map, which can strengthen the model's understanding of the dependencies between feature channels. In the banking environment, this mechanism is crucial for distinguishing similar backgrounds or objects (such as folders of different brands) because they may be very similar visually.

[0100] Based on edge detection, semantic segmentation algorithms (such as FCN, Mask R-CNN, or DeepLab) can also be applied to build a trained image recognition model to classify the areas defined by the edges between classes in the target image and obtain the category recognition results of the target object. These algorithms can determine the category of each pixel or pixel group based on pixel-level information.

[0101] Specifically, in the design of the overall model, the overall model consists of a parallel edge detection branch and a semantic segmentation branch, which are independent of each other. The input image passes through a deep learning edge detection network to generate a feature map containing inter-class edge information. The generated feature map is then explicitly integrated into the downsampling process of feature extraction in the semantic segmentation branch to improve the problem of edge area destruction caused by the reduction in resolution during the downsampling process, thereby improving the model's ability to distinguish edge areas and their surrounding pixels, and solving the problem of small inter-class differences.

[0102] The overall flow of the feature map of the edge-guided remote sensing image semantic segmentation model (i.e., image recognition model) is as follows Figure 5 As shown in the figure (Feature Extraction represents the feature extraction network layer, such as the CNN layer, Edge represents the edge detection result, that is, the inter-class edge, the inter-class edge is input to the branch of image semantic detection, Segmentation represents semantic segmentation, that is, detecting the category of the target object), the feature map obtains the inter-class edge through the semantic edge detection branch, strongly supervises the edge label at the tail of the edge branch, and is explicitly fused with the downsampling link of the feature extraction stage of the semantic segmentation branch to tell the model where the edge is, so as to enhance the model's attention to the edge area.

[0103] Specifically, the model adopts a top-down design structure in the design of the semantic edge detection branch, such as Figure 6 As shown, a Channel Spatial Attention Module (CSM) is added to each stage of the feature extraction network to aggregate spatial context and capture feature importance. The resulting feature map is stage = {stage1,...,stage5}. Each pair of adjacent feature maps is then fed into a Feature Fusion Module (FFM). This fusion of two different-level feature maps enhances high-level feature representation and improves edge transitivity. Finally, a 1×1 convolution is used to perform channel dimensionality reduction, generating low-level binary edges and high-level semantic edges. These are then fed into an Adaptive Fusion Module (AFM), which uses adaptive learning to assign different weights to different locations of each edge slice.

[0104] The input feature map passes through three parallel branches to obtain inter-class edges. The left branch is the feature extraction process. The model gradually integrates multi-scale features from different layers of the high-resolution image through the right branch, and the final prediction task is placed at the end of the model. At this time, high-level semantic information is generated by the right branch combined with low-level spatial information. This design takes into account both edge localization and edge classification tasks. The feature map used for classification is gradually enhanced with edge details.

[0105] Specifically, the semantic segmentation branch is an encoder-decoder network, mainly including a spatial attention module and a channel attention module. The backbone of the model encoder is based on HRNet, and the designed modules are built on the four parallel branches of HRNet. HRNet only serves as a feature extraction network and does not output training results from the tail. In the decoder part, a spatial attention mechanism is added to the first two branches to capture long-range relationships in the object space to aggregate spatial context information, and a channel attention mechanism is added to the last two branches to capture the dependencies of channel-dimensional features. Finally, the feature maps that have passed through the two attention mechanism modules are integrated through cascading operations and upsampling operations. The encoder and decoder of the model are trained through joint supervision through auxiliary loss and main loss.

[0106] For the self-attention mechanism, the parameters are simplified according to the following formula: R represents the feature map, N represents the image height by width, and C represents the number of channels. S represents a number much smaller than N.

[0107] That is, when sampling the feature map, instead of using all the feature points, we only use the feature points with significant characteristics. This will greatly reduce the computational cost and will not affect the size of the feature map after the non-local block. The computational complexity of matrix multiplication is reduced from O(CN 2 ) becomes O(CNS).

[0108] Taking spatial attention as an example, the feature map A∈R C×H×W Three 1×1 convolutional layers are used to generate three feature maps Q (query), K (key) and V (value), where H represents the image height and W represents the image width, where {Q, K, V}∈R C×N Then, the feature maps K and V are sampled through parallel strided convolution and adaptive pooling. The downsampled feature maps obtained in the model of this paper have heights of 1, 3, 8, and 16 respectively. The number of key points is calculated as follows: S = ∑ w∈{1,3,8,16} w 2 =270.

[0109] The global information of the image is aggregated by global average pooling to obtain a 1×1 feature map, 3×3 and 8×8 feature maps are obtained by adaptive pooling, and a feature map of size 16×16 is obtained by strided convolution. The feature pyramid is as follows: Figure 7 As shown in the figure (Max Pooling represents the maximum pooling operation, Adaptive Pooling represents the adaptive pooling operation, Strided Conv represents the strided convolution or the strided convolution, in which the step size of each movement is greater than 1, and Flatten Concatenation represents the conversion of a multi-dimensional feature map into a one-dimensional feature vector).

[0110] Afterwards, the generated multiple feature maps are expanded and spliced, such as Figure 8 As shown, (Conv1×1 represents 1x1 convolution, Feature Pyramid represents feature pyramid, and Flatten represents converting a multidimensional array or tensor into a one-dimensional array) the feature maps K and V are reconstructed as {K, V}∈R C×S , where S<<N, then perform matrix multiplication on the transposed matrix of the feature map Q and the feature map K, and normalize them through the Softmax layer to obtain the correlation map T S ∈R N×S , T S Each item in is the correlation information between each significant position and other positions, and then the correlation map T S Perform matrix multiplication and reconstruction with the transposed matrix of the feature map V to obtain the output feature map E of the improved spatial self-attention mechanism SSA SSA ∈R C×H×W , and finally output the feature map E S Multiplying the original feature map A by a learnable parameter α and performing element-wise summation, assigning weights to the original feature map in a learnable manner. This module calculates the correlation information of each salient position in the space, captures long-range information in the space, and maintains semantic consistency.

[0111] Compared with the traditional self-attention mechanism module, the correlation map calculated by the above method only selects pixels at significant positions instead of all positions in the space, which will lead to performance degradation. To compensate for this shortcoming, this paper designs a spatial decoding attention mechanism, which can improve the performance of the entire spatial attention mechanism without significantly increasing the amount of computation. Spatial Decoder Attention (SDA) Figure 9 shown.

[0112] The output of the spatial decoding attention mechanism consists of feature maps from two different layers of the feature extraction network, namely the high-level map A high With the low-level mapping A low In the model designed in this scheme, the output result of the second branch of the feature extraction network is The result after upsampling is taken as A high , the output of the first branch Treat as A low , It is the phased output of the semantic segmentation branch feature extraction network. The attention mechanism is loaded onto all phased outputs and is an intermediate result. Represents the result of the edge detection branch output. Feature map A highGenerate vector through 1×1 convolution C h Indicates the number of high-resolution feature channels, N h Represents high-resolution feature height by width, feature map A low After 3×3 convolution, the channel dimension is increased to generate a vector C h Indicates the number of high-resolution feature channels, N l Represents the low-resolution feature height by width, and then the feature maps K and V are reconstructed into The transposed matrix of the feature map Q is matrix multiplied with the feature map K, and then the correlation information of the high-level map and the low-level map is obtained through Softmax normalization. Then the correlation map is transposed, multiplied and reconstructed with the feature map V to obtain the output result of the spatial decoding attention mechanism. H l Indicates that low-resolution features are high, W l Indicates the low-resolution feature width, and then the result E SDA By channel splicing and the original feature map Combined and used as the output of the subsequent module.

[0113] In the aforementioned solution, the semantic edge detection branch of the edge-guided semantic segmentation model adopts a top-down architecture, integrating features through two parallel branches. Low-level information guides high-level information selection, and low-level binary edges are fused with high-level semantic edges through dimensionality reduction after shared connections. The semantic segmentation branch shifts the computational paradigm of the self-attention mechanism, eliminating the need to calculate correlations across all points and instead focusing on the relationships between pixels relevant to the classification and each position. Furthermore, the semantic segmentation branch fuses multi-level information through the same decoding attention mechanism to mitigate the performance loss associated with parameter reduction.

[0114] Specifically, this solution introduces a multi-scale branched parallel HRNet to address the limitations of the encoder-decoder structure, reduce the loss of spatial information, and aggregate features in both spatial and channel dimensions based on the differences in spatial and semantic information contained in the four branches. Spatial attention is used to capture long-range relationships between pixels to address the large intra-class variability, and channel attention is used to calculate dependencies between features. To address the computational complexity of the correlation graph calculation in the self-attention mechanism, this paper uses a feature pyramid to sample the feature map to reduce the amount of computation. At the same time, a decoding attention mechanism is used to compensate for the performance degradation caused by the reduced computational effort.

[0115] Specifically, this solution adopts a semantic edge detection network to guide the design of semantic segmentation downsampling to solve the problem of poor segmentation accuracy in edge areas of images. The semantic edge detection network designed in this solution includes two parallel branches. The branches introduced at each stage of feature extraction aggregate spatial context information and capture feature importance information through a convolutional layer with a spatial dimension and a channel dimension of one. The two different levels of feature maps are then fused to enhance high-level feature representation and strengthen the edge transitivity. Different weights are assigned to different positions of each edge slice through adaptive weight fusion. In the edge fusion process, edge information is explicitly integrated into the downsampling process of feature extraction to enhance the learning of edge areas.

[0116] During the specific implementation process, the above-mentioned target object image is preprocessed to obtain a preprocessed object image, including one of the following: performing the above-mentioned image segmentation or the above-mentioned data expansion on the above-mentioned target object image to obtain the above-mentioned preprocessed object image; performing the above-mentioned image segmentation on the above-mentioned target object image, and performing the above-mentioned data expansion on the target object image after the image segmentation to obtain the above-mentioned preprocessed object image; performing the above-mentioned data expansion on the above-mentioned target object image, and performing the above-mentioned image segmentation on the target object image after the data expansion to obtain the above-mentioned preprocessed object image.

[0117] In this solution, when processing high-resolution images, image segmentation can avoid unnecessary waste of computing resources while maintaining the ability to recognize small targets. Through data expansion, a more comprehensive and varied training set can be created, increasing the diversity of the training set. In this way, there will be enough training data for model training, which can further improve the accuracy of subsequent model recognition.

[0118] Specifically, when performing image segmentation, the original image can be divided into multiple small image blocks. For example, a large image can be segmented into 256x256 pixel blocks. Of course, any other feasible pixel size can also be used, such as 1024x1024 pixel blocks or 1080x1080 pixel blocks. This ensures that each segmented image block contains complete or partial information about the target object.

[0119] Specifically, when performing data expansion, various methods can be used, such as image flipping, adjusting brightness (±30%), adjusting contrast (±50%), adjusting hue (±20%), etc., to generate a variety of images and increase the diversity of the training set.

[0120] Image segmentation ensures that each image block contains key visual information, such as the customer's or employee's facial features, handheld documents, or operating posture, which helps improve the model's training efficiency and recognition speed. Data augmentation, by simulating different environmental conditions such as adjusting brightness (±30%), contrast (±50%), and hue (±20%), enables the model to adapt to the conditions of insufficient lighting, excessive brightness, or color shift that may be encountered inside a bank. This is crucial for ensuring accurate target recognition under any lighting conditions.

[0121] In some embodiments, the target object image is segmented to obtain the preprocessed object image, which can be specifically achieved by the following steps: when the target image includes one target object, a part of the target object image is cut off to obtain the preprocessed object image, so that the preprocessed object image contains the complete inter-class edge; when the target image includes multiple target objects, the target object image is segmented to obtain multiple preprocessed object images, so that each preprocessed object image contains the complete inter-class edge of at least one target object.

[0122] In this scheme, when there is only one target object in the target image, the target image is segmented in a way that retains the complete inter-class edges, removing part of the background or irrelevant areas, and leaving the key information containing the target object and its edges. This can reduce the size of the target image and avoid errors in loading images caused by target objects that are too large.

[0123] In practice, the specific method and parameter selection for image segmentation have a direct impact on the final recognition results. For example, when there is only one target object in the target image, the image segmentation can be performed by setting a window that is appropriately larger than the target object size, such as 1.5 times the area of ​​the target object's minimum bounding rectangle, to ensure that the edges between classes are not segmented.

[0124] In the case of multiple target objects, an adaptive window segmentation strategy can be adopted to ensure efficiency and resource utilization. Initially, a larger window, such as one-quarter of the entire image, is set for the first segmentation. Subsequently, the window size is dynamically adjusted based on the density of target objects in each segmentation window to ensure that each window contains the complete edge of at least one target object. For example, if a window is found to contain more than three targets, it can be further segmented into four smaller sub-windows until each window contains at least one but no more than two targets.

[0125] During the specific implementation process, before acquiring the target image, the above method also includes the following steps: acquiring the light intensity of the target area, wherein the above light intensity is detected by a light sensor; when the above light intensity is greater than a preset light intensity threshold, reducing the exposure time of the image acquisition device and reducing the gain of the above image acquisition device, wherein the above image acquisition device is used to acquire the above target video of the above target area; when the above light intensity is less than the above preset light intensity threshold, increasing the above exposure time of the above image acquisition device and increasing the above gain of the above image acquisition device.

[0126] In this solution, in an environment with sufficient lighting, if the exposure time and gain of the image acquisition device are set too high, the image will be overexposed, edge details will be lost, and too much noise will be introduced, affecting the performance of the image recognition model. By reducing the exposure time and gain under high-light conditions, a clear and noise-free image can be obtained, ensuring good image quality.

[0127] Specifically, a light sensor (such as a photoresistor or photodiode) is used to detect the light intensity in the target area in real time. For example, a light sensor at the entrance of a bank hall can continuously monitor the changes in light from day to night.

[0128] When the light intensity is greater than a preset light intensity threshold (e.g., 1000 lux), reduce the exposure time (e.g., to 1 / 1000 second) and gain (e.g., to 0.5x) of the image acquisition device to prevent overexposure of the image and reduce noise.

[0129] When the light intensity is less than the preset light intensity threshold (for example, 500 lux), increase the exposure time (for example, to 1 / 60 second) and gain (for example, to 2x) of the image acquisition device to ensure the image acquisition quality in low light environments and avoid images that are too dark and features that are unclear.

[0130] In some embodiments, after obtaining the image recognition model to be trained, the method further includes the following steps: obtaining an occlusion training set, wherein the occlusion training set is an image data set obtained by performing occlusion processing on a partial area of ​​the target object image, and the occlusion processing includes using graphics to occlude a partial area of ​​the target object, color blocks to occlude a partial area of ​​the target object, or a natural object image to partially occlude a partial area of ​​the target object; obtaining a recognition network, wherein the recognition network is one of a ResNet network, a MobileNet network, or a Mask R-CNN network; using the occlusion training set to train the recognition network to obtain an occlusion recognition network to identify the occluded image area; determining the occlusion recognition network as the occlusion recognition module of the image recognition model to be trained, and adding the occlusion recognition module and the attention mechanism module before the decoder layer of the image recognition model to be trained, so that the output of the occlusion recognition module serves as one of the inputs of the attention mechanism module, and the output of the attention mechanism module serves as the input of the decoder layer to obtain the image recognition model.

[0131] In this solution, the occlusion recognition module learns the characteristics of occluded areas through training, allowing it to identify occluded areas during testing and pass this information to the attention mechanism module. The attention mechanism module adjusts its weight distribution based on the occlusion information, making the model pay more attention to unoccluded areas, thereby maintaining good recognition performance even in occluded scenes.

[0132] Specifically, randomly generated occlusion patterns, such as black blocks or images of natural objects, are used to occlude the original target object image to generate an occlusion training set. The occluded area can be part of the target object's ID card, document, or face. The occlusion training set is prepared to simulate occlusion situations that may be encountered in actual surveillance videos, providing the model with the ability to handle occlusions.

[0133] We selected the MobileNet network as the underlying recognition network and trained it using the prepared occlusion training set to create an occlusion recognition network. Through occlusion training, the model learned to recognize target objects even when parts of them were obscured.

[0134] Add an occlusion detection module and an attention mechanism module before the decoder layer of the image recognition model to be trained. The output of the occlusion detection module serves as the input to the attention mechanism module, guiding the model to focus more on unobstructed key features during the decoding process, such as the intact portion of an ID card or documents not obscured by hands. By integrating the occlusion detection module and the attention mechanism, the model can intelligently identify occluded areas and automatically adjust its attention when processing images, ensuring accurate recognition of the target object even in the presence of occlusion.

[0135] In bank surveillance videos, it's common for objects to be partially obscured. For example, an employee might cover their ID card or face with their hand while communicating, or a customer might temporarily block a document with their body while filling it out. These occlusions can severely impact the performance of traditional image recognition models.

[0136] By implementing the above technical solutions, the bank's internal monitoring system can handle occlusion more intelligently:

[0137] The occlusion recognition module can accurately determine which areas in the image are occluded, such as the area where the work badge is occluded by 30%.

[0138] After receiving occlusion information, the attention mechanism module automatically adjusts the weight to pay more attention to the unobstructed areas. For example, the weight of the unobstructed ID card part is increased to 1.5 times, while the weight of the occluded area is reduced to 0.5 times.

[0139] The enhanced image recognition model can quickly identify key target objects when processing obscured videos. For example, even if the ID card is partially obscured, it can accurately identify the identity of the employee wearing the ID card, ensuring internal security and compliance of the bank.

[0140] The occlusion recognition network contains multi-level feature extraction layers and can process input images of different scales to adapt to various occlusion situations. The training of the occlusion recognition network includes two tasks: occlusion recognition and unoccluded area classification, to enhance the model's understanding of occlusion situations and its ability to recognize unoccluded areas. Each image in the occlusion training set is annotated with occluded areas to guide the learning of the occlusion recognition module. The attention mechanism module calculates the weights of occluded and unoccluded areas, prioritizes the feature information of unoccluded areas, and improves the model's detection and classification accuracy of complete targets. The attention mechanism module can adopt a soft attention mechanism to dynamically change the model's attention to different areas of the image through learnable weight adjustments to adapt to changes in the degree of occlusion.

[0141] To simulate occlusions that might occur in bank surveillance videos, the original target object images were occluded to generate an occlusion training set. For example, a 200×200 pixel black square was used to occlude the target object's ID card area, or images of randomly shaped natural objects were used to occlude the target object's hands to simulate occlusion scenarios when an account manager is working with documents or communicating with customers.

[0142] Select a model from ResNet, MobileNet, or Mask R-CNN as the basis for the recognition network. For example, select MobileNet. Train the recognition network using the occlusion training set to obtain an occlusion recognition network specifically designed for processing occluded areas.

[0143] The trained occlusion detection network is used as the occlusion detection module and integrated into the image recognition model to be trained. The occlusion detection module and the attention mechanism module are added before the decoder layer of the model. The output of the occlusion detection module serves as the input of the attention mechanism module, guiding it to focus more on unoccluded key areas during the decoding process while reducing the impact of occlusions on model performance.

[0144] Specifically, the above solution of the present application includes processes such as video stream pulling for dual-recording video intelligent detection, video frame deduplication, model detection, and image preprocessing. The detection category model uses an image recognition model based on semantic edge guidance.

[0145] Existing intelligent detection solutions for dual-recorded videos primarily rely on algorithms such as object detection, skeleton recognition, and pose detection. Object detection in dual-recorded videos faces challenges including difficulty accurately capturing target location and boundary information, limited ability to handle occlusion and scale variations, and susceptibility to background interference, resulting in low detection accuracy. Dual-recorded videos often suffer from low inter-class variability and high intra-class variability. Object detection tasks are often constrained by speed and timeliness, and lack the use of self-attention mechanisms to capture spatial interactions between different locations and feature correlations across different feature channels, resulting in poor model performance. Traditional encoder-decoder convolutional neural networks use cascaded convolutional layers and pooling operations to aggregate semantic information, resulting in a loss of localization accuracy and the destruction of spatial detail. Edges are one of the most fundamental features in digital images, and the edge regions of an image contain rich information. However, convolutional neural networks typically only produce inaccurate boundary predictions. The strong correlation between the model-generated entropy heat map and the label error map indicates that inter-class edges are the areas where the model's segmentation is most disrupted.

[0146] The solution in this application introduces semantic edge detection into the semantic segmentation task, extracts inter-class edges through lightweight semantic edge detection, and explicitly incorporates inter-class edges into the downsampling process through shared connections. In the semantic segmentation branch, parallel branches are used to avoid the destruction of the spatial structure of traditional codecs. The self-attention mechanism after dimensionality reduction captures the correlation between position and features without increasing the number of parameters, computational complexity, or processing time, thereby enhancing feature representation and improving model performance.

[0147] Specifically, the inventors of this application experimented with different network models. The experimental results are shown in Table 1 below. It can be seen that the semantic segmentation network based on HRNet achieves better performance with similar parameter and computational complexity as the original HRNet, and the computational complexity is only half of that of the target detection network.

[0148] Table 1

[0149]

[0150] This application's dual-recording video detection solution uses a semantic segmentation algorithm instead of an object detection algorithm. It introduces spatial and channel self-attention mechanisms to address scale shifts and susceptibility to background interference. By reducing the dimensionality of the self-attention mechanism's correlation graph and introducing a decoding attention mechanism, model performance is improved without increasing parameters. The introduction of an edge-guided branch addresses the difficulty in capturing boundary information and refines edge representation.

[0151] The embodiment of the present application also provides a semantic-based image recognition device. It should be noted that the semantic-based image recognition device of the embodiment of the present application can be used to execute the semantic-based image recognition method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0152] The following introduces the semantic-based image recognition device provided in the embodiments of the present application.

[0153] Figure 10 This is a structural block diagram of a semantic-based image recognition device according to an embodiment of the present application. Figure 10 As shown, the device includes:

[0154] A first acquisition unit 10 is configured to acquire a target image, wherein the target image includes an image of a target object of a category to be identified and a background;

[0155] An extraction unit 20 is configured to extract the inter-class edges of the target object, wherein the inter-class edges include edge pixels of the target object and some pixels of an adjacent area of ​​the target object;

[0156] an identification unit 30 configured to determine the inter-class edge and the image region enclosed by the inter-class edge in the target image as a target object image, and to identify the target object image to obtain a category identification result of the target object, wherein the category identification result is one of a person, an animal, and an object;

[0157] The processing unit 40 is configured to send the category identification result to the security monitoring system of the financial institution, so that the security monitoring system performs security monitoring based on the category identification result.

[0158] In the prior art, the edges of objects in the image are often ignored during target category recognition, which results in a decrease in the accuracy of target category recognition of the image. Through this embodiment, after obtaining the target image, the boundary area between the target object and the non-target object in the target image is identified. This area is used as the edge area, that is, the inter-class edge. This can enhance the edge representation without missing the pixel points of the target object. The image semantics is the meaning expressed by the image content. Therefore, this scheme uses image semantic recognition to identify the target object image. In addition, since the inter-class edge of the target object is determined in advance in this scheme, the edge representation of the target object is enhanced. In this way, when identifying the target object image, the edge of the target object in the target object image will not be ignored, and the recognition can be performed according to the complete pixel area of ​​the target object, thereby improving the recognition accuracy. In addition, the security monitoring system of the financial institution can perform security monitoring more accurately based on the accurate category recognition results.

[0159] During the specific implementation process, the first acquisition unit includes a first acquisition module, a frame extraction module, a calculation module and a first determination module. The first acquisition module is used to acquire the target video; the frame extraction module is used to extract the target video once at an interval of a predetermined number of frames to obtain multiple initial images; the calculation module is used to use the perceptual hash algorithm to calculate the similarity between any two of the multiple initial images; the first determination module is used to delete one of the two initial images whose similarity is greater than or equal to the similarity threshold, until the similarity between any two of the multiple initial images is less than the similarity threshold, and determine that all the remaining non-deleted initial images are the target images.

[0160] In this solution, reducing the number of frames of video processing means reducing the amount of data and computing power, thereby speeding up the subsequent similarity calculation and image screening process. When two frames of images are highly similar in structure and content, they will be considered redundant, thereby deleting duplicate information and ensuring that the image set received by the security monitoring system is minimally redundant. The deduplicated image set reduces the demand for computing resources.

[0161] In some embodiments, the extraction unit includes a detection module, a second acquisition module and an expansion module, the detection module is used to perform edge detection on the target object to obtain an edge curve, wherein the edge curve is a curve composed of pixel points of the boundary of the target object detected; the second acquisition module is used to obtain the normal of the edge curve, wherein there are multiple normals, and one pixel point on the edge curve corresponds to one normal, and the normal is a straight line perpendicular to the tangent of the pixel point on the edge curve; the expansion module is used to expand each pixel point on the edge curve by a preset number of pixels along the direction of the corresponding normal to obtain the inter-class edge.

[0162] In this scheme, by appropriately expanding the edge curve, it can be ensured that the obtained inter-class edge can capture all key edge information without over-expanding and causing excessive background interference, which can further enhance the representation of edge information.

[0163] During the specific implementation process, the recognition unit includes a preprocessing module, a training module and a processing module. The preprocessing module is used to preprocess the above-mentioned target object image to obtain a preprocessed object image, wherein the above-mentioned preprocessing method includes image segmentation and / or data expansion, wherein the above-mentioned data expansion method includes one or more of image flipping, changing image brightness, changing image contrast, and changing image hue; the training module is used to combine the above-mentioned target object image and / or the above-mentioned preprocessed object image into a training image set, and use the above-mentioned training image set and the corresponding category label to train the image recognition model to obtain a category recognition model, wherein the above-mentioned category label is the category of the object in the image of the above-mentioned training image set; the processing module is used to input the above-mentioned target object image into the above-mentioned category recognition model to obtain the above-mentioned category recognition result corresponding to the above-mentioned target object image.

[0164] In this solution, a more comprehensive and varied training set can be created through image segmentation and data expansion. This way, there is enough training data for model training, which increases the diversity of the training set. During the training process, the model can be exposed to target objects under various transformation conditions. This enables the model to better understand the shape, color, and texture of the object when facing new images, maintain a high classification accuracy, and thus improve the recognition accuracy.

[0165] In some embodiments, the above-mentioned device also includes a second acquisition unit and a first adding unit. The second acquisition unit is used to obtain the image recognition model to be trained before using the above-mentioned training image set and the corresponding category label to train the image recognition model to obtain the category recognition model. The above-mentioned image recognition model to be trained is one of the HRNet model, U-NET model, and GAN model; the first adding unit is used to add an attention mechanism module before the decoder layer of the above-mentioned image recognition model to be trained, so that the output of the above-mentioned attention mechanism module is used as the input of the above-mentioned decoder layer to obtain the above-mentioned image recognition model.

[0166] In this scheme, an attention mechanism module is added before the decoder layer of the selected image recognition model to be trained. The attention mechanism module can enhance the model's attention to key areas in the image, ensuring that the model prioritizes high-weight information during upsampling and decoding, thereby avoiding the loss of edge information and ensuring a high recognition accuracy of the model.

[0167] During the specific implementation process, the preprocessing module includes a first preprocessing submodule, a second preprocessing submodule and a third preprocessing submodule. The first preprocessing submodule is used to perform the above-mentioned image segmentation or the above-mentioned data expansion on the above-mentioned target object image to obtain the above-mentioned preprocessed object image; the second preprocessing submodule is used to perform the above-mentioned image segmentation on the above-mentioned target object image, and perform the above-mentioned data expansion on the target object image after the image segmentation to obtain the above-mentioned preprocessed object image; the third preprocessing submodule is used to perform the above-mentioned data expansion on the above-mentioned target object image, and perform the above-mentioned image segmentation on the target object image after the data expansion to obtain the above-mentioned preprocessed object image.

[0168] In this solution, when processing high-resolution images, image segmentation can avoid unnecessary waste of computing resources while maintaining the ability to recognize small targets. Through data expansion, a more comprehensive and varied training set can be created, increasing the diversity of the training set. In this way, there will be enough training data for model training, which can further improve the accuracy of subsequent model recognition.

[0169] In some embodiments, the preprocessing module is further used to, when the target image includes one target object, cut out a portion of the target object image to obtain the preprocessed object image, so that the preprocessed object image contains the complete inter-class edge; the preprocessing module is further used to, when the target image includes multiple target objects, segment the target object image to obtain multiple preprocessed object images, so that each preprocessed object image contains the complete inter-class edge of at least one target object.

[0170] In this scheme, when there is only one target object in the target image, the target image is segmented in a way that retains the complete inter-class edges, removing part of the background or irrelevant areas, and leaving the key information containing the target object and its edges. This can reduce the size of the target image and avoid errors in loading images caused by target objects that are too large.

[0171] During the specific implementation process, the above-mentioned device also includes a third acquisition unit, a first adjustment unit and a second adjustment unit. The third acquisition unit is used to obtain the light intensity of the target area before obtaining the target image, wherein the above-mentioned light intensity is obtained by detection by a light sensor; the first adjustment unit is used to reduce the exposure time of the image acquisition device and reduce the gain of the above-mentioned image acquisition device when the above-mentioned light intensity is greater than a preset light intensity threshold, wherein the above-mentioned image acquisition device is used to acquire the above-mentioned target video of the above-mentioned target area; the second adjustment unit is used to increase the above-mentioned exposure time of the above-mentioned image acquisition device and increase the above-mentioned gain of the above-mentioned image acquisition device when the above-mentioned light intensity is less than the above-mentioned preset light intensity threshold.

[0172] In this solution, in an environment with sufficient lighting, if the exposure time and gain of the image acquisition device are set too high, the image will be overexposed, edge details will be lost, and too much noise will be introduced, affecting the performance of the image recognition model. By reducing the exposure time and gain under high-light conditions, a clear and noise-free image can be obtained, ensuring good image quality.

[0173] In some embodiments, the apparatus further includes a fourth acquisition unit, a fifth acquisition unit, a training unit, and a second adding unit, wherein the fourth acquisition unit is configured to acquire an occlusion training set after acquiring the image recognition model to be trained, wherein the occlusion training set is an image data set obtained by performing occlusion processing on a partial area of ​​the target object image, and the occlusion processing includes using graphics to occlude a partial area of ​​the target object, a color block to occlude a partial area of ​​the target object, or a natural object image to partially occlude a partial area of ​​the target object; the fifth acquisition unit is configured to acquire a recognition network, wherein the recognition network is one of a ResNet network, a MobileNet network, or a Mask R-CNN network; the training unit is configured to train the recognition network using the occlusion training set to obtain an occlusion recognition network to identify occluded image areas; the second adding unit is configured to determine the occlusion recognition network as the occlusion recognition module of the image recognition model to be trained, and to add the occlusion recognition module and the attention mechanism module before the decoder layer of the image recognition model to be trained, so that the output of the occlusion recognition module serves as one of the inputs of the attention mechanism module, and the output of the attention mechanism module serves as the input of the decoder layer to obtain the image recognition model.

[0174] In this solution, the occlusion recognition module learns the characteristics of occluded areas through training, allowing it to identify occluded areas during testing and pass this information to the attention mechanism module. The attention mechanism module adjusts its weight distribution based on the occlusion information, making the model pay more attention to unoccluded areas, thereby maintaining good recognition performance even in occluded scenes.

[0175] The semantic-based image recognition device includes a processor and memory. The first acquisition unit, extraction unit, recognition unit, and processing unit are all stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0176] The processor includes a core, which retrieves the corresponding program unit from memory. One or more cores can be configured, and kernel parameters can be adjusted to address the low accuracy of image object classification recognition in existing technologies, resulting in low security monitoring accuracy for financial institutions.

[0177] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0178] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the semantic-based image recognition method.

[0179] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the semantic-based image recognition method when running.

[0180] An embodiment of the present invention provides a device comprising a processor, a memory, and a program stored in the memory and executable by the processor. When the processor executes the program, it implements at least the steps of a semantic-based image recognition method. The device herein may be a server, a PC, a PAD, a mobile phone, or the like.

[0181] A computer program product includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the semantic-based image recognition method in each embodiment of the present application are implemented.

[0182] The present application provides an image recognition system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the above-mentioned semantic-based image recognition methods.

[0183] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0184] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0189] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0190] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0191] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0192] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A semantic-based image recognition method, characterized in that: include: Acquire a target image, wherein the target image includes an image of a target object of a category to be identified and a background; Extracting an inter-class edge of the target object, wherein the inter-class edge includes edge pixels of the target object and some pixels of an adjacent area of ​​the target object; Determining the inter-class edge and the image region surrounded by the inter-class edge in the target image as a target object image, and recognizing the target object image to obtain a category recognition result of the target object, wherein the category recognition result is one of a category of a person, an animal, and a category of an object; The category identification result is sent to a security monitoring system of a financial institution, so that the security monitoring system performs security monitoring based on the category identification result.

2. The method according to claim 1, characterized in that Get the target image, including: Get the target video; Extract a frame from the target video at intervals of a predetermined number of frames to obtain a plurality of initial images; Using a perceptual hash algorithm, calculating the similarity between any two of the multiple initial images; One of the two initial images whose similarity is greater than or equal to a similarity threshold is deleted until the similarity between any two of the multiple initial images is less than the similarity threshold, and all the remaining initial images that have not been deleted are determined to be the target images.

3. The method according to claim 1, characterized in that Extracting the inter-class edge of the target object, comprising: Performing edge detection on the target object to obtain an edge curve, wherein the edge curve is a curve composed of pixel points of the boundary of the target object detected; Obtaining a normal line of the edge curve, wherein there are multiple normal lines, one normal line corresponds to each pixel point on the edge curve, and the normal line is a straight line perpendicular to a tangent line of the pixel point on the edge curve; Each pixel point on the edge curve is expanded by a preset number of pixel points along the direction of the corresponding normal line to obtain the inter-class edge.

4. The method according to claim 1, wherein Recognizing the target object image to obtain a category recognition result of the target object includes: Preprocessing the target object image to obtain a preprocessed object image, wherein the preprocessing method includes image segmentation and / or data expansion, wherein the data expansion method includes one or more of image flipping, changing image brightness, changing image contrast, and changing image hue; The target object images and / or the pre-processed object images are combined into a training image set, and an image recognition model is trained using the training image set and corresponding category labels to obtain a category recognition model, wherein the category label is the category of the object in the image of the training image set; The target object image is input into the category recognition model to obtain the category recognition result corresponding to the target object image.

5. The method according to claim 4, characterized in that Before using the training image set and the corresponding category labels to train the image recognition model to obtain the category recognition model, the method further includes: Obtain an image recognition model to be trained, wherein the image recognition model to be trained is one of an HRNet model, a U-NET model, and a GAN model; An attention mechanism module is added before the decoder layer of the image recognition model to be trained, so that the output of the attention mechanism module is used as the input of the decoder layer to obtain the image recognition model.

6. The method according to claim 4, characterized in that Preprocessing the target object image to obtain a preprocessed object image includes one of the following: performing the image segmentation or the data expansion on the target object image to obtain the preprocessed object image; performing the image segmentation on the target object image, and performing the data expansion on the target object image after the image segmentation to obtain the preprocessed object image; The data expansion is performed on the target object image, and the image segmentation is performed on the target object image after the data expansion to obtain the preprocessed object image.

7. The method according to claim 6, characterized in that Performing the image segmentation on the target object image to obtain the preprocessed object image includes: In a case where the target image includes one target object, cutting out a portion of the target object image to obtain the pre-processed object image, so that the pre-processed object image includes the complete inter-class edge; In the case where the target image includes a plurality of target objects, the target object image is segmented to obtain a plurality of pre-processed object images, such that each of the pre-processed object images contains a complete inter-class edge of at least one target object.

8. A semantic-based image recognition device, characterized in that: include: A first acquisition unit is configured to acquire a target image, wherein the target image includes an image of a target object of a category to be identified and a background; an extraction unit, configured to extract an inter-class edge of the target object, wherein the inter-class edge includes edge pixels of the target object and some pixels of an adjacent area of ​​the target object; an identification unit, configured to determine the inter-class edge and the image region surrounded by the inter-class edge in the target image as a target object image, and identify the target object image to obtain a category identification result of the target object, wherein the category identification result is one of a person, an animal, and an object; The processing unit is configured to send the category identification result to a security monitoring system of a financial institution, so that the security monitoring system performs security monitoring based on the category identification result.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the semantic-based image recognition method according to any one of claims 1 to 7 are implemented.

10. An image recognition system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the semantic-based image recognition method described in any one of claims 1 to 7.

Images