Code identification method, device and equipment and readable storage medium

Abstract

The invention provides a code recognition method, device and equipment and a readable storage medium, and relates to the technical field of computers, the method comprises the following steps: obtaining an image comprising a to-be-recognized code, and carrying out connected region analysis on the image to obtain a first foreground region; the first foreground region comprises a connected region where the to-be-identified code is located; obtaining hues of pixel points in the image, and dividing the image according to the hues to obtain a second foreground region; the second foreground area comprises an area formed by pixel points of the code to be identified; fusing the first foreground region and the second foreground region to obtain a fused foreground region, and segmenting the image based on the fused foreground region to obtain at least one sub-image; and inputting the at least one sub-image into a pre-trained code recognition model to obtain a to-be-recognized code in the image. According to the code method, the to-be-identified code can be rapidly and accurately extracted from the image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a code recognition method, device, equipment and readable storage medium. Background Art

[0002] Verilog Hardware Description Language is a language that describes the hardware structure and behavior of a digital system in text form.

[0003] In the process of Verilog code development, it is usually necessary to aggregate and communicate Verilog codes between different devices. However, the devices on the development side usually do not support direct copying of Verilog codes to local devices. In the related art, the Verilog codes to be aggregated are usually entered manually in the devices on the development side. However, this method has the problems of high error rate and low efficiency. Summary of the invention

[0004] The embodiments of the present application provide a code recognition method, apparatus, device and readable storage medium to solve the problem of low efficiency and easy error in obtaining codes in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a code recognition method, comprising: Acquire an image including a code to be identified, and perform connected region analysis on the image to obtain a first foreground region; the first foreground region includes the connected region where the code to be identified is located; Acquire the hue of the pixel points in the image, and divide the image according to the hue to obtain a second foreground area; the second foreground area includes an area composed of the pixel points of the code to be identified; fusing the first foreground area and the second foreground area to obtain a fused foreground area, and segmenting the image based on the fused foreground area to obtain at least one sub-image; At least one of the sub-images is input into a pre-trained code recognition model to obtain a code to be recognized in the image.

[0006] In a second aspect, an embodiment of the present application provides a code recognition device, including: A first acquisition module is used to acquire an image including a code to be identified, and perform a connected region analysis on the image to obtain a first foreground region; the first foreground region includes a connected region where the code to be identified is located; A second acquisition module is used to acquire the hue of the pixel points in the image, and divide the image according to the hue to obtain a second foreground area; the second foreground area includes an area composed of the pixel points of the code to be identified; A third acquisition module, configured to fuse the first foreground area and the second foreground area to obtain a fused foreground area, and segment the image based on the fused foreground area to obtain at least one sub-image; The fourth acquisition module is used to input at least one of the sub-images into a pre-trained code recognition model to obtain the code to be recognized in the image.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method of the first aspect.

[0008] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method of the first aspect.

[0009] In this embodiment, the image including the code to be identified is subjected to a connected region analysis to obtain a first foreground region; the first foreground region thus obtained is the code region to be identified determined from the shape level. The hue of the pixels in the image is obtained, and the image is divided into regions according to the hue to obtain a second foreground region; the second foreground region thus obtained is the code region to be identified determined from the color level. The color of the code is usually different from the background color. Therefore, the first foreground region obtained based on the connected region analysis and the second foreground region obtained based on the hue of the pixels can accurately reflect the region where the code to be identified is located, and the fused foreground region obtained by fusing the first foreground region and the second foreground region further improves the accuracy of reflecting the location of the code to be identified. Furthermore, the image is segmented based on the fused foreground region, and then the obtained sub-image is input into a pre-trained code recognition model, so that the code to be identified in the image can be accurately and efficiently obtained. The problem of low input efficiency and easy error caused by manual code input in the related art is solved.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 This is a schematic diagram of an application scenario of a code recognition method provided in an embodiment of the present application; Figure 2 This is a flowchart of a code recognition method provided by an embodiment of the present application; Figure 3 This is a flowchart of a code recognition method provided by an embodiment of the present application; Figure 4 This is a flowchart of a code recognition method provided by an embodiment of the present application; Figure 5 This is a flowchart of a code recognition method provided by an embodiment of the present application; Figure 6 This is a flowchart of a code recognition method provided by an embodiment of the present application; Figure 7 This is a flowchart of a code recognition method provided by an embodiment of the present application; Figure 8 is a block diagram of a code recognition device provided by an embodiment of the present invention; Fig. 9 is a block diagram of an electronic device provided by an embodiment of the present invention; Fig.10 is a block diagram of another electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, the term "and / or" in the specification and claims is used to describe the association relationship of associated objects, indicating that three kinds of relationships can exist, for example, A and / or B can be represented: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the front and back associated objects are a kind of "or" relationship. In the embodiment of the present application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0015] Figure 1 This is a schematic diagram of an application scenario of a code recognition method provided in an embodiment of the present application, referring to Figure 1 , the application scenario at least includes a server platform 10 and a local computing device 20.

[0016] Among them, the server platform 10 can be a server on the development side. In the development process of some codes (for example, Verilog codes), a large number of retrievals, code summaries and text communications that are strongly related to the codes are required. The server platform 10 on the development side may not support the direct copying of development information to the local computing device 20. Therefore, in the relevant technology, the code developed in the server platform 10 needs to be input into the local computing device 20 by manual input. In application scenarios involving large-scale code processing, such as summarizing code signals in documents, manual input will not only lead to repetitive and inefficient work, but may also cause spelling errors and missing input problems.

[0017] In order to solve the problems of the related art, the local computing device 20 of this embodiment obtains an image containing a code to be identified provided by the service platform 10, and obtains the code to be identified in the image according to the following method: a connected region analysis is performed on the image including the code to be identified to obtain a first foreground region; the first foreground region is a connected region where the code to be identified is located; the image is divided into regions according to the grayscale values ​​of the pixels in the image to obtain a second foreground region; the second foreground region is an area composed of the pixels of the code to be identified; the first foreground region and the second foreground region are fused to obtain a fused foreground region, and the image is segmented based on the fused foreground region to obtain at least one sub-image; at least one sub-image is input into a pre-trained code recognition model to obtain the code to be identified in the image. Based on this embodiment, the code to be identified in the image can be automatically identified, avoiding the problems of low efficiency and high error rate caused by manual code input in the related art.

[0018] It should be noted that Figure 1 What is shown is only one application scenario of the method of this embodiment. The picture containing the code to be recognized processed by this embodiment can also come from a cloud server, a mobile terminal, a tablet computer, a storage device and other devices, and is not limited to the server platform. In this embodiment, the device for recognizing the code to be recognized in the image can be a server, a mobile terminal, a tablet computer and other electronic devices, which are not limited here.

[0019] Figure 2 The flowchart of a code recognition method provided by this embodiment is shown. Figure 2 , the method may include the following steps: Step 101, obtaining an image including a code to be identified, and performing connected region analysis on the image to obtain a first foreground region.

[0020] The first foreground area is a connected area where the code to be identified is located.

[0021] For example, the image in this step is an original image including the code to be identified, and the image is preprocessed to obtain a preprocessed image. The preprocessed image is morphologically processed to obtain a morphologically processed image, and then the morphologically processed image is subjected to connected region analysis to obtain a first foreground region. For example, the preprocessing method may include grayscale processing and binarization processing; the preprocessed image may be morphologically processed by a closing operation.

[0022] For example, according to the gray value of each pixel in the image, the connected pixel points are identified as a connected domain through the connected region analysis method, thereby obtaining the first foreground area where the code to be identified is located. The first foreground area has a corresponding identifier. Specifically, the centroid of the connected domain is obtained through the connected region analysis, and the centroid is determined as the seed point of the first foreground area, and an identifier is set for the seed point, for example, the identifier can be set to 1.

[0023] Furthermore, the seed points of the background area can be set based on empirical data. For example, the code to be identified is usually located in the middle area of ​​the image. Therefore, the four corner vertices and the midpoints of the four edges of the image can be determined as the seed points of the background area, and an identifier can be set for the background area. For example, the identifier can be set to 0.

[0024] Furthermore, in addition to the first foreground region obtained by the connected region analysis and the determined background region, the image also includes other regions, which are uncertain regions where it is uncertain whether they include the code to be identified. Flags are set for the other regions, for example, the flags can be set to negative numbers (for example, -1).

[0025] Therefore, based on the connected region analysis method, the first foreground area, background area and other areas are obtained. Each area has a corresponding identifier. These identifiers and the pixel points corresponding to the identifiers can form an identifier image with the same size as the image. The identifier image is a shape marker identifier image used to reflect the shape of each area in the image.

[0026] Step 102, obtaining the hue of the pixel points in the image, and dividing the image into regions according to the hue to obtain a second foreground region.

[0027] The second foreground area is an area formed by the pixels of the code to be identified.

[0028] For example, the original image is converted into a color space to obtain the hue of the pixels in the image; wherein the original image is an original red, green, and blue (RGB) image. Specifically, the original image is converted into a color space to obtain an image based on a color model of hue, saturation, and value (HSV). Among the three channels of the HSV image, the H (Hue) channel represents the hue, and the hue H can be used to distinguish the types of colors, and different colors can be effectively perceived based on the hue H.

[0029] Different colors can be set in the compiler according to different parts or functions of the code to be identified, and different colors correspond to different color marker identifiers. Furthermore, after acquiring the image of the HSV color model, different preset hue thresholds are set for the H channel to distinguish different color areas in the code to be identified according to the hue threshold. For example, the background, comments and code body in the code to be identified have different hues, and corresponding preset hue thresholds are set for different hues; then foreground seed points are set for different second foreground areas in the image. Furthermore, the identifiers of different second foreground area seed points are different from each other, and are different from the identifiers of the first foreground area seed points.

[0030] For example, the seed point used to identify the first foreground area in the shape marker is marked as 1, and the positive integer used to identify the comment, code body, etc. in the second foreground area needs to be different from the positive integer in the shape marker. For example, it can be set to 2, 3 to N.

[0031] Further, the background seed point can be set by referring to the method of step 101, and other areas except the second foreground area and the background seed point can be determined, and then negative numbers (such as -1) can be set for the pixel points in other areas. Based on the second foreground area, the background area and other areas, a color marker image of the same size as the input image can be generated.

[0032] Step 103: fuse the first foreground area and the second foreground area to obtain a fused foreground area, and segment the image based on the fused foreground area to obtain at least one sub-image.

[0033] The fusion of the first foreground area and the second foreground area comprises: if a pixel point belongs to at least one of the first foreground area and the second foreground area, determining it as a pixel point of the fused foreground area and setting a flag for it.

[0034] Specifically, if the pixel point belongs to both the first foreground area and the second foreground area, a new identifier is set for it, wherein the new identifier is different from the identifiers of the first foreground area and the second foreground area. If the pixel point belongs to only one of the foreground areas, the identifier of the foreground area to which the pixel point belongs is determined as the identifier of the pixel point.

[0035] Further, if the pixel point is a pixel point of other areas, the identifier of the other areas is determined as the identifier of the pixel point; if the pixel point is a pixel point of the background area, the identifier of the background area is determined as the identifier of the pixel point.

[0036] For example, the identification of the foreground area, the identification of other areas, and the identification of the background area are fused to form a fused marker identification. The fused marker identification is input into the watershed segmentation model to obtain at least one sub-image.

[0037] Step 104: input at least one sub-image into a pre-trained code recognition model to obtain a code to be recognized in the image.

[0038] For example, a pre-trained code recognition model is used to extract visual features and semantic features from a sub-image, and then the visual features and semantic features are fused. Based on the fused features, the code to be recognized in the image is predicted.

[0039] For example, the code to be recognized in the sub-image can be extracted by using the optical character recognition (OCR) technology, wherein the OCR technology can convert text data in the image file into electronic text.

[0040] In the related art, a general model that can be used to directly recognize characters in an image can be formed through OCR technology. Then the image containing text data is input into the model to output the recognized content. In the related art, there is a lack of text recognition models dedicated to code extraction (for example, hardware description language Verilog code), and directly using the existing general recognition model to extract code has at least the following problems: the format of code and other text is different, for example, code usually has indentation, comments and other content; and the general model has poor adaptability to indentation, comments and other content in the code, resulting in low accuracy of using the general model to obtain the code; the general model lacks understanding and learning of code semantics, resulting in misrecognition of symbolic elements in the code and neglect of the hierarchical structure. The general model is not suitable for code recognition, so in the related art, the code is usually entered into the local computing device by manual input, which leads to low efficiency and high error rate.

[0041] In this embodiment, a connected region analysis is performed on an image including a code to be identified to obtain a first foreground region; the first foreground region thus obtained is the code region to be identified determined from the shape level. The hue of the pixels in the image is obtained, and the image is divided into regions according to the hue to obtain a second foreground region; the second foreground region thus obtained is the code region to be identified determined from the color level. The color of the code is usually different from the background color. Therefore, the first foreground region obtained based on the connected region analysis and the second foreground region obtained based on the hue of the pixels can accurately reflect the region where the code to be identified is located, and the fused foreground region obtained by fusing the first foreground region and the second foreground region further improves the accuracy of reflecting the location of the code to be identified. Furthermore, the image is segmented based on the fused foreground region, and then the obtained sub-image is input into a pre-trained code recognition model, so that the code to be identified in the image can be accurately and efficiently obtained. The problem of low input efficiency and easy error caused by manual code input in the related art is solved.

[0042] Figure 3 The flowchart of another code recognition method provided by this embodiment is shown. Referring to 3, the method may include the following steps: Step 201: Acquire an image including a code to be identified.

[0043] For example, the code to be identified may be a Verilog code, or other code recorded in an image format.

[0044] Step 202, grayscale the image to obtain a grayscale image.

[0045] Images containing codes (eg, Verilog codes) usually have the characteristics of a single background color, less noise, and a clear color contrast between foreground characters and background.

[0046] For example, if the image is an RGB image, the RGB image can be grayed to further highlight the contrast between the foreground characters and the background in the image, thereby improving the accuracy of subsequent segmentation and recognition. In addition, graying can reduce the amount of calculation in subsequent steps and improve processing efficiency.

[0047] Furthermore, grayscale processing is the process of converting a color image into a grayscale image. Color images usually contain color information of three channels: red (R), green (G), and blue (B), where the grayscale value of each channel generally ranges from 0 to 255, while grayscale images have only one channel, and the grayscale value is also between 0 and 255, representing different grayscale levels from black (0) to white (255).

[0048] For example, a weighted average method can be used to grayscale the image including the code. Specifically, corresponding weights are set for the R, G, and B channels, and the image is grayscaled according to the set weights. The weights can be set according to the sensitivity of the human eye to color. For example, the weights of the R, G, and B channels can be set to 0.299, 0.578, and 0.114, respectively.

[0049] Step 203, performing image binarization processing on the grayscale image to obtain a binary image; Specifically, according to analysis, images including code (e.g., Verilog code) usually have a single background color, and there is a large difference between the background color and the foreground text color. Based on different user settings, code characters with different functions may appear in different colors. For example, the color of the comment part in the code can be set to red, blue, etc. Through binarization processing, the foreground characters and background of the image can be more clearly distinguished.

[0050] Furthermore, in order to ensure the integrity of the text after binarization, the adaptive threshold method can be used to simplify the grayscale image. Binarization is an image processing technology used to convert the grayscale values ​​of pixels in an image into only two values, which can be 0 and 255, where grayscale values ​​0 and 255 represent black and white, respectively.

[0051] Specifically, the grayscale image is divided into multiple sub-regions, and for each sub-region: according to the grayscale value of the pixel in the sub-region and the grayscale value threshold set for the sub-region, the grayscale value in the sub-region is binarized. The grayscale value threshold is equivalent to the binarization threshold for binarization. For example, if the grayscale value of the pixel is greater than the grayscale value threshold, it is directly set to 255, otherwise it is set to 0.

[0052] Furthermore, the codes are usually distributed in straight lines in the image, so the image is divided into rectangular sub-regions, and then the average grayscale value of all pixels in the sub-region is calculated, and the average grayscale value threshold corresponding to the sub-region is determined by the average grayscale value.

[0053] Step 204: Perform connected region analysis on the binary image to obtain a first foreground region.

[0054] The first foreground area is a connected area where the code to be identified is located.

[0055] The connected region analysis is used to mark interconnected pixels as a connected domain to determine the foreground character region. The foreground character region obtained based on the connected region analysis is the first foreground region in this step.

[0056] Furthermore, after the connected region analysis is completed, the centroid of each connected region is output, and the centroid is the seed point of the first foreground region, and the seed point is the shape maker of the first foreground region.

[0057] Step 205, obtaining the hue of the pixel points in the image, and dividing the image according to the hue to obtain a second foreground area.

[0058] The second foreground area is an area formed by the pixels of the code to be identified.

[0059] For example, obtaining the hue of the pixel in the image in step 205 may include: performing color space conversion on the image to obtain the hue of the pixel in the image.

[0060] Step 206, fusing the first foreground area and the second foreground area to obtain a fused foreground area; The first foreground region and the second foreground region have a shape marker and a color marker respectively. The first foreground region and the second foreground region are fused by fusing the shape marker and the color marker to obtain a foreground seed point corresponding to the fused foreground region.

[0061] For example, the pixels in the fused foreground area have a first identifier; the pixels in the first foreground area have a corresponding first preset identifier, and the pixels in the second foreground area have a corresponding second preset identifier. The following method can be used to obtain the corresponding second preset identifier of the pixels in the second foreground area: if the hue reaches a preset hue threshold, then according to the corresponding relationship between the hue threshold and the identifier, obtain the identifier corresponding to the preset hue threshold; determine the identifier corresponding to the preset hue threshold as the second preset identifier of the pixel.

[0062] During the programming process, different colors can be set for codes with different functions, for example, the code body and code comments are set to white and blue respectively. Different colors have corresponding preset hue thresholds. If the hue of a pixel reaches the preset hue threshold, it means that the pixel belongs to the color corresponding to the preset hue threshold. Different colors correspond to different identifiers, and correspondingly, the preset hue thresholds corresponding to different colors also have different identifiers. According to the method of this embodiment, corresponding second preset identifiers can be set for the pixels corresponding to the code in the second foreground area.

[0063] The first identification of the pixel point in the fused foreground area can be obtained according to sub-steps A1 to A2: Sub-step A1, if the pixel point in the image belongs to the first foreground area and the second foreground area, obtain the maximum value of the first preset identifier and the second preset identifier, and set the sum of the maximum value and the preset value as the first identifier of the pixel point.

[0064] For a pixel point in the image, if the pixel point includes a first preset identifier and a second preset identifier, it is determined that the pixel point belongs to a first foreground area and a second foreground area.

[0065] For example, the first preset identifier of the pixel point in the first foreground area is 1, and the second preset identifiers of the pixel points in the second foreground area include 2, 3 to N. In one embodiment, the identifiers of the pixel points are 1 and 3, and the sum N+1 of the maximum value N and the preset value (for example, 1) among the first preset identifier (1) and the second preset identifier (2, 3 to N) is determined as the first identifier of the pixel point.

[0066] Sub-step A2: If the pixel point in the image belongs to the first foreground area or the second foreground area, a preset identifier of the foreground area to which the pixel point belongs is determined as a first identifier of the pixel point.

[0067] For example, if a pixel has a first preset identifier 1, and an identifier of other regions -1, it is determined that the pixel belongs to the first foreground region, and its identifier in the first foreground region is retained, and identifier 1 is determined as its first identifier. For example, if a pixel has a second preset identifier N, and an identifier of other regions -1, it is determined that the pixel belongs to the first foreground region, and its identifier in the first foreground region is retained, and identifier N is determined as its first identifier.

[0068] Step 207, performing edge detection on the image to obtain edge detection results; The fonts of code characters in images are usually thin, so before edge detection is performed on the preprocessed image, morphological processing is performed on it. For example, morphological processing can be performed through a closing operation. Through the closing operation, small holes and broken parts in the characters can be filled to avoid excessive division of small characters, making adjacent characters easier to be recognized as a whole, thereby improving the accuracy of edge detection results.

[0069] For example, step 207 may include sub-steps B1 to B3: Sub-step B1, grayscale processing is performed on the image to obtain a grayscale image.

[0070] The method of this sub-step can refer to the description of the aforementioned step 202, which will not be repeated here.

[0071] Sub-step B2, performing image binarization processing on the grayscale image to obtain a binary image.

[0072] The method of this sub-step can refer to the description of the aforementioned step 203, which will not be repeated here.

[0073] Sub-step B3, performing edge detection on the binary image to obtain an edge detection result.

[0074] For example, edge detection is performed on the binarized image according to the grayscale value of each pixel in the binarized image to obtain an edge detection result.

[0075] Step 208, obtaining the second identifier of the pixel points in the background area of ​​the image and the third identifier of the pixel points in other areas; The other regions are regions in the image other than the fused foreground region and the background region.

[0076] The pixels in the background area have a second identifier, and the pixels in other areas have a third identifier; This step also includes obtaining background seed points, specifically, generating background seed points based on the inferred background position. For example, the four corner vertices and the midpoints of the four edges of the image can be selected as background seed points.

[0077] Furthermore, foreground seed points, background seed points, and other pixel points other than foreground seed points and background seed points can be identified with different numerical values. For example, foreground seed points and background seed points are identified as different positive integers, for example, they can be identified as 1 and 0 respectively; the remaining pixel points can be identified as a negative number, for example, they can be identified as -1.

[0078] The numerical identification of foreground seed points, background seed points, and other pixel points can form a shape marker identification image with the same size as the input image.

[0079] For example, when processing an image based on a connected domain analysis method to obtain a first foreground area, a first preset identifier is set for the first foreground area (for example, the identifier is 1), and identifiers are set for the background area and other areas respectively, the identifier set for the background area is 0, and the identifiers set for other areas are negative numbers (for example, -1); when obtaining a second foreground area based on hue, and a second preset identifier is set for it (for example, 2, 3, ...), identifiers are set for the background area and other areas except the second foreground area and the background area, for example, they are set to 0 and a negative number (for example, -1), respectively.

[0080] The second identifier of the pixel points in the background area of ​​the image and the third identifier of the pixel points in other areas can be obtained according to the following method: After the shape marker and the color marker are fused, if a pixel in the image is marked as 0 in both markers, it means that the pixel is a pixel in the background area, and 0 is determined as the second identifier of other areas.

[0081] After the shape marker and the color marker are fused, if a pixel in the image is marked as a negative number in both markers, it means that the pixel is a pixel of another area, and the negative number (eg, -1) in the aforementioned embodiment is determined as the third identifier of another area.

[0082] Step 209: input the first identifier, the second identifier, the third identifier and the edge detection result into a watershed segmentation model to obtain at least one sub-image.

[0083] Specifically, when performing image segmentation through the watershed segmentation model, it is necessary to set the seed point and calculate the image gradient. Furthermore, the watershed algorithm is a method for image segmentation based on the topological structure of the image. The Canny operator is an algorithm for edge detection, and the Canny operator is specifically used to find edges based on the characteristics of drastic changes in grayscale values ​​at the edges of the image. In this embodiment, the Canny operator can be used to perform edge detection on the code area of ​​the preprocessed image to generate a gradient image, and use it as the boundary input of the watershed algorithm.

[0084] For example, step 209 may include sub-steps C1 to C2: Sub-step C1, inputting the first identifier, the second identifier, the third identifier and the edge detection result into a watershed segmentation model to obtain at least one sub-region.

[0085] Furthermore, according to the first identifier, the second identifier and the third identifier, a seed point marker having the same size as the image can be obtained, and the marker is the fused marker. The fused marker and the edge detection result are input into the watershed segmentation model to obtain at least one sub-region.

[0086] Sub-step C2: if the area of ​​the sub-region is less than or equal to the preset area threshold, merge the sub-region and the sub-region adjacent to the sub-region, and determine the image portion corresponding to the obtained merged region as a sub-image.

[0087] The number of pixels in the sub-region can be obtained, and the area of ​​the sub-region can be represented by the number of pixels.

[0088] Before performing the merge, you need to confirm the adjacency of the segmentation results. Specifically, in the segmentation results of the watershed algorithm, the same area will be marked with the same area identifier, and different areas will have different identifiers. You can traverse all the pixels in the segmentation results. If the area identifiers of the pixel and its surrounding pixels are different, it can be determined that the area where the pixel is located is adjacent to the areas where the surrounding pixels are located.

[0089] Furthermore, sub-step C2 may include the following sub-steps: Sub-step C21, obtaining the difference between the grayscale values ​​of the pixels in the sub-region and the adjacent sub-region.

[0090] The color Euclidean distance between the grayscale value of the pixel in the sub-region and the grayscale value of the pixel in the adjacent sub-region is obtained, and the color Euclidean distance is determined as the difference degree value. Specifically, the RGB value of the pixel is obtained, and the R value, B value, and G value of the pixel in the two sub-regions are used as input parameters of the Euclidean distance function to obtain the color Euclidean distance between the grayscale value of the pixel in the sub-region and the grayscale value of the pixel in the adjacent sub-region.

[0091] Sub-step C22: if the area of ​​the sub-region is smaller than the preset area threshold, or the difference value is smaller than or equal to the preset difference threshold, then the sub-region and the sub-region adjacent to the sub-region are merged.

[0092] By merging regions, we can reduce over-segmentation, avoid segmenting the same character or the same line of characters into different sub-regions, and avoid code prediction based on sub-regions, which leads to low code prediction accuracy.

[0093] Furthermore, in sub-step C2, merging the sub-region and the sub-region adjacent to the sub-region and determining the obtained merged region as a sub-image may include the following sub-steps: Sub-step C23, obtaining the region outline of the merged region.

[0094] For example, the corner points of the merged area may be obtained, and the area outline of the merged area may be determined according to the corner points.

[0095] Sub-step C24, if the region outline is an outline other than a rectangle, obtain the circumscribed rectangle of the merged region, and determine the image portion corresponding to the circumscribed rectangle as a sub-image.

[0096] The codes are usually distributed in straight lines in the image, so the ideal result of segmentation is to segment the codes into long straight rectangles by lines. Therefore, after the regions are merged, the outer contour of each merged region is obtained and determined as the region contour of the merged region.

[0097] If the outer contour is not a rectangle, the minimum circumscribed matrix is ​​fitted according to the corner points of the merged area, and the four vertices of the rectangle are output. The rectangle determined by the four vertices is applied to the original image, and the image part corresponding to the rectangular area is the final sub-image.

[0098] Furthermore, in sub-step C2, merging the sub-region and the sub-region adjacent to the sub-region, and determining the image portion corresponding to the obtained merged region as the sub-image, may include the following sub-steps: Sub-step C25, merging the sub-region and the sub-region adjacent to the sub-region, performing image enhancement processing on the merged region, and obtaining an enhanced image.

[0099] For example, the image enhancement processing of the merged area can be performed by using histogram equalization and sharpening methods. Through histogram equalization processing, the contrast of the light and dark areas of the image can be enhanced, making the background and characters easier to distinguish; through sharpening processing, high-frequency areas such as edges and details in the image can be enhanced to make the edges of characters clearer. For example, the image can be sharpened by using high-pass filter sharpening method.

[0100] Sub-step C26, performing size normalization processing on the enhanced image to obtain a sub-image.

[0101] For example, a bilinear interpolation algorithm may be used to normalize the size of the enhanced image.

[0102] Step 210, extracting image features and semantic features of the sub-image; For example, a text recognition model (Convolutional Recurrent Neural Network, CRNN) can be used to extract features from the input image to obtain visual features. Figure 4 , extract the visual features of the image through the convolutional layer. Then predict the label distribution of the feature vector through the recurrent and transcription layers and integrate the output results. Further, the visual features are extracted through the convolutional layer, and the integrated results of the recurrent and transcription layer outputs are used for semantic feature extraction. Furthermore, the output structure of the convolutional layer and the semantic features obtained by the semantic feature extraction are fused with attention features, and the fusion results are input to the output layer for prediction to obtain the predicted code.

[0103] Specifically, CRNN is a deep learning model for image sequence recognition, which mainly consists of two parts: Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). It combines the feature extraction capability of CNN with the sequence modeling capability of RNN to process image data with sequence features, such as text images, speech spectrograms, etc.

[0104] In this embodiment, the extraction of semantic features is based on the preliminary prediction results of the visual part, and the visual features are extracted through the input representation and encoder layer of the pre-trained model (Bidirectional Encoder Representations from Transformers, BERT). Among them, the BERT model is a pre-trained language model based on the Transformer architecture. The pre-trained BERT model is based on large-scale data training and has strong robustness and generalization. Based on the pre-trained BERT model, the downstream tasks can converge quickly based on the initial parameters during the fine-tuning stage.

[0105] Step 211, perform feature fusion on the image features and the semantic features to obtain fused features.

[0106] Feature fusion is used to establish cross-modal feature associations between images and semantics. The convolutional features of the fused visual model and the features extracted from the semantic sequence are used for the final prediction.

[0107] For example, the attention mechanism is used for feature fusion. Specifically, the visual feature is input as the query feature Q, and the semantic feature is input as the key feature K and the value feature V. The softmax (QK T / )V+H V Perform feature fusion.

[0108] Where Q = W Q H V , K=W K H L , V = W V H L , W Q , W K and W V are the learnable weight matrices, H V and H L They are visual features and semantic features respectively; is the vector dimension.

[0109] Using the attention mechanism for feature fusion enables the model to focus on the most relevant parts for the current task when processing a large amount of information (such as multiple elements in sequence data). In the field of deep learning, the attention mechanism is mainly used to process sequence or structured data such as sentences in natural language processing and pixel areas in images.

[0110] Step 212: predict the code based on the fusion features to obtain the code to be identified in the image For example, a fully connected layer is used to predict the code to be recognized. The fully connected layer is consistent with the output layer of BERT. Specifically, prediction is performed through softmax(hW+b), where W is the weight matrix and b is the bias vector.

[0111] Furthermore, the output layer and semantic feature extraction model are semantically fine-tuned.

[0112] Specifically, first manually generate Verilog signals and ensure that the content of the Verilog signals fully complies with the code syntax rules, and use common abbreviations of the naming rules when naming signals, such as set, clr (clear), clk (clock), rst / rstn (reset / reset-negative), addr (address), req (request), ack (acknowledge), etc., and use underscores to separate them.

[0113] When training the model, 15% of the generated Verilog signals are randomly selected as training data, some internal characters are identified, and these characters are masked or replaced with incorrect characters to perform semantic training on the model. The purpose of semantic training is to predict the original characters that are masked or replaced. Based on the aforementioned semantic training, training is performed using a small amount of annotated image data.

[0114] In this embodiment, image processing and deep learning methods are used to realize automatic recognition of code signals based on images, avoid repetitive labor and human errors caused by manual signal integration, and improve work efficiency. This method combines the visual features of the image and the semantic features of the signal sequence in the model design, has a better feature representation of the Verilog code, has a higher recognition accuracy than the existing general OCR recognition model, and has a stronger perception of symbols, abbreviations, formats, etc. in the signal. This method will segment different signal areas to realize batch recognition of multiple signals, without the need for multiple inputs according to a single signal image, and has higher work efficiency.

[0115] Refer to the following Figure 5 , the method of the present application is further illustrated by reference to Figure 5 , the method may include the following steps: Step S1, preprocessing the image of the Verilog code to obtain a preprocessed image, wherein the preprocessing process includes grayscale processing and binary simplification processing.

[0116] The method of this step has been explained in the aforementioned steps 202 to 203 and will not be repeated here.

[0117] Furthermore, step S1 may include the following sub-steps: Sub-step S11, gray-scale processing is performed on the image of the Verilog code to obtain a gray-scale image.

[0118] The method of this step has been described in the aforementioned step 202 and will not be repeated here.

[0119] Sub-step S12, binarizing and simplifying the grayscale image.

[0120] The method of this step has been described in the aforementioned step 203 and will not be repeated here.

[0121] Step S2, performing morphological processing, connected region analysis and edge detection on the preprocessed image to obtain shape marker identification.

[0122] For example, morphological processing is performed on the preprocessed image through a closing operation.

[0123] Sub-step S21, performing edge detection processing on the pre-processed image to obtain an edge detection processing result.

[0124] The method of this step can refer to the description of the aforementioned step 208 and will not be repeated here.

[0125] Sub-step S22, performing connected region analysis on the morphological processing result, and numerically marking the foreground region, the background region and other regions to obtain shape marker markings.

[0126] The method of this step can refer to the description of the aforementioned steps 204 to 207, which will not be repeated here.

[0127] Step S3, performing color space conversion on the original image to obtain a color marker.

[0128] The method of this step can refer to the description of the aforementioned step 205, which will not be repeated here.

[0129] Step S4: fuse the shape marker and the color marker to obtain the final marker.

[0130] The method of this step can refer to the description of the aforementioned step 206 and will not be repeated here.

[0131] For example, the fusion method may include the following sub-steps: Sub-step S41, if the pixel point at the corresponding point of the shape marker and the color marker is marked as a foreground seed point, it is marked as a new positive integer (such as N+1).

[0132] Where N is the maximum value of shape marker and color marker.

[0133] Sub-step S42: if the pixel is identified as foreground only in one marker and is identified as a negative number in another marker, the foreground identification is retained.

[0134] For example, if a pixel is marked as foreground in the shape marker, which is marked as 1 in the shape marker, and is marked as a negative number in the color marker, the identity of the pixel is retained as the label 1 in the shape marker.

[0135] Sub-step S43: if the pixel point is marked as a negative number in both markers, the negative number mark is retained.

[0136] Sub-step S44: if the pixel point is marked as 0 in both markers, the mark is retained.

[0137] It is marked as 0 in the marker, indicating that the pixel is a background seed point. Since the background seed points in the two markers are set the same, there is no conflict and it is still retained.

[0138] According to the method shown in sub-steps S41 to S44, a new seed point marker having the same size as the input image, ie, the final marker, can be generated.

[0139] Step S5, taking the edge detection result and the final marker as input, performing watershed segmentation, and outputting a preliminary region segmentation result.

[0140] The segmentation result in this step is the sub-region obtained by segmentation in the above-mentioned embodiment.

[0141] Step S6, performing post-segmentation processing on the image, specifically including region merging and shape regularization.

[0142] When merging regions, it is necessary to confirm the adjacency relationship of the pixels. After confirming the adjacency relationship, the adjacency relationship can be stored in the form of a dictionary, such as {a:b1,b2,…}, which means that sub-region b1 and sub-region b2 are both adjacent regions of sub-region a.

[0143] Furthermore, it is possible to determine whether to merge with the adjacent area from the perspectives of the area of ​​the segmented area and the average color of the code. The specific steps include: Sub-step S61, traverse the pixel points, calculate the area of ​​each marked segmented region, and set an area threshold. The area smaller than the threshold will be merged into the adjacent area.

[0144] For example, the area threshold can be set according to user needs, for example, it can be set to 50.

[0145] Sub-step S62, based on the RGB original image, calculate the color Euclidean distance of adjacent areas, if the distance is less than the set color threshold, merge the two areas.

[0146] According to the above two merging strategies, all marked areas are traversed and the adjacency relationship dictionary is updated. Further, the merging process can be repeated to avoid the existence of over-segmented areas after a single merging.

[0147] After the regions are merged, the outer contour of each region is obtained, and the minimum external matrix is ​​fitted on this basis and the four vertices of the rectangle are output. The rectangle represented by the four vertices is projected onto the original image, and the area corresponding to the rectangle in the original image is the final result of the segmented region.

[0148] Step S7, performing image enhancement and normalization processing on the segmented image.

[0149] Through the image segmentation, segmentation region enhancement and size normalization processing in the above steps, the signal region can be highlighted.

[0150] Specifically, the segmented sub-images will be sequentially enhanced and resized to meet the needs of the recognition model and improve recognition accuracy.

[0151] Step S8, using a method combining visual features and semantic features to perform code character recognition.

[0152] The method of this step has been explained in the aforementioned steps 210 to 211 and will not be repeated here.

[0153] Specific, combined Figure 6 , preprocess the original image to obtain a preprocessed image, perform image segmentation and segmentation region enhancement on the preprocessed image to obtain an image enhanced sub-image, normalize the sub-image size to obtain a processed sub-image. Perform visual feature extraction on the processed sub-image, perform text prediction and semantic feature extraction, fuse the visual features and semantic features, input the fusion result into the output layer, perform code prediction through the output layer, and obtain the recognized code.

[0154] Step S8 may include the following sub-steps: Sub-step S81, extracting visual features from the size-normalized image using a visual feature model, and outputting a recognition result.

[0155] Specifically, text prediction is performed based on the extracted visual features to obtain recognition results.

[0156] Sub-step S82, inputting the recognition result into the semantic model and extracting semantic features.

[0157] Among them, the semantic model can be a BERT model.

[0158] Sub-step S83, the semantic features are fused with the initially extracted visual features through an attention fusion mechanism.

[0159] Sub-step S84, inputting the fused features into the output layer to generate semantically corrected prediction results.

[0160] In this embodiment, first, an image containing a Verilog signal definition or declaration needs to be input into the system, such as a signal screenshot of a code interface. The system will automatically pre-process, segment, and recognize the input image. After processing, the recognition result of the signal will be directly output.

[0161] The input image may contain multiple signal instances. This system supports images containing multiple signals, and different signals in the recognition results will be listed in separate rows.

[0162] Reference Figure 7 , obtain the preprocessed image, perform morphological processing and connected domain analysis on the preprocessed image in turn, and obtain the shape marker; perform edge detection on the preprocessed image to obtain the edge detection result. Perform color space conversion on the original image to obtain the color marker. Fuse the shape marker and the color marker, use the watershed segmentation method to segment the image according to the edge detection result and the fused marker, and perform post-segmentation processing on the segmented image, and then perform image enhancement and normalization processing on the post-segmentation processed image to obtain the processed image. Combined with Figure 6 , the processed image is subjected to model recognition processing to identify the code to be recognized in the image. This embodiment proposes an image-based text recognition method for the signal extraction scenario of code (for example, Verilog code), and realizes efficient and accurate code signal recognition through steps such as image preprocessing, signal area positioning, visual feature extraction, and semantic feature extraction.

[0163] refer to Figure 8 , which shows a code recognition device provided in an embodiment of the present application, the device 30 includes: The first acquisition module 301 is used to acquire an image including a code to be identified, and perform a connected region analysis on the image to obtain a first foreground region; the first foreground region includes a connected region where the code to be identified is located; The second acquisition module 302 is used to acquire the hue of the pixel points in the image, and divide the image according to the hue to obtain a second foreground area; the second foreground area includes an area composed of the pixel points of the code to be identified; A third acquisition module 303 is used to fuse the first foreground area and the second foreground area to obtain a fused foreground area, and to segment the image based on the fused foreground area to obtain at least one sub-image; The fourth acquisition module 304 is used to input at least one sub-image into a pre-trained code recognition model to obtain a code to be recognized in the image.

[0164] For example, the pixel points in the fused foreground area have a first identifier; the third acquisition module 303 may include: The first acquisition submodule is used to perform edge detection on the image to obtain edge detection results; The second acquisition submodule is used to acquire the second identifier of the pixel points in the background area of ​​the image and the third identifier of the pixel points in other areas; the other areas are the areas in the image except the fused foreground area and the background area; The third acquisition submodule is used to input the first identifier, the second identifier, the third identifier and the edge detection result into a watershed segmentation model to obtain at least one sub-image.

[0165] Optionally, the pixel points in the first foreground area have a corresponding first preset identifier, and the pixel points in the second foreground area have a corresponding second preset identifier; the device 30 further includes: a fifth acquisition module, configured to acquire a maximum value of the first preset identifier and the second preset identifier if the pixel in the image belongs to the first foreground area and the second foreground area, and set the sum of the maximum value and the preset value as the first identifier of the pixel; The sixth acquisition module is used to determine the preset identifier of the foreground area to which the pixel point belongs as the first identifier of the pixel point if the pixel point in the image belongs to the first foreground area or the second foreground area.

[0166] Optionally, the second acquisition module 302 includes: The fourth acquisition submodule is used to perform color space conversion on the image to obtain the hue of the pixel points in the image; A fifth acquisition submodule, configured to acquire, if the hue reaches a preset hue threshold, an identifier corresponding to the preset hue threshold according to a correspondence between the hue threshold and the identifier; The first determination submodule is used to determine the identifier corresponding to the preset color tone threshold as the second preset identifier of the pixel point.

[0167] Optionally, the third acquisition submodule includes: A first acquisition unit, used for inputting the first identifier, the second identifier, the third identifier and the edge detection result into a watershed segmentation model to obtain at least one sub-region; The first merging unit is used to merge the sub-region and the sub-region adjacent to the sub-region if the region area of ​​the sub-region is less than or equal to a preset region area threshold, and determine the obtained merged region as a sub-image.

[0168] Optionally, the first merging unit includes: A first acquisition subunit is used to acquire a difference value between the grayscale values ​​of pixels in a subregion and an adjacent subregion; The first merging subunit is used to merge the subregion and the subregion adjacent to the subregion if the area of ​​the subregion is smaller than a preset area threshold, or the difference value is smaller than or equal to the preset difference threshold.

[0169] Optionally, the first merging unit includes: A second acquisition subunit is used to acquire the area outline of the merged area; The second merging subunit is used to obtain the circumscribed rectangle of the merged area if the area outline is other than a rectangle, and determine the area corresponding to the circumscribed rectangle as a sub-image.

[0170] Optionally, the first merging unit includes: A first processing subunit is used to merge the subregion and the subregion adjacent to the subregion, and perform image enhancement processing on the merged region to obtain an enhanced image; The second processing sub-unit is used to perform size normalization processing on the enhanced image to obtain a sub-image.

[0171] Optionally, the first acquisition submodule includes: A second acquisition unit is used to grayscale the image to obtain a grayscale image; The second acquisition unit is used to perform image binarization processing on the grayscale image to obtain a binary image; The third acquisition unit is used to perform edge detection on the binary image to obtain an edge detection result.

[0172] Optionally, the fourth acquisition module 304 includes: An extraction submodule, used to extract image features and semantic features of sub-images; The feature fusion submodule is used to fuse image features and semantic features to obtain fused features; The sixth acquisition submodule is used to predict the code based on the fusion features to obtain the code to be identified in the image.

[0173] Optionally, the first acquisition module 301 obtains the first foreground area, including: The seventh acquisition submodule is used to grayscale the image to obtain a grayscale image; An eighth acquisition submodule is used to perform image binarization processing on the grayscale image to obtain a binary image; The ninth acquisition submodule is used to perform connected region analysis on the binary image to obtain a first foreground region.

[0174] In this embodiment, a connected region analysis is performed on an image including a code to be identified to obtain a first foreground region; the first foreground region thus obtained is the code region to be identified determined from the shape level. The hue of the pixels in the image is obtained, and the image is divided into regions according to the hue to obtain a second foreground region; the second foreground region thus obtained is the code region to be identified determined from the color level. The color of the code is usually different from the background color. Therefore, the first foreground region obtained based on the connected region analysis and the second foreground region obtained based on the hue of the pixels can accurately reflect the region where the code to be identified is located, and the fused foreground region obtained by fusing the first foreground region and the second foreground region further improves the accuracy of reflecting the location of the code to be identified. Furthermore, the image is segmented based on the fused foreground region, and then the obtained sub-image is input into a pre-trained code recognition model, so that the code to be identified in the image can be accurately and efficiently obtained. The problem of low input efficiency and easy error caused by manual code input in the related art is solved.

[0175] Fig. 9 4 is a block diagram of an electronic device 400 according to an exemplary embodiment. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0176] Reference Fig. 9 , the electronic device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0177] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0178] The memory 404 is used to store various types of data to support the operation of the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, images, multimedia, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0179] The power supply component 406 provides power to the various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0180] The multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the demarcation of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0181] The audio component 410 is used to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the electronic device 400 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0182] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0183] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of components, such as the display and keypad of the electronic device 400, and the sensor assembly 414 can also detect the position change of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0184] The communication component 416 is used to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0185] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a code recognition method provided in an embodiment of the present application.

[0186] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the above instructions can be executed by a processor 420 of an electronic device 400 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0187] Fig.10 is a block diagram of an electronic device 500 according to an exemplary embodiment. For example, the electronic device 500 may be provided as a server. Fig.10 , the electronic device 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532, for storing instructions that can be executed by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform a code recognition method provided in an embodiment of the present application.

[0188] The electronic device 500 may also include a power supply component 526 configured to perform power management of the electronic device 500, a wired or wireless network interface 550 configured to connect the electronic device 500 to a network, and an input / output (I / O) interface 558. The electronic device 500 may operate based on an operating system stored in the memory 532, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0189] An embodiment of the present application also provides a computer program product, including a computer program, and a code recognition method implemented when the computer program is executed by a processor.

[0190] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0191] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0192] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0193] The above is a detailed introduction to a code recognition method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A code recognition method, characterized in that: include: Acquire an image including a code to be identified, and perform connected region analysis on the image to obtain a first foreground region; The first foreground area includes a connected area where the code to be identified is located; Acquire the hue of the pixel points in the image, and divide the image according to the hue to obtain a second foreground area; the second foreground area includes an area composed of the pixel points of the code to be identified; fusing the first foreground area and the second foreground area to obtain a fused foreground area, and segmenting the image based on the fused foreground area to obtain at least one sub-image; At least one of the sub-images is input into a pre-trained code recognition model to obtain a code to be recognized in the image.

2. The method according to claim 1, characterized in that The pixel points in the fused foreground area have a first identifier; and the image is segmented based on the fused foreground area to obtain at least one sub-image, including: Performing edge detection on the image to obtain an edge detection result; Obtaining a second identifier of a pixel point in a background area of ​​the image and a third identifier of a pixel point in other areas; the other areas are areas of the image other than the fused foreground area and the background area; The first identifier, the second identifier, the third identifier and the edge detection result are input into a watershed segmentation model to obtain at least one of the sub-images.

3. The method according to claim 2, characterized in that The pixels in the first foreground area have a corresponding first preset identifier, and the pixels in the second foreground area have a corresponding second preset identifier; the method further includes: If a pixel point in the image belongs to the first foreground area and the second foreground area, obtaining a maximum value between the first preset identifier and the second preset identifier, and setting a sum of the maximum value and the preset value as the first identifier of the pixel point; If a pixel point in the image belongs to the first foreground area or the second foreground area, a preset identifier of the foreground area to which the pixel point belongs is determined as a first identifier of the pixel point.

4. The method according to claim 2, characterized in that: Inputting the first identifier, the second identifier, the third identifier and the edge detection result into a watershed segmentation model to obtain at least one sub-image includes: Inputting the first identifier, the second identifier, the third identifier and the edge detection result into a watershed segmentation model to obtain at least one sub-region; If the area of ​​the sub-region is less than or equal to a preset area threshold, the sub-region and a sub-region adjacent to the sub-region are merged, and the obtained merged region is determined as the sub-image.

5. The method according to claim 4, characterized in that If the area of ​​the sub-region is less than or equal to a preset area threshold, merging the sub-region with a sub-region adjacent to the sub-region includes: Obtaining a difference value between the grayscale values ​​of pixels in the sub-region and the adjacent sub-region; If the area of ​​the sub-region is less than or equal to the preset area threshold, or the difference value is less than or equal to the preset difference threshold, the sub-region and the sub-region adjacent to the sub-region are merged.

6. The method according to claim 4, characterized in that The merging of the sub-region and a sub-region adjacent to the sub-region, and determining the obtained merged region as the sub-image, includes: Obtaining a region outline of the merged region; If the region outline is an outline other than a rectangle, the bounding rectangle of the merged region is obtained, and the region corresponding to the bounding rectangle is determined as the sub-image.

7. The method according to claim 1, characterized in that Inputting at least one of the sub-images into a pre-trained code recognition model to obtain a code to be recognized in the image, comprising: Extracting image features and semantic features of the sub-image; Performing feature fusion on the image feature and the semantic feature to obtain a fused feature; Code prediction is performed based on the fusion features to obtain the code to be recognized in the image.

8. A code recognition device, characterized in that: The device comprises: A first acquisition module is used to acquire an image including a code to be identified, and perform a connected region analysis on the image to obtain a first foreground region; the first foreground region includes a connected region where the code to be identified is located; A second acquisition module is used to acquire the hue of the pixel points in the image, and divide the image according to the hue to obtain a second foreground area; the second foreground area includes an area composed of the pixel points of the code to be identified; A third acquisition module, configured to fuse the first foreground area and the second foreground area to obtain a fused foreground area, and segment the image based on the fused foreground area to obtain at least one sub-image; The fourth acquisition module is used to input at least one of the sub-images into a pre-trained code recognition model to obtain the code to be recognized in the image.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images