Waybill photo auditing method and device based on multi-modal analysis

Abstract

The invention relates to the field of image analysis, in particular to a waybill photo auditing method and device based on multi-modal analysis, and the method comprises the steps: obtaining a waybill photo, and determining a feature region judgment mode according to a surrounding coefficient and an influence threshold; determining a correlation analysis mode according to the characteristic line interaction coefficient and the characteristic area disorder threshold, wherein the correlation analysis mode is an optimization mode determined according to the characteristic area category or a correlation area determined according to the correlation combination; determining associated region categories according to the region influence coefficients and the region fluctuation amplitudes; determining a fuzzy state of each second-class association region based on the smoothing coefficient and the cross influence coefficient, and determining a region processing mode according to the fuzzy state; and determining whether the waybill information is qualified or not according to the matching coefficient of the waybill information corresponding to the first-class associated region and the text information corresponding to the second-class associated region and the sensitivity coefficient of the text information corresponding to the second-class associated region. According to the method and the device, the accuracy of a waybill photo auditing process can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image analysis, and in particular to a method and device for waybill photo review based on multimodal analysis. Background Art

[0002] In recent years, with the rapid development of e-commerce, the volume of express delivery business has grown rapidly, bringing unprecedented opportunities to the logistics industry. However, in the process of reviewing the waybill photos, due to the different quality of the waybill photos, the existence of blurred handwriting and overlapping handwriting, review errors may occur, which may lead to the wrong delivery of the express. Therefore, how to process the waybill photos to improve the accuracy of the review process is a technical problem that needs to be solved urgently by those skilled in the art.

[0003] Chinese patent publication number CN107622247B discloses a method for locating and extracting express waybills, including: step 1: select the existing printed pattern or character key area in the reference express waybill as the reference position, and perform a graph representation; step 2: perform a graph representation of the express waybill to be processed based on the candidate key area obtained by image segmentation; step 3: calculate the similarity between the express waybill to be processed and the reference express waybill according to the attributes of the graph to match the graph, determine the position mapping of the reference express waybill to the express waybill to be processed, and realize the positioning of the area of ​​interest of the user on the express waybill to be processed; step 4: perform post-processing operations such as image rotation, segmentation, character block relocation, etc. according to the positioning results of the relevant information, and complete the extraction of the recipient's name, telephone number, receiving address and other information of interest to the user in the express waybill. It can be seen that the above technical solution has the following problems: the area of ​​interest is determined only based on the reference express waybill, and the information extraction process is not adaptively optimized according to the actual state of the area, resulting in the problem of poor recognition accuracy of the waybill information. Summary of the invention

[0004] To this end, the present invention provides a method and device for waybill photo review based on multimodal analysis, so as to overcome the problem in the prior art that the information extraction process is not adaptively optimized according to the actual status of the region, resulting in poor accuracy in waybill information recognition.

[0005] To achieve the above object, the present invention provides a method for reviewing waybill photos based on multimodal analysis, comprising: Obtain the waybill photo and determine the feature area determination method based on the encirclement coefficient and the influence threshold. The feature area determination method is to determine the feature area based on the connected domain or the correlation coefficient; Determine the association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold, and the association analysis method is to determine the optimization method according to the characteristic area category or determine the association area according to the association combination; The characteristic area category is determined according to the longitudinal displacement coefficient and the characteristic influence threshold, and the optimization method is to determine the associated area according to the distance reference value and the similarity coefficient, or to determine the associated area according to the characteristic line; Determine the category of associated regions according to the regional influence coefficient and the regional fluctuation amplitude; The fuzzy state of each of the two types of associated regions is determined based on the smoothing coefficient and the cross-influence coefficient, and the region processing method is determined according to the fuzzy state. The region processing method is to determine the pooling method according to the pixel fluctuation value or to determine the cutting method of the pre-cut feature area according to the gradient change value; The pooling method is to adjust the pooling moving step of the pooling layer according to the hierarchical influence coefficient or to adjust the pooling window according to the comprehensive evaluation value, and the cutting method is to cut according to the gradient line or pixel similarity; Whether the waybill information is qualified is determined according to the matching coefficient between the waybill information corresponding to the first-class associated area and the text information corresponding to the second-class associated area and the sensitivity coefficient of the text information corresponding to the second-class associated area.

[0006] Further, if the enclosing coefficient is less than the preset enclosing coefficient and the influence threshold is less than the preset influence threshold, the characteristic area determination method is to determine the characteristic area according to the connected domain; If the enclosing coefficient is greater than or equal to the preset enclosing coefficient or the influence threshold is greater than or equal to the preset influence threshold, the feature area determination method is to determine the feature area according to the correlation coefficient.

[0007] Furthermore, if the characteristic line interaction coefficient is greater than or equal to the preset characteristic line interaction coefficient or the characteristic region disorder threshold is greater than or equal to the preset characteristic region disorder threshold, the association analysis method is to determine the association region according to the association combination, wherein, The smallest rectangle that can contain the feature area corresponding to a single association combination is recorded as an association area; The association combination is determined according to the area normalization degree and the distance reference value.

[0008] Furthermore, if the characteristic line interaction coefficient is less than the preset characteristic line interaction coefficient and the characteristic area disorder threshold is less than the preset characteristic area disorder threshold, the association analysis method is to determine the optimization method according to the characteristic area category, wherein, For the longitudinal feature area, the optimization method is to determine the associated area based on the distance reference value and the similarity coefficient; For the horizontal feature area, the optimization method is to determine the associated area based on the feature line; When determining the associated region according to the feature line, the region to be analyzed is determined according to the matching reference value. For a single region to be analyzed, if the density of the feature region is less than the preset density of the feature region, the region to be analyzed is segmented by the cutting line to obtain the associated region; If the density of the feature area is greater than or equal to the preset density of the feature area, the area to be analyzed is taken as the associated area.

[0009] Furthermore, the characteristic area categories include: a longitudinal characteristic area whose longitudinal slip coefficient is less than a preset longitudinal slip coefficient and whose characteristic influence threshold is less than a preset characteristic influence threshold, and a lateral characteristic area whose longitudinal slip coefficient is greater than or equal to the preset longitudinal slip coefficient or whose characteristic influence threshold is greater than or equal to the preset characteristic influence threshold.

[0010] Furthermore, the associated area categories include: a type of associated area whose regional influence coefficient is greater than or equal to a preset regional influence coefficient and whose regional fluctuation amplitude is greater than or equal to a preset regional fluctuation amplitude, and a type of associated area whose regional influence coefficient is less than a preset regional influence coefficient or whose regional fluctuation amplitude is less than a preset regional fluctuation amplitude.

[0011] Furthermore, if the blur state of the second type of associated area is that the smoothing coefficient is less than the preset smoothing coefficient or the cross-influence coefficient is less than the preset cross-influence coefficient, the area processing method is to determine the pooling method according to the pixel fluctuation value; If the pixel fluctuation value is greater than or equal to the preset pixel fluctuation value, the pooling method is to reduce the pooling moving step size of the pooling layer according to the hierarchical influence coefficient. If the pixel fluctuation value is less than the preset pixel fluctuation value, the pooling method is to reduce the pooling window area according to the comprehensive evaluation value; The pooling layer is determined according to the pixel threshold and the level threshold, and the reduction value of the pooling moving step corresponding to a single pooling layer is positively correlated with the level influence coefficient corresponding to the pooling layer; The reduction value of the pooling window area is positively correlated with the comprehensive evaluation value.

[0012] Further, if the fuzzy state of the second type of associated region is that the smoothness coefficient is greater than or equal to the preset smoothness coefficient and the cross-influence coefficient is greater than or equal to the preset cross-influence coefficient, the region processing method is to determine the cutting method of the pre-cut feature region according to the gradient change value; For a single pre-cut feature area, If the gradient change value is greater than or equal to the preset gradient change value, the cutting method is to cut according to the gradient line; If the gradient change value is less than the preset gradient change value, the cutting method is to cut according to the pixel similarity; The pre-cut feature area is a feature area in the second type of associated area where the evaluation threshold is greater than a preset evaluation threshold.

[0013] Further, whether the waybill information is qualified is determined according to the matching coefficient between the waybill information corresponding to the first type of associated area and the text information corresponding to the second type of associated area and the sensitivity coefficient of the text information corresponding to the second type of associated area, wherein: If the matching coefficient is less than the preset matching coefficient or the sensitivity coefficient is greater than or equal to the preset sensitivity coefficient, the waybill information is unqualified; If the matching coefficient is greater than or equal to the preset matching coefficient and the sensitivity coefficient is less than the preset sensitivity coefficient, the waybill information is qualified.

[0014] The present invention also provides a device for reviewing waybill photos based on multimodal analysis, comprising: Waybill collection module, used to obtain waybill photos; A feature area recognition module, which is connected to the waybill collection module, is used to determine a feature area determination method according to an encirclement coefficient and an influence threshold, wherein the feature area determination method is to determine the feature area according to a connected domain or a correlation coefficient; An association analysis module, which is connected to the waybill collection module and the feature area identification module respectively, and is used to determine an association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold. The association analysis method is to determine an optimization method according to the characteristic area category or determine the association area according to the association combination, and determine the association area category according to the regional influence coefficient and the regional fluctuation amplitude; A region processing module, which is connected to the association analysis module, is used to determine the fuzzy state of each of the two types of association regions based on the smoothing coefficient and the cross-influence coefficient, and determine the region processing method according to the fuzzy state. The region processing method is to determine the pooling method according to the pixel fluctuation value or to determine the cutting method of the pre-cut feature area according to the gradient change value; A waybill review module is connected to the association analysis module and the area processing module, and is used to determine whether the waybill information is qualified based on the matching coefficient between the waybill information corresponding to the first type of association area and the text information corresponding to the second type of association area and the sensitivity coefficient of the text information corresponding to the second type of association area.

[0015] Compared with the prior art, the beneficial effect of the present invention lies in that, in the technical scheme of the present invention, the actual state of the connected domain is effectively reflected by the encirclement coefficient and the influence threshold, and then different feature area determination methods are adaptively selected according to the encirclement coefficient and the influence threshold, so that the selection of the feature area determination method is more in line with the actual application scenario, avoiding the problem of poor character recognition accuracy caused by the inability to divide the connected domain corresponding to the same character into the same feature area, helping to reduce misrecognition and missed recognition, thereby improving the overall accuracy of character recognition, and then improving the accuracy of waybill photo review.

[0016] Furthermore, in the present invention, the disorder degree of the feature area is effectively reflected by the feature line interaction coefficient and the feature area disorder threshold, and then different association analysis methods are adaptively selected according to the feature line interaction coefficient and the feature area disorder threshold, so that the selected association analysis method can better adapt to the actual state of the feature area, avoiding the defect that the relevant information cannot be effectively combined when the degree of disorder of the feature area in the waybill photo is large, thereby improving the accuracy of information recognition.

[0017] Furthermore, in the present invention, the feature area category is determined according to the longitudinal slip coefficient and the feature influence threshold, and the arrangement state of the feature area is effectively reflected by the longitudinal slip coefficient and the feature influence threshold. Then, different optimization methods are adaptively selected according to the feature area category, so that the associated area determined by the optimization method can effectively combine information with a high degree of association, thereby improving the accuracy of information recognition.

[0018] Furthermore, the present invention effectively reflects the degree of association between connected domains in the region through the regional influence coefficient and the regional fluctuation amplitude, and then determines the category of the associated region according to the regional influence coefficient and the regional fluctuation amplitude, which can effectively distinguish the character region and the barcode region, thereby facilitating improving the accuracy of subsequent information review.

[0019] Furthermore, in the present invention, the fuzzy state of each of the two related areas is determined based on the smoothing coefficient and the cross-influence coefficient, and the smoothing coefficient and the cross-influence coefficient are used to effectively reflect the degree of fuzziness and overlap of the characters in the two related areas. Then, different area processing methods are adaptively selected according to the fuzzy state, so that the selected area processing method can improve the accuracy of character recognition and reduce the problem of poor accuracy in waybill photo review caused by incorrect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of a method for reviewing waybill photos based on multimodal analysis of the present invention; Figure 2 This is a flow chart of the present invention for determining the association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold; Figure 3 A flow chart of a method for determining a region processing according to a fuzzy state according to the present invention; Figure 4 This is a module connection diagram of a device for waybill photo review of the method for waybill photo review based on multimodal analysis of the present invention. DETAILED DESCRIPTION

[0021] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0023] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0024] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0025] See also Figures 1 to 3 As shown, the present invention provides a method for reviewing waybill photos based on multimodal analysis, comprising: Obtain the waybill photo and determine the feature area determination method based on the encirclement coefficient and the influence threshold. The feature area determination method is to determine the feature area based on the connected domain or the correlation coefficient; Determine the association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold, and the association analysis method is to determine the optimization method according to the characteristic area category or determine the association area according to the association combination; The characteristic area category is determined according to the longitudinal displacement coefficient and the characteristic influence threshold, and the optimization method is to determine the associated area according to the distance reference value and the similarity coefficient, or to determine the associated area according to the characteristic line; Determine the category of associated regions according to the regional influence coefficient and the regional fluctuation amplitude; The fuzzy state of each of the two types of associated regions is determined based on the smoothing coefficient and the cross-influence coefficient, and the region processing method is determined according to the fuzzy state. The region processing method is to determine the pooling method according to the pixel fluctuation value or to determine the cutting method of the pre-cut feature area according to the gradient change value; The pooling method is to adjust the pooling moving step of the pooling layer according to the hierarchical influence coefficient or to adjust the pooling window according to the comprehensive evaluation value, and the cutting method is to cut according to the gradient line or pixel similarity; Whether the waybill information is qualified is determined according to the matching coefficient between the waybill information corresponding to the first-class associated area and the text information corresponding to the second-class associated area and the sensitivity coefficient of the text information corresponding to the second-class associated area.

[0026] The application scenario of the present invention is the review of waybill photos. In the present invention, the waybill photos are converted into grayscale images through OpenCV. The grayscale image contains a number of pixels, and the pixel value corresponding to a single pixel is greater than or equal to 0 and less than or equal to 255. This is easy for those skilled in the art to understand and will not be described in detail. The grayscale image contains several connected domains. A single connected domain is a region containing several pixels. The pixel values ​​of each pixel in the connected domain are all greater than 0 and the pixel values ​​of the adjacent pixels corresponding to the connected domain are all 0. The confirmation method of the adjacent pixels is that, for a single connected domain, each pixel in the connected domain is recorded as a target pixel, and other pixels except the target pixel are recorded as reference pixels. Each reference pixel adjacent to the target pixel is recorded as an adjacent pixel corresponding to the connected domain. In the present invention, several historical records are correspondingly provided, and any historical record records the influence threshold, correlation coefficient, feature area disorder threshold, regional normalization degree, and feature area density, etc., of at least one historical record in the historical process of the waybill photo review, and each historical record corresponds to a qualified mark, which records whether the waybill photo review process meets the user's needs. The qualified mark can be recorded manually. It can be understood that the user can determine whether the stability detection process meets the needs according to the self-set indicators. The self-set indicators can be but not limited to the false positive rate, which will not be elaborated here. Among them, the false positive rate is the number of times the waybill photo is wrongly judged to be appropriate; It should be noted that when determining the feature area determination method based on the encirclement coefficient and the influence threshold, the feature area determination method is determined one by one for each connected domain that is not recorded as a feature area until each connected domain is recorded as a feature area; When determining the association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold, the association analysis method of each characteristic area to be analyzed is determined one by one until each characteristic area to be analyzed is recorded as an associated area, and the characteristic area to be analyzed is a characteristic area that is not recorded in the associated area.

[0027] Specifically, if the enclosing coefficient is less than the preset enclosing coefficient and the influence threshold is less than the preset influence threshold, the characteristic area determination method is to determine the characteristic area according to the connected domain; If the enclosing coefficient is greater than or equal to the preset enclosing coefficient or the influence threshold is greater than or equal to the preset influence threshold, the feature area determination method is to determine the feature area according to the correlation coefficient.

[0028] The feature area determination method is that when determining the feature area according to the connected domain, the smallest rectangle that can contain the connected domain is recorded as a feature area; When the feature area determination method is to determine the feature area according to the correlation coefficient, for a single connected domain, the connected domain is recorded as a target connected domain, and the minimum rectangle that can contain the target connected domain and each matching connected domain corresponding to the target connected domain is recorded as a feature area; the matching connected domain is a connected domain whose correlation coefficient with the target connected domain is greater than a preset correlation coefficient; The positional relationship between each edge corresponding to a single feature area and each edge corresponding to the grayscale image is vertical or parallel; The method of confirming the encirclement coefficient is as follows: for a single connected domain, the connected domain is recorded as the target connected domain, the minimum rectangle that can contain the target connected domain is recorded as the first rectangle, and the minimum rectangle that can contain the single adjacent connected domain corresponding to the target connected domain is recorded as the second rectangle. The positional relationship between the sides corresponding to the first rectangle and the second rectangle and the sides corresponding to the grayscale image are both vertical or parallel. The encirclement coefficient corresponding to the target connected domain is the maximum value of the sub-encirclement coefficients corresponding to the adjacent connected domains of the target connected domain. The sub-encirclement coefficient = the area of ​​the overlapping area of ​​the first rectangle and the second rectangle / the smaller value of the area of ​​the first rectangle and the area of ​​the second rectangle; the line connecting any point in the single adjacent connected domain corresponding to the target connected domain and any point in the target connected domain does not pass through other connected domains except the adjacent connected domain and the target connected domain; The influence threshold is the minimum value among the sub-influence thresholds corresponding to each second region. The sub-influence threshold is confirmed in the following manner: for a single second region, the second region is recorded as the target second region, and the sub-influence threshold corresponding to the target second region = edge length similarity + (sum of the areas of the target second region and the first region / area of ​​the minimum rectangle that can contain the target second region and the first region). The edge length similarity is the maximum value of the length similarities corresponding to the edges of the target second region. For a single edge in the target second region, the edge is recorded as the target edge, and the minimum value of the length difference between the target edge and the edges in the first region is recorded as the length similarity corresponding to the target edge. The correlation coefficient is confirmed by: If the distance reference value is greater than the preset distance reference value, the correlation coefficient = interval influence coefficient + outlier coefficient; If the distance reference value is less than the preset distance reference value, the correlation coefficient = encirclement coefficient + influence threshold; The confirmation method of the interval influence coefficient is as follows: for any two connected domains, they are respectively recorded as the first connected domain and the second connected domain, and the minimum rectangle that can contain the first connected domain and the second connected domain is recorded as the reference rectangle. The interval influence coefficient = intersection edge length + interval reference value, the intersection edge length is the length of the intersection edge among the four sides corresponding to the reference rectangle, the intersection edge is the length of the four sides corresponding to the reference rectangle overlapping with the remaining connected domain, the remaining connected domain is other connected domains except the first connected domain and the second connected domain, the interval reference value is the shortest distance from the first connected domain to the second connected domain, and the confirmation method of the outlier coefficient is as follows: the remaining connected domain through which the line connecting the midpoint of the first connected domain and the midpoint of the second connected domain passes, and the first connected domain and the second connected domain are all recorded as analysis connected domains, the maximum distance in the neighboring intervals corresponding to each analysis connected domain is recorded as the maximum interval, the neighboring interval corresponding to a single analysis connected domain is the maximum value of the shortest distances from the analysis connected domain to each analysis connected domain adjacent to the analysis connected domain, and the midpoint of a single connected domain is the center of the circumscribed circle of the connected domain; The values ​​of the preset encirclement coefficient, the preset influence threshold and the preset correlation coefficient can be determined by the user according to the actual application scenario. The greater the user's demand for improving the accuracy of feature area determination, the smaller the values ​​of the preset encirclement coefficient and the preset influence threshold, and the larger the value of the preset correlation coefficient. The values ​​of the preset encirclement coefficient, the preset influence threshold and the preset correlation coefficient are provided. The preset encirclement coefficient is 80%. The historical records of the feature area are detected according to the correlation coefficient, and the average value of the influence threshold corresponding to the historical records that can meet the user's needs is recorded as the preset influence threshold, and the average value of the correlation coefficient corresponding to the historical records that can meet the user's needs is recorded as the preset correlation coefficient.

[0029] Specifically, if the characteristic line interaction coefficient is greater than or equal to the preset characteristic line interaction coefficient or the characteristic area disorder threshold is greater than or equal to the preset characteristic area disorder threshold, the association analysis method is to determine the association area according to the association combination, wherein, The smallest rectangle that can contain the feature area corresponding to a single association combination is recorded as an association area; The association combination is determined according to the area normalization degree and the distance reference value.

[0030] Among them, the characteristic line interaction coefficient is the maximum value of the interaction reference values ​​corresponding to each characteristic line, the interaction reference value corresponding to a single characteristic line is the sum of the lengths of the characteristic segments corresponding to the characteristic line, the characteristic segment is the line segment where the characteristic line coincides with the interior of the characteristic area, and the interior of the characteristic area is the other area in the characteristic area except the edge of the characteristic area; The method of confirming the characteristic line is that, for a single characteristic area, the straight lines where the two longer sides of the characteristic area are located are taken as characteristic lines, and a single characteristic area corresponds to two characteristic lines; A similarity analysis is performed on each feature area. When a similarity analysis is performed on a single feature area, the feature area is recorded as a target feature area, and other feature areas except the target feature area are recorded as reference feature areas. A set of reference feature areas whose similarity coefficient with the target feature area is greater than or equal to a preset similarity coefficient is recorded as a similarity set, and similarity analysis is continued for feature areas that are not recorded in the similarity set until all feature areas are recorded in the similarity set; the smallest rectangle that can contain each feature area in a single feature set is recorded as a set area, and each feature set corresponds to a set area; For any two feature areas, similarity coefficient = length similarity + area similarity, length similarity = 1 / |length of the longer side of one feature area - length of the longer side of another feature area|, area similarity = 1 / |area of ​​one feature area - area of ​​another feature area|; The disorder threshold of the feature area = the maximum value of the disorder coefficients corresponding to each similar set + the number of similar sets. The method of confirming the disorder coefficient is that for a single similar set, the similar set is recorded as the target similar set. The disorder coefficient corresponding to the target similar set = the number of set areas that overlap with the set area corresponding to the target similar set + the area of ​​the set area corresponding to the target similar set that overlaps with other set areas / the area of ​​the set area corresponding to the target similar set; Determine an associated combination according to the area normalization degree and the distance reference value, wherein a combined analysis is performed for each feature area. When a combined analysis is performed for a single feature area, the feature area is recorded as a first target feature area, and other feature areas other than the first target feature area that are not recorded in the associated combination are recorded as first reference feature areas, and a set of the first reference feature area and the first target feature area whose area normalization degree with the first target feature area is greater than a preset area normalization degree and whose distance reference value is less than a preset distance reference value is recorded as an associated combination, and the combined analysis is continued for the feature areas that are not recorded in the associated combination until all feature areas are recorded in the associated combination, and then the combined analysis is stopped; The confirmation method of regional normalization is: If the similarity coefficient is greater than or equal to the preset similarity coefficient, the region normalization degree is determined according to the similarity coefficient, wherein the region normalization degree is positively correlated with the similarity coefficient; If the similarity coefficient is less than the preset similarity coefficient, the regional normalization degree is determined according to the coverage influence threshold, wherein the regional normalization degree is positively correlated with the coverage influence threshold; The method for confirming the covering influence threshold is that, for any two feature areas, they are recorded as the first feature area and the second feature area respectively, and each feature area adjacent to the second feature area is recorded as the third feature area. The covering influence threshold is the number of third feature areas whose similarity coefficient with the first feature area is greater than or equal to the preset similarity coefficient; the distance reference value is the shortest distance from one feature area to another feature area; The values ​​of the preset feature line interaction coefficient, the preset feature area disorder threshold, the preset similarity coefficient, the preset area normalization degree and the preset distance reference value can be determined by the user according to the actual application scenario. The greater the user's demand for determining the associated area according to the associated combination, the smaller the values ​​of the preset feature line interaction coefficient and the preset feature area disorder threshold are. A value of the preset feature line interaction coefficient and the preset feature area disorder threshold is provided. The historical records of determining the optimization method according to the feature area category are detected, and the average value of the feature area disorder threshold corresponding to the historical records that can meet the user's needs is recorded as the preset feature area disorder threshold. The preset feature line interaction coefficient is 2mm; the greater the user's demand for improving the accuracy of the area normalization degree determination, the value of a preset similarity coefficient is increased, and the preset similarity coefficient is 0.3; the greater the user's demand for the degree of information relevance in the associated area, the larger the value of the preset area normalization degree, and the smaller the value of the preset distance reference value is. A value of the preset area normalization degree and the preset distance reference value is provided. The historical records of determining the associated combination according to the area normalization degree and the distance reference value are detected, and the average value of the area normalization degree corresponding to the historical records that can meet the user's needs is recorded as the preset area normalization degree. The preset distance reference value is 2mm.

[0031] Specifically, if the characteristic line interaction coefficient is less than the preset characteristic line interaction coefficient and the characteristic area disorder threshold is less than the preset characteristic area disorder threshold, the association analysis method is to determine the optimization method according to the characteristic area category, where: For the longitudinal feature area, the optimization method is to determine the associated area based on the distance reference value and the similarity coefficient; For the horizontal feature area, the optimization method is to determine the associated area based on the feature line; When determining the associated region according to the feature line, the region to be analyzed is determined according to the matching reference value. For a single region to be analyzed, if the density of the feature region is less than the preset density of the feature region, the region to be analyzed is segmented by the cutting line to obtain the associated region; If the density of the feature area is greater than or equal to the preset density of the feature area, the area to be analyzed is taken as the associated area.

[0032] Among them, for the longitudinal feature area, the optimization method is to determine the associated area according to the distance reference value and the similarity coefficient, wherein a longitudinal analysis is performed on each longitudinal feature area, and when a longitudinal analysis is performed on a single longitudinal feature area, the longitudinal feature area is recorded as a target longitudinal feature area, and other longitudinal feature areas except the target longitudinal feature area are recorded as reference longitudinal feature areas, and a set of reference longitudinal feature areas whose distance reference values ​​from the target longitudinal feature area are less than a preset distance reference value and whose similarity coefficients are greater than a preset similarity coefficient and the target longitudinal feature area are recorded as a longitudinal combination, and the longitudinal analysis is continued for the longitudinal feature areas that are not recorded in the longitudinal combination until all the longitudinal feature areas are recorded in the longitudinal combination, and the minimum rectangle that can contain each longitudinal feature area in a single longitudinal combination is recorded as an associated area; Determining the region to be analyzed according to the matching reference value, including: performing region matching for each feature line to be analyzed in order of distance thresholds from small to large, when performing region matching for a single feature line to be analyzed, recording the feature line to be analyzed as a target feature line to be analyzed, recording feature lines to be analyzed other than the target feature line to be analyzed as reference feature lines to be analyzed, recording a region in the grayscale image between the reference feature line to be analyzed having the largest matching reference value with the target feature line to be analyzed and the target feature line to be analyzed as a region to be analyzed, and continuing to perform matching analysis for each feature line to be analyzed until each feature line to be analyzed is recorded in the region to be analyzed; The feature lines to be analyzed are the feature lines that are not recorded in the area to be analyzed and the feature lines that are located at the bottom of the grayscale image and are recorded in the area to be analyzed; The distance threshold is the shortest distance from a single feature line to the top of the grayscale image; The matching reference value is confirmed in the following way: for any two feature lines, the lateral feature area not divided by the two feature lines is recorded as a complete area, the lateral feature area divided by the two feature lines is recorded as an incomplete area, and the grayscale image area between the two feature lines is recorded as a reference area. The matching reference value = spacing coefficient / (complete matching degree - number of splits). The complete matching degree is the average value of the sub-matching degrees corresponding to each complete area in the reference area. The sub-matching degree is confirmed in the following way: for a single complete area, the complete area is recorded as the target complete area, and the other complete areas in the reference area where the target complete area is located are recorded as reference complete areas. The sub-matching degree corresponding to the target complete area is the average value of the similarity coefficients between the target complete area and each reference complete area; the number of splits is the number of incomplete areas in the reference area. For a single area to be analyzed, the characteristic area density = the sum of the areas of each lateral characteristic area in the area to be analyzed / the area of ​​the smallest rectangle that can contain each lateral characteristic area in the area to be analyzed; The value of the preset feature area density can be determined by the user according to the actual application scenario. The higher the user's demand for the degree of information matching in a single associated area, the greater the value of the preset feature area density. A value of the preset feature area density is provided, and the historical records of segmenting the area to be analyzed by the cutting line are detected, and the average value of the feature area density corresponding to the historical records that can meet the user's needs is recorded as the preset feature area density; The method for confirming the cutting line is that, for a single area to be analyzed, the cutting line is a straight line in the sub-area with the smallest connectivity density that is parallel to the bisector and does not pass through the characteristic area of ​​the area to be analyzed. The same number of bisector points are set on the sides where the area to be analyzed coincides with the two characteristic lines, and the bisector points with the same order on both sides are connected in pairs to obtain several sub-areas. The number of bisector points is positively correlated with the density of the characteristic area. The bisector is a line connecting the bisector points with the same order on both sides. The method for confirming the connectivity density is that, for a single sub-area, the connectivity density = the total area of ​​the connected domain in the sub-area / the area of ​​the sub-area.

[0033] Specifically, the feature area categories include: a longitudinal feature area whose longitudinal slip coefficient is less than a preset longitudinal slip coefficient and whose feature influence threshold is less than a preset feature influence threshold, and a lateral feature area whose longitudinal slip coefficient is greater than or equal to the preset longitudinal slip coefficient or whose feature influence threshold is greater than or equal to the preset feature influence threshold.

[0034] Among them, the confirmation method of the longitudinal dislocation coefficient and the feature influence threshold is as follows: for a single feature area, the feature area is recorded as the second target feature area, the straight line passing through the center point of the second target feature area and parallel to the longer side of the grayscale image is recorded as the reference straight line, the feature area through which the reference straight line passes and whose similarity coefficient with the second target feature area is greater than the preset similarity coefficient is recorded as the radiation feature area, the feature influence threshold = the number of radiation feature areas + the influence area distribution coefficient, the longitudinal dislocation coefficient is the average value of the offset distance of the radiation lines corresponding to each radiation feature area; The influence zone distribution coefficient is the average value of the influence distances corresponding to each influence zone. Each radiation characteristic zone and the second target characteristic zone are recorded as influence zones. For a single influence zone, the influence zone is recorded as the target influence zone, and the minimum value of the shortest distances from other influence zones to the target influence zone is recorded as the influence distance corresponding to the target influence zone. The radiation line corresponding to a single radiation feature area is a straight line passing through the center point of the radiation feature area and parallel to the longer side of the grayscale image, and the center point is the center point of the circumscribed circle of the single feature area; the offset distance is the shortest distance from a single radiation line to the reference straight line; The values ​​of the preset longitudinal slip coefficient and the preset feature influence threshold can be determined by the user according to the actual application scenario. The larger the values ​​of the preset longitudinal slip coefficient and the preset feature influence threshold are, the greater the user's demand for determining the feature area as a longitudinal feature area. A value of the preset longitudinal slip coefficient and the preset feature influence threshold is provided, and the historical records of the user determining the feature area as a longitudinal feature area are detected. The average value of the longitudinal slip coefficient corresponding to the historical records that can meet the user's needs is recorded as the preset longitudinal slip coefficient, and the average value of the feature influence threshold corresponding to the historical records that can meet the user's needs is recorded as the preset feature influence threshold.

[0035] Specifically, the associated area categories include: a type of associated area whose regional influence coefficient is greater than or equal to the preset regional influence coefficient and whose regional fluctuation amplitude is greater than or equal to the preset regional fluctuation amplitude, and a type of associated area whose regional influence coefficient is less than the preset regional influence coefficient or whose regional fluctuation amplitude is less than the preset regional fluctuation amplitude.

[0036] The method for confirming the regional influence coefficient and the regional fluctuation amplitude is as follows: for a single associated region, the affected region is recorded as the target associated region, the regional influence coefficient is the number of connected domains in the target associated region with a rectangular shape, and the regional fluctuation amplitude is the standard deviation of the characteristic distances corresponding to each characteristic region in the associated region; the characteristic distance corresponding to a single characteristic region is the length of the shorter side in the characteristic region; The values ​​of the preset area influence coefficient and the preset area fluctuation amplitude can be determined by the user according to the actual application scenario. The higher the accuracy of the user's judgment of the barcode area, the larger the values ​​of the preset area influence coefficient and the preset area fluctuation amplitude. A value of the preset area influence coefficient and the preset area fluctuation amplitude is provided, and the historical records of determining the associated area as a type of associated area are detected. The average value of the area influence coefficient corresponding to the historical records that can meet the user's needs is recorded as the preset area influence coefficient, and the average value of the area fluctuation amplitude corresponding to the historical records that can meet the user's needs is recorded as the preset area fluctuation amplitude.

[0037] Specifically, if the blur state of the two types of related regions is that the smoothing coefficient is less than the preset smoothing coefficient or the cross-influence coefficient is less than the preset cross-influence coefficient, the region processing method is to determine the pooling method according to the pixel fluctuation value; If the pixel fluctuation value is greater than or equal to the preset pixel fluctuation value, the pooling method is to reduce the pooling moving step size of the pooling layer according to the hierarchical influence coefficient. If the pixel fluctuation value is less than the preset pixel fluctuation value, the pooling method is to reduce the pooling window area according to the comprehensive evaluation value; The pooling layer is determined according to the pixel threshold and the level threshold, and the reduction value of the pooling moving step corresponding to a single pooling layer is positively correlated with the level influence coefficient corresponding to the pooling layer; The reduction value of the pooling window area is positively correlated with the comprehensive evaluation value.

[0038] Among them, the reduction value of the pooling moving step corresponding to a single pooling layer = the level influence coefficient corresponding to the pooling layer × α1, the reduction value of the pooling window area = the comprehensive evaluation value × α2, and a value of α1 and α2 is provided, α1 is 0.05, and α2 is 0.02; The confirmation method of the smoothing coefficient and the cross-influence coefficient is as follows: for a single two-category association area, each pixel point with a pixel value greater than 0 in the two-category connected domain is recorded as an analysis pixel point, the smoothing coefficient = 1 / (the maximum value of the pixel value corresponding to each analysis pixel point - the minimum value of the pixel value corresponding to each analysis pixel point), the cross-influence coefficient is the maximum value of the sub-cross coefficients corresponding to each feature area in the single two-category association area, the sub-cross coefficient corresponding to a single feature area is the average value of the distribution thresholds corresponding to each intersection point in the feature area, for a single intersection point in a single feature area, the intersection point is recorded as the target intersection point, and the average value of the shortest distance from the target intersection point to other intersection points in the feature area is recorded as the distribution threshold; for a single point in a connected domain, if the number of line segments passing through the point is greater than or equal to 2, the point is an intersection point, and a single connected domain contains several line segments; The values ​​of the preset smoothing coefficient and the preset cross-influence coefficient can be determined by the user according to the actual application scenario. The larger the values ​​of the preset smoothing coefficient and the preset cross-influence coefficient are, the greater the user's demand for determining the pooling method according to the pixel fluctuation value is. A value of the preset smoothing coefficient and the preset cross-influence coefficient is provided, and the historical records of determining the pooling method according to the pixel fluctuation value are detected, and the average value of the smoothing coefficient corresponding to the historical records that can meet the user's needs is recorded as the preset smoothing coefficient, and the average value of the cross-influence coefficient corresponding to the historical records that can meet the user's needs is recorded as the preset cross-influence coefficient; The pixel fluctuation value is the standard deviation of the pixel values ​​corresponding to each analyzed pixel point in a single second-class association area. Layer influence coefficient = standard deviation of pixel values ​​corresponding to each analysis pixel in a single pooling layer + number of analysis pixels in a single layer; comprehensive evaluation value = smoothing coefficient + cross influence coefficient; The pooling moving step is the distance that the pooling window moves in the two-category associated area in a single time, and the pooling window area is the area of ​​the pooling window. The preliminary design of the pooling window area preliminarily set in the present invention is 3×3, and the pooling moving step is 2. After the pooling method is confirmed, the pooling process is performed on the two-category associated area, and the pooling window is slid according to the pooling moving step. After each sliding, the maximum pixel value in the pooling window is used as the representative value. It should be noted that when the pooling method is to reduce the pooling moving step of the pooling layer according to the hierarchical influence coefficient, the two pooling layers need to be merged after the pooling process is completed. The pixel value of each pixel in the merged grayscale image is in [0,250]. The pooling process can effectively smooth the noise and mutation in the image, and the OCR technology is used to perform text recognition on the two-category associated area after the pooling process. This is content that is easy for technicians in this field to understand and will not be described in detail.

[0039] The pooling layer is determined according to the pixel threshold and the level threshold, wherein the pixel frequency corresponding to each pixel value is detected respectively, the pixel value is 0, 1, 2, ..., 250, for a single pixel value, the pixel value is recorded as the target pixel value, the pixel frequency corresponding to the target pixel value is the number of pixels whose pixel value is the target pixel value in a single two-category associated area, the pixel value whose difference coefficient is greater than the preset difference coefficient is recorded as the pixel threshold, the pixel threshold with the largest level threshold is used as the segmentation reference to segment the grayscale image into two pooling layers, a single pooling layer only contains each pixel in the two-category associated area whose pixel value is less than or equal to the pixel threshold or only contains each pixel in the two-category associated area whose pixel value is greater than the pixel threshold; it should be noted that the moving step size of each pooling layer is adjusted and then the two pooling layers are merged, and the pixel value of the pixel point in the merged image For a single pixel value, record the pixel value as the target pixel value, the difference coefficient corresponding to the target pixel value = the average value of the frequency difference values ​​corresponding to the pixel values ​​adjacent to the target pixel value, the frequency difference value corresponding to the single pixel value adjacent to the target pixel value = |the pixel point frequency corresponding to the single pixel value adjacent to the target pixel value - the pixel point frequency corresponding to the target pixel value|; The values ​​of the preset pixel fluctuation value and the preset difference coefficient can be determined by the user according to the actual application scenario. The larger the value of the preset pixel fluctuation value is, the greater the user's demand for reducing the pooling window area according to the comprehensive evaluation value. A value of the preset pixel fluctuation value is provided, and the historical records of reducing the pooling window area according to the comprehensive evaluation value are detected, and the average value of the pixel fluctuation values ​​corresponding to the historical records that can meet the user's needs is recorded as the preset pixel fluctuation value. The historical records of determining the pooling layer according to the pixel threshold and the hierarchical threshold are detected, and the average value of the difference coefficients corresponding to the historical records that can meet the user's needs is recorded as the preset difference coefficient.

[0040] Specifically, if the fuzzy state of the second type of associated region is that the smoothness coefficient is greater than or equal to the preset smoothness coefficient and the cross-influence coefficient is greater than or equal to the preset cross-influence coefficient, the region processing method is to determine the cutting method of the pre-cut feature region according to the gradient change value; For a single pre-cut feature area, If the gradient change value is greater than or equal to the preset gradient change value, the cutting method is to cut according to the gradient line; If the gradient change value is less than the preset gradient change value, the cutting method is to cut according to the pixel similarity; The pre-cut feature area is a feature area in the second type of associated area where the evaluation threshold is greater than a preset evaluation threshold.

[0041] Among them, the gradient change value is the maximum value of the sub-gradient change values ​​corresponding to each feature area in a single two-class association area. The sub-gradient change value is confirmed by setting the same number of equally divided points on the two longer sides of the feature area for a single feature area, and connecting the equally divided points with the same order in the two sides to obtain several sub-feature areas. The number of equally divided points is positively correlated with the number of intersection points in the feature area. The method for confirming the gradient change value is as follows: for a single sub-feature region, the sub-feature region is recorded as a target sub-feature region, and the maximum value of the intersection difference corresponding to the sub-feature region adjacent to the target sub-feature region is recorded as the gradient change value corresponding to the target sub-feature region, and the intersection difference corresponding to the single sub-feature region adjacent to the target sub-feature region = |the number of intersections in the single sub-feature region adjacent to the target sub-feature region - the number of intersections in the target sub-feature region|; The cutting method is cutting according to the gradient line, including: segmenting according to each gradient line, when segmenting according to a single gradient line, the feature area can be divided into two areas through the gradient line, and the area with the largest area in the two areas is recorded as a segmentation area, and each gradient line corresponds to a segmentation area; for a single pre-cut feature area, each sub-feature area with a gradient change value greater than a preset gradient change value is recorded as a change area, and a dividing line that passes through 2 change areas is recorded as a gradient line; The cutting method is cutting according to pixel similarity, including: performing cutting analysis on each analysis pixel in a single pre-cut feature area, when performing segmentation analysis on a single analysis pixel, recording the analysis similarity as the target analysis pixel, recording other analysis pixels except the target analysis pixel as the reference analysis pixel, recording a set of reference analysis pixels whose pixel similarity with the target analysis pixel is greater than the preset pixel similarity as a cutting combination, and continuing to perform cutting analysis on analysis pixels not recorded in the cutting combination until all analysis pixels are recorded in the cutting combination; recording the minimum rectangle that can contain a single cutting combination as a cutting area, and each cutting combination corresponds to a cutting area; pixel similarity=1-(the absolute value of the difference between the pixel values ​​corresponding to the two analysis pixels / the larger value of the pixel values ​​corresponding to the two analysis pixels); For each cut area in the two-type associated area and the feature area whose evaluation threshold is less than the preset evaluation threshold, text recognition is performed by OCR technology to obtain the text corresponding to each of the two-type associated areas. This is easy to understand for those skilled in the art and will not be described in detail. Evaluation threshold = standard deviation of pixel values ​​corresponding to each analyzed pixel point in a single feature area + gradient change value corresponding to a single feature area; The user can determine the values ​​of the preset gradient change value, the preset evaluation threshold and the preset pixel similarity according to the actual application scenario. The larger the value of the preset gradient change value, the greater the user's demand for cutting according to pixel similarity. A value of the preset gradient change value is provided, and the historical records of cutting according to the gradient line are detected. The average value of the gradient change values ​​corresponding to the historical records that can meet the user's needs is recorded as the preset gradient change value. The greater the user's demand for improving the accuracy of text recognition, the smaller the value of the preset evaluation threshold, and the larger the value of the preset pixel similarity. A value of the preset evaluation threshold and the preset pixel similarity is provided, and the preset pixel similarity is 70%. The historical records of text recognition for each uncut feature area are detected, and the average value corresponding to the historical records that can meet the user's needs is recorded as the preset evaluation threshold.

[0042] Specifically, whether the waybill information is qualified is determined according to the matching coefficient between the waybill information corresponding to the first type of associated area and the text information corresponding to the second type of associated area and the sensitivity coefficient of the text information corresponding to the second type of associated area, wherein: If the matching coefficient is less than the preset matching coefficient or the sensitivity coefficient is greater than or equal to the preset sensitivity coefficient, the waybill information is unqualified; If the matching coefficient is greater than or equal to the preset matching coefficient and the sensitivity coefficient is less than the preset sensitivity coefficient, the waybill information is qualified.

[0043] Wherein, the matching coefficient = the number of keywords that are the same in the text information and the waybill information / (the larger value of a1 and a2), the text corresponding to all the two types of associated areas is recorded as text information, the total amount of keywords in the text information is recorded as a1, and the number of keywords in the waybill information is recorded as a2, the sensitivity coefficient is the number of sensitive characters in the text information / the total amount of sensitive characters in the waybill information, and the sensitive characters are the characters corresponding to the recipient's name and the recipient's phone number in the waybill information; the waybill information corresponding to the first type of associated area is identified by barcode recognition technology to obtain the waybill information, and the waybill information includes but is not limited to the recipient's name, recipient's address, recipient's phone number, sender's address and sender's phone number of the express corresponding to the waybill photo, which is easy for a person skilled in the art to understand and will not be described in detail; The values ​​of the preset matching coefficient and the preset sensitivity coefficient can be determined by the user according to the actual application scenario. The greater the accuracy of the user's review of the waybill photo, the greater the value of the preset matching coefficient and the smaller the value of the preset sensitivity coefficient. The values ​​of the preset matching coefficient and the preset sensitivity coefficient are provided, and the matching coefficient is 80% and the sensitivity coefficient is 60%.

[0044] See also Figure 4 As shown, it is a module connection diagram of a device for reviewing waybill photos based on multimodal analysis of the present invention. The present invention also provides a device for reviewing waybill photos based on multimodal analysis, including: Waybill collection module, used to obtain waybill photos; A feature area recognition module, which is connected to the waybill collection module, is used to determine a feature area determination method according to an encirclement coefficient and an influence threshold, wherein the feature area determination method is to determine the feature area according to a connected domain or a correlation coefficient; An association analysis module, which is connected to the waybill collection module and the feature area identification module respectively, and is used to determine an association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold. The association analysis method is to determine an optimization method according to the characteristic area category or determine the association area according to the association combination, and determine the association area category according to the regional influence coefficient and the regional fluctuation amplitude; A region processing module, which is connected to the association analysis module, is used to determine the fuzzy state of each of the two types of association regions based on the smoothing coefficient and the cross-influence coefficient, and determine the region processing method according to the fuzzy state. The region processing method is to determine the pooling method according to the pixel fluctuation value or to determine the cutting method of the pre-cut feature area according to the gradient change value; A waybill review module is connected to the association analysis module and the area processing module, and is used to determine whether the waybill information is qualified based on the matching coefficient between the waybill information corresponding to the first type of association area and the text information corresponding to the second type of association area and the sensitivity coefficient of the text information corresponding to the second type of association area.

[0045] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for reviewing waybill photos based on multimodal analysis, characterized in that: include: Obtain the waybill photo and determine the feature area determination method based on the encirclement coefficient and the influence threshold. The feature area determination method is to determine the feature area based on the connected domain or the correlation coefficient; Determine the association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold, and the association analysis method is to determine the optimization method according to the characteristic area category or determine the association area according to the association combination; The characteristic area category is determined according to the longitudinal displacement coefficient and the characteristic influence threshold, and the optimization method is to determine the associated area according to the distance reference value and the similarity coefficient, or to determine the associated area according to the characteristic line; Determine the category of associated regions according to the regional influence coefficient and the regional fluctuation amplitude; The fuzzy state of each of the two types of associated regions is determined based on the smoothing coefficient and the cross-influence coefficient, and the region processing method is determined according to the fuzzy state. The region processing method is to determine the pooling method according to the pixel fluctuation value or to determine the cutting method of the pre-cut feature area according to the gradient change value; The pooling method is to adjust the pooling moving step of the pooling layer according to the hierarchical influence coefficient or to adjust the pooling window according to the comprehensive evaluation value, and the cutting method is to cut according to the gradient line or pixel similarity; Whether the waybill information is qualified is determined according to the matching coefficient between the waybill information corresponding to the first-class associated area and the text information corresponding to the second-class associated area and the sensitivity coefficient of the text information corresponding to the second-class associated area.

2. The method for reviewing waybill photos based on multimodal analysis according to claim 1, characterized in that: If the enclosing coefficient is less than the preset enclosing coefficient and the influence threshold is less than the preset influence threshold, the feature area determination method is to determine the feature area according to the connected domain; If the enclosing coefficient is greater than or equal to the preset enclosing coefficient or the influence threshold is greater than or equal to the preset influence threshold, the feature area determination method is to determine the feature area according to the correlation coefficient.

3. The method for reviewing waybill photos based on multimodal analysis according to claim 2, characterized in that: If the characteristic line interaction coefficient is greater than or equal to the preset characteristic line interaction coefficient or the characteristic area disorder threshold is greater than or equal to the preset characteristic area disorder threshold, the association analysis method is to determine the association area according to the association combination, where: The smallest rectangle that can contain the feature area corresponding to a single association combination is recorded as an association area; The association combination is determined according to the area normalization degree and the distance reference value.

4. The method for reviewing waybill photos based on multimodal analysis according to claim 3 is characterized in that: If the characteristic line interaction coefficient is less than the preset characteristic line interaction coefficient and the characteristic area disorder threshold is less than the preset characteristic area disorder threshold, the association analysis method is to determine the optimization method according to the characteristic area category, where: For the longitudinal feature area, the optimization method is to determine the associated area based on the distance reference value and the similarity coefficient; For the horizontal feature area, the optimization method is to determine the associated area based on the feature line; When determining the associated region according to the feature line, the region to be analyzed is determined according to the matching reference value. For a single region to be analyzed, if the density of the feature region is less than the preset density of the feature region, the region to be analyzed is segmented by the cutting line to obtain the associated region; If the density of the feature area is greater than or equal to the preset density of the feature area, the area to be analyzed is taken as the associated area.

5. The method for reviewing waybill photos based on multimodal analysis according to claim 4, characterized in that: The characteristic area categories include: a longitudinal characteristic area whose longitudinal slip coefficient is less than a preset longitudinal slip coefficient and whose characteristic influence threshold is less than a preset characteristic influence threshold, and a lateral characteristic area whose longitudinal slip coefficient is greater than or equal to the preset longitudinal slip coefficient or whose characteristic influence threshold is greater than or equal to the preset characteristic influence threshold.

6. The method for reviewing waybill photos based on multimodal analysis according to claim 5, characterized in that: The associated area categories include: a type of associated area whose regional influence coefficient is greater than or equal to the preset regional influence coefficient and whose regional fluctuation amplitude is greater than or equal to the preset regional fluctuation amplitude, and a type of associated area whose regional influence coefficient is less than the preset regional influence coefficient or whose regional fluctuation amplitude is less than the preset regional fluctuation amplitude.

7. The method for reviewing waybill photos based on multimodal analysis according to claim 6, characterized in that: If the blur state of the second type of associated area is that the smoothing coefficient is less than the preset smoothing coefficient or the cross-influence coefficient is less than the preset cross-influence coefficient, the area processing method is to determine the pooling method according to the pixel fluctuation value; If the pixel fluctuation value is greater than or equal to the preset pixel fluctuation value, the pooling method is to reduce the pooling moving step size of the pooling layer according to the hierarchical influence coefficient. If the pixel fluctuation value is less than the preset pixel fluctuation value, the pooling method is to reduce the pooling window area according to the comprehensive evaluation value; The pooling layer is determined according to the pixel threshold and the level threshold, and the reduction value of the pooling moving step corresponding to a single pooling layer is positively correlated with the level influence coefficient corresponding to the pooling layer; The reduction value of the pooling window area is positively correlated with the comprehensive evaluation value.

8. The method for reviewing waybill photos based on multimodal analysis according to claim 7, characterized in that: If the fuzzy state of the second type of associated area is that the smoothness coefficient is greater than or equal to the preset smoothness coefficient and the cross-influence coefficient is greater than or equal to the preset cross-influence coefficient, the area processing method is to determine the cutting method of the pre-cut feature area according to the gradient change value; For a single pre-cut feature area, If the gradient change value is greater than or equal to the preset gradient change value, the cutting method is to cut according to the gradient line; If the gradient change value is less than the preset gradient change value, the cutting method is to cut according to the pixel similarity; The pre-cut feature area is a feature area in the second type of associated area where the evaluation threshold is greater than a preset evaluation threshold.

9. The method for reviewing waybill photos based on multimodal analysis according to claim 8, characterized in that: Whether the waybill information is qualified is determined according to the matching coefficient between the waybill information corresponding to the first-class associated area and the text information corresponding to the second-class associated area and the sensitivity coefficient of the text information corresponding to the second-class associated area, wherein: If the matching coefficient is less than the preset matching coefficient or the sensitivity coefficient is greater than or equal to the preset sensitivity coefficient, the waybill information is unqualified; If the matching coefficient is greater than or equal to the preset matching coefficient and the sensitivity coefficient is less than the preset sensitivity coefficient, the waybill information is qualified.

10. A device for verifying waybill photos using the method for verifying waybill photos based on multimodal analysis as described in any one of claims 1 to 9, characterized in that: include: Waybill collection module, used to obtain waybill photos; A feature area recognition module, which is connected to the waybill collection module, is used to determine a feature area determination method according to an encirclement coefficient and an influence threshold, wherein the feature area determination method is to determine the feature area according to a connected domain or a correlation coefficient; An association analysis module, which is connected to the waybill collection module and the feature area identification module respectively, and is used to determine an association analysis method according to the characteristic line interaction coefficient and the characteristic area disorder threshold. The association analysis method is to determine an optimization method according to the characteristic area category or determine the association area according to the association combination, and determine the association area category according to the regional influence coefficient and the regional fluctuation amplitude; A region processing module, which is connected to the association analysis module, is used to determine the fuzzy state of each of the two types of association regions based on the smoothing coefficient and the cross-influence coefficient, and determine the region processing method according to the fuzzy state. The region processing method is to determine the pooling method according to the pixel fluctuation value or to determine the cutting method of the pre-cut feature area according to the gradient change value; A waybill review module is connected to the association analysis module and the area processing module, and is used to determine whether the waybill information is qualified based on the matching coefficient between the waybill information corresponding to the first type of association area and the text information corresponding to the second type of association area and the sensitivity coefficient of the text information corresponding to the second type of association area.

Images