A comic strip intelligent classification system and method combining image and text analysis

Abstract

The application relates to the technical field of image processing, and particularly discloses a combination of an image and a text analysis-based comic strip intelligent classification system and method, which acquires comic strip page image data through scanning and extracts boundary contour data and text content data of each sub-shot area; equal-arc-length sampling is performed on the boundary contour to generate a contour point set, and a reading track vector is generated according to the spatial distribution of the contour point set; a spatial distribution description with the contour point as a reference origin is constructed based on the reading track vector, a shape feature code with translation and rotation invariance is obtained, a closed intensity value and an extension amplitude value are parsed from the shape feature code, reading weights are calculated according to the closed intensity value and the extension amplitude value, geometric shape correction is performed on the spatial position, a corrected position coding feature is obtained, a narrative association path is constructed according to the corrected position coding feature and the appearance order of key referring words in the text content, each sub-shot area is arranged into a logical time sequence, the image and text features are associated, and a classification result is output.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an intelligent classification system and method for comic books that combines image and text analysis. Background Technology

[0002] With the advancement of digital library construction and cultural heritage protection, a large number of printed comic books are being scanned into digital image formats for storage and archiving. As a narrative medium combining text and images, comic books consist of pages composed of multiple panels arranged in a specific layout, each panel containing an image and corresponding dialogue or narration. In practical collection management and digitization applications, comic books need to be categorized and organized according to themes, styles, or periods of creation to facilitate retrieval and research.

[0003] In existing intelligent classification methods for comic books, the reading order reconstruction method based on spatial coordinates cannot reasonably determine the reading weight of the panel area according to the geometric shape of the panel when faced with borderless white space composition and irregularly shaped panels such as circles and fans. It is easy to mistakenly sort the flashback panels with a large area in the center of the page before the current narrative panels at the edge of the page. At the same time, the image-text association relies solely on the page coordinates forcibly aligned and lacks the means to extract narrative clues from the text content to correct the time sequence. This leads to the disruption of the narrative timeline between reality and memory, and the misalignment of image and text features. Ultimately, the classification result deviates from the actual narrative logic of the comic book due to the special page composition. Summary of the Invention

[0004] The purpose of this invention is to provide a smart classification system and method for comic books that combines image and text analysis, so as to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions: A method for intelligent classification of comic books that combines image and text analysis includes the following steps: S1: Collect image data of the comic strip page, and extract the boundary contour data of each panel area and the text content data associated with each panel area from the page image data; S2: Based on the boundary contour data, sample contour points of equal arc length for each segmentation area to obtain the contour point set corresponding to each segmentation area, and generate a reading trajectory vector to represent the geometric shape of the corresponding segmentation area based on the spatial distribution of contour points in each contour point set. S3: Based on the reading trajectory vector, construct a spatial distribution description for each segmentation area with the outline point as the reference origin, and obtain the morphological feature code corresponding to each segmentation area. The morphological feature code is used to characterize the geometric topological properties of the corresponding segmentation area under translation and rotation. S4: Based on the morphological feature encoding of each segmentation area, calculate the reading weight of each segmentation area on the page, and re-encode the spatial position of each segmentation area according to the reading weight to obtain the position encoding feature after geometric morphology correction. S5: Based on the positional encoding features of each scene area and its associated text content data, construct the narrative connection path between each scene area, arrange each scene area into a logical temporal sequence according to the narrative connection path, and associate the visual features and text features of each scene area according to the logical temporal sequence to output the classification result.

[0006] As a further aspect of the present invention: the process of generating the reading trajectory vector is as follows: After obtaining the contour point set of each storyboard area, determine the relative orientation data of each contour point relative to the reference point inside the storyboard area. Based on the relative orientation data, all contour points are arranged in order of direction surrounding the internal reference point to obtain a contour point sequence. Based on the directional change between adjacent contour points in the contour point sequence, extract the geometric morphological feature values ​​of the contour point set. A reading trajectory vector is generated based on geometric morphological feature values. The reading trajectory vector is used to characterize the closed shape and spatial extension of the storyboard area outline.

[0007] As a further aspect of the present invention: the extraction of geometric morphological feature values ​​of the contour point set specifically includes: Obtain the two direction vectors formed by every three consecutive contour points in the contour point sequence, determine the direction change between each direction vector, and obtain the direction change sequence; Based on the positive and negative signs of the directional change quantities in the sequence, the sequence of directional change quantities is divided into multiple continuous segments of change in the same direction. Determine the number of directional changes contained in each continuous unidirectional change segment, sort the continuous unidirectional change segments in descending order of the number, and select the continuous unidirectional change segments with the highest number of changes. Based on the directional change values ​​of each selected continuous unidirectional change segment, geometric morphological feature values ​​are generated to characterize the curvature of the boundary of the storyboard area.

[0008] As a further aspect of the present invention: the generation process of the morphological feature code is as follows: Obtain the sequence of contour points corresponding to the reading trajectory vector, and for each current contour point in the sequence, determine the relative distance and relative orientation of the remaining contour points relative to the current contour point. Based on all the relative distances and relative orientations corresponding to each current contour point, the remaining contour points are assigned to the corresponding spatial distribution areas according to the hierarchical assignment of relative distances and the interval assignment of relative orientations. The number of contour points in each spatial distribution area is counted, and a spatial distribution sub-code for each current contour point is generated based on the relative ratio of the number of contour points in each spatial distribution area. The spatial distribution sub-encodes of all current contour points in the contour point sequence are combined according to the arrangement order of the contour point sequence to generate the morphological feature code of the storyboard region.

[0009] As a further aspect of the present invention: the generation of spatial distribution sub-encoding for each current contour point specifically includes: Obtain the specific values ​​of the number of contour points in each spatial distribution area, and determine the maximum and minimum values ​​from the specific values ​​of all spatial distribution areas. Based on the relative proportion between the number of contour points and the maximum value in each spatial distribution area, the hierarchical classification of each spatial distribution area is determined, and the hierarchical label corresponding to each spatial distribution area is obtained. The hierarchical labels corresponding to each spatial distribution region are combined according to the division order of each spatial distribution region to generate the spatial distribution sub-code of the current contour point. The spatial distribution sub-code is used to characterize the dense distribution features of contour points around the current contour point in various spatial directions.

[0010] As a further aspect of the present invention: obtaining the geometrically corrected position coding features specifically includes: Obtain the hierarchical labels of the number of contour points in each spatial distribution area in the morphological feature encoding, and parse out the closure strength value, which represents the degree of closure of the storyboard area, and the extension amplitude value, which represents the extension range of the boundary of the storyboard area, from the hierarchical labels. The first weighting coefficient is determined based on the closure strength value, and the second weighting coefficient is determined based on the extension amplitude value. The product of the first weighting coefficient and the second weighting coefficient is used as the reading weight of the storyboard area. Based on the reading weight of the storyboard area, the horizontal and vertical coordinate values ​​of the original spatial coordinates of the storyboard area are scaled by the same proportion to obtain the corrected horizontal and vertical position data of the storyboard area. The horizontal and vertical position data after correction of the storyboard area are combined into a geometrically corrected position coding feature.

[0011] As a further aspect of the present invention: the analysis process of the extension amplitude value is as follows: Obtain the order and repetition frequency of each level label in the hierarchical label, and determine the highest and lowest frequency levels based on the repetition frequency of each level label; Based on the continuous distribution span of the spatial distribution area corresponding to the highest frequency level in the circumferential direction around the current contour point, the closure intensity value of the segmentation area where the current contour point is located is determined. The larger the continuous distribution span, the higher the closure intensity value. Based on the scattering span of the spatial distribution area corresponding to the lowest frequency level in the circumferential direction around the current contour point, determine the extension amplitude value of the segment area where the current contour point is located. The larger the scattering span, the higher the extension amplitude value. The closure strength value and extension amplitude value are combined to form the geometric shape analysis result of the storyboard area.

[0012] As a further aspect of the present invention: the construction of narrative connection paths between various scene regions specifically includes: Obtain the location encoding features of each storyboard area and the associated text content data, and extract key pronouns that refer to the same object from the text content data. Key pronouns include object name and object title. Based on the relative arrangement of the positional coding features of each scene area on the page, and combined with the synonymy of key pronouns in the text content data of each scene area, the initial association path between each scene area is determined. Based on the order in which key pronouns appear on the page in the text content data of each scene area, the temporal weight of each path node in the initial association path is determined. The path node with the highest temporal weight is taken as the starting point of the path, and the remaining path nodes are arranged in sequence along the initial association path to generate the narrative association path.

[0013] As a further aspect of the present invention: the process of generating the initial association path is as follows: Obtain the positional coding features of each segmentation area, determine the relative orientation relationship between the center points of each segmentation area based on the horizontal and vertical positional data in each positional coding feature, and calculate the proximity degree between each segmentation area based on the relative orientation relationship. Obtain the referents of key pronouns in the text content data of each storyboard area, count the number of times the same referents are shared between any two storyboard areas, and determine the strength of the referential association between each storyboard area based on the number of referent co-occurrences. The proximity degree between each segmentation area is multiplied and summed with the referential association strength to obtain the association weight value between each segmentation area. Based on the association weight value, the storyboard area with the highest association weight value is paired as the initial path backbone, and the remaining storyboard areas are connected to the initial path backbone in descending order of their association weight value with the initial path backbone to generate the initial association path.

[0014] A comic book intelligent classification system combining image and text analysis includes: The data acquisition module collects image data of the comic strip pages and extracts the boundary contour data of each panel area and the text content data associated with each panel area from the page image data. The contour sampling and trajectory generation module samples contour points of equal arc length for each segmentation area based on the boundary contour data, obtains the contour point set corresponding to each segmentation area, and generates a reading trajectory vector to represent the geometric shape of the corresponding segmentation area based on the spatial distribution of the contour points in each contour point set. The morphological feature encoding module constructs a spatial distribution description for each segmentation area based on the reading trajectory vector, with the outline point as the reference origin, and obtains the morphological feature encoding corresponding to each segmentation area. The morphological feature encoding is used to characterize the geometric topological properties of the corresponding segmentation area under translation and rotation. The position weight correction module calculates the reading weight of each segment area on the page based on the morphological feature encoding of each segment area, and re-encodes the spatial position of each segment area according to the reading weight to obtain the position encoding features after geometric morphological correction. The narrative association and classification module constructs narrative association paths between each scene area based on the location encoding features of each scene area and its associated text content data. Based on the narrative association paths, each scene area is arranged into a logical temporal sequence. Based on the logical temporal sequence, the visual features and text features of each scene area are correlated, and the classification results are output.

[0015] The beneficial effects of this invention are: (1) By extracting the boundary contour data of the storyboard area and generating morphological feature codes with translation and rotation invariance, it is possible to accurately distinguish storyboard areas with different geometric shapes, especially for borderless white space compositions and irregularly shaped storyboards such as circles and fans. This avoids the problem that large flashback frames are misjudged as the primary reading object because they occupy the central position of the page. This allows the geometric features of the storyboard area to be reasonably utilized in subsequent classification processing, thereby improving the accuracy of classification of the work on the page.

[0016] (2) By extracting key pronouns from text content data and constructing narrative connection paths in combination with position encoding features, the logical temporal sequence of the panel area can be determined according to the order in which key pronouns appear on the page. This makes the text-image connection no longer rely solely on the spatial coordinates of the page for forced alignment. Instead, it combines the narrative clues at the text level with the position information after geometric correction, effectively solving the problem of disordered narrative flow reconstruction caused by the special page composition. This makes the final output classification result more in line with the actual reading logic of comic strips. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the process of generating the reading trajectory vector in this invention; Figure 3 This is a system block diagram of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, this invention is an intelligent classification method for comic books that combines image and text analysis, including the following steps: S1: Collect image data of the comic strip page, and extract the boundary contour data of each panel area and the text content data associated with each panel area from the page image data; S2: Based on the boundary contour data, sample contour points of equal arc length for each segmentation area to obtain the contour point set corresponding to each segmentation area, and generate a reading trajectory vector to represent the geometric shape of the corresponding segmentation area based on the spatial distribution of contour points in each contour point set. S3: Based on the reading trajectory vector, construct a spatial distribution description for each segmentation area with the outline point as the reference origin, and obtain the morphological feature code corresponding to each segmentation area. The morphological feature code is used to characterize the geometric topological properties of the corresponding segmentation area under translation and rotation. S4: Based on the morphological feature encoding of each segmentation area, calculate the reading weight of each segmentation area on the page, and re-encode the spatial position of each segmentation area according to the reading weight to obtain the position encoding feature after geometric morphology correction. S5: Based on the positional encoding features of each scene area and its associated text content data, construct the narrative connection path between each scene area, arrange each scene area into a logical temporal sequence according to the narrative connection path, and associate the visual features and text features of each scene area according to the logical temporal sequence to output the classification result.

[0021] In S1, image data of the comic book page is collected, and boundary contour data of each panel area and associated text content data of each panel area are extracted from the page image data, specifically including: The physical comic book to be processed is placed one by one into the worktable of the flatbed scanner in page order. The optical resolution of the flatbed scanner is set to no less than 300 pixels per inch, and the color mode is set to 24-bit true color. During the scanning process, the physical position of each page is kept parallel to the edge of the worktable. The original image data of each page is generated and stored in local storage media in the order of page number.

[0022] For each original page image data, tilt correction and background normalization are performed sequentially: the tilt angle is determined by detecting the positioning marks at the four corners of the original page image data, and the original page image data is rotated and corrected according to the tilt angle to obtain the corrected page image data; the brightness distribution of all pixels in the corrected page image data is statistically analyzed, and the brightness value with the highest frequency in the brightness distribution is used as the background brightness threshold. Pixels in the corrected page image data whose brightness value is higher than the background brightness threshold are uniformly reset to the background brightness value to obtain the normalized page image data.

[0023] The normalized page image data is used to perform scene segmentation region detection: the normalized page image data is divided into multiple non-overlapping pixel sub-blocks, the brightness variance value of the pixels inside each pixel sub-block is calculated, and pixel sub-blocks with brightness variance values ​​greater than a preset variance threshold are marked as edge sub-blocks. Based on the distribution position of all edge sub-blocks in the normalized page image data, continuously distributed and adjacent edge sub-blocks are combined into at least one closed connected region. Each closed connected region is defined as a scene segmentation region, and the boundary contour data of each scene segmentation region is extracted. The boundary contour data contains a two-dimensional coordinate sequence of each pixel point constituting the boundary of the closed connected region, where the horizontal axis of the two-dimensional coordinates corresponds to the horizontal direction of the normalized page image data, and the vertical axis corresponds to the vertical direction of the normalized page image data.

[0024] After extracting the boundary contour data of each scene area, for each scene area, all pixel data enclosed by that scene area are extracted from the normalized page image data. All extracted pixel data are input into the optical character recognition engine. The optical character recognition engine outputs the character encoding sequence contained in the scene area and the confidence value corresponding to each character encoding sequence. Character encoding sequences with confidence values ​​lower than the preset confidence threshold are removed. The remaining character encoding sequences are arranged and combined in order from top to bottom and from left to right according to their spatial distribution position in the scene area, and are used as the text content data associated with that scene area.

[0025] All boundary contour data and text content data of the storyboard areas are stored according to the corresponding numbers of the storyboard areas on the page, forming a structured data record for each page of the comic strip, which is used for subsequent sampling and processing of the contour points of the storyboard areas.

[0026] Please see Figure 2As shown, in S2, based on the boundary contour data, contour points of equal arc length are sampled for each storyboard area to obtain the contour point set corresponding to each storyboard area. Then, based on the spatial distribution of the contour points in each contour point set, a reading trajectory vector representing the geometric shape of the corresponding storyboard area is generated, specifically including: For the boundary contour data of each segmentation area, the boundary contour data contains a two-dimensional coordinate sequence of each pixel that constitutes the boundary of the closed connected region. Contour points are sampled from this two-dimensional coordinate sequence in a manner that takes points at equal arc length intervals.

[0027] Specifically, the sum of the Euclidean distances between all adjacent pixels in the boundary contour data is calculated to obtain the total perimeter of the contour of the scene segment. The total perimeter is then divided by a preset number of sampling points to obtain the equal arc length sampling step, where the preset number of sampling points is 64. An arbitrary pixel is selected from the boundary contour data as the starting sampling point. Moving clockwise along the boundary contour data, one pixel is taken at intervals of one equal arc length sampling step as a sampling contour point until the starting sampling point is reached. All the collected sampling contour points are arranged in the order of collection to form the contour point set of the scene segment.

[0028] For each storyboard area's contour point set, determine the geometric center of that contour point set as an internal reference point.

[0029] Specifically, the horizontal coordinate values ​​of all sampled contour points in the contour point set are summed and then divided by the total number of sampled contour points to obtain the horizontal coordinate value of the internal reference point; the vertical coordinate values ​​of all sampled contour points in the contour point set are summed and then divided by the total number of sampled contour points to obtain the vertical coordinate value of the internal reference point.

[0030] The relative orientation data of each sampling contour point relative to the internal reference point is determined as follows: For each sampling contour point, the first difference between the horizontal axis coordinate value of the sampling contour point and the horizontal axis coordinate value of the internal reference point, and the second difference between the vertical axis coordinate value of the sampling contour point and the vertical axis coordinate value of the internal reference point are calculated, and the first difference and the second difference are used as the relative orientation data of the sampling contour point.

[0031] Based on the relative orientation data of each sampled contour point, all sampled contour points are arranged in order of direction around the internal reference point to obtain a contour point sequence.

[0032] Specifically, for each sampled contour point, the directional angle of that point relative to the internal reference point is calculated based on its first and second differences. The directional angle is calculated as follows: taking the internal reference point as the origin and the horizontal rightward direction as the starting edge, rotate counterclockwise to the angle value traversed by the line connecting the internal reference point and the sampled contour point. All sampled contour points are arranged in ascending order of directional angle to form a sequence of contour points connected end to end.

[0033] Based on the directional change between adjacent sampled contour points in the contour point sequence, the geometric morphological feature values ​​of the contour point set are extracted.

[0034] Specifically, the process involves: sequentially extracting every three consecutive sampled contour points from the contour point sequence; using the direction from the first to the second sampled contour point as the first direction vector and the direction from the second to the third sampled contour point as the second direction vector; calculating the angle required for the first direction vector to rotate to the second direction vector; and using this angle as the direction change for the three consecutive sampled contour points. This process is repeated throughout the entire contour point sequence to obtain the direction change sequence. Then, based on the sign of each direction change in the direction change sequence, the sequence is divided into multiple consecutive unidirectional change segments. Within each consecutive unidirectional change segment, all direction changes have the same sign, and the signs of any two adjacent consecutive unidirectional change segments are opposite. The number of direction changes contained in each consecutive unidirectional change segment is determined, and these segments are sorted from most to least frequent. The top three consecutive unidirectional change segments with the most frequent changes are selected as the main change segments.

[0035] Take the absolute values ​​of the directional changes of the three selected main change segments, add them together, and then divide by 3. The resulting value is the geometric morphological characteristic value of the segmentation area.

[0036] The geometric shape feature value of the storyboard area is used as the unique element of the reading trajectory vector of the storyboard area, or the geometric shape feature value is combined with the total number of sampled contour points in the contour point set to form the reading trajectory vector of the storyboard area.

[0037] For storyboard areas with a circular outline and smooth boundaries, the absolute values ​​of the directional change quantities in the sequence are small and the positive and negative signs alternate frequently. The number of consecutive unidirectional change segments is not significantly different, and the geometric morphological characteristic values ​​are small. For storyboard areas with a narrow or elongated outline or sharp corners, the sequence of directional change quantities will show several consecutive large-angle changes in the same direction, and the geometric morphological characteristic values ​​will be large. Thus, the reading trajectory vector can quantitatively distinguish the geometric closure shape and spatial extension of different storyboard areas.

[0038] In S3, based on the reading trajectory vector, a spatial distribution description with the contour points as the reference origin is constructed for each storyboard region, obtaining the morphological feature encoding corresponding to each storyboard region. The morphological feature encoding is used to characterize the geometric topological properties of the corresponding storyboard region under translation and rotation changes, specifically including: First, the reading trajectory vectors for each scene segment are obtained. Each reading trajectory vector contains the geometric morphological feature values ​​and the total number of sampled contour points for that scene segment. For each scene segment, all sampled contour points are extracted from its contour point sequence, and each sampled contour point is used as the current reference origin. Then, a logarithmic polar coordinate grid centered on this current reference origin is constructed. Specifically, with the current reference origin as the center, five radial intervals are divided in a logarithmically equal interval. The boundary radius of each radial interval is set to 1 / 10, 1 / 100, 1 / 1000, 1 / 10000, and 1 / 100000 of the maximum distance from the current reference origin to all sampled contour points in the contour point sequence, respectively. In the angular direction, 12 angular intervals are divided in an angular interval, each spanning 30 degrees. The five radial intervals and the twelve angular intervals are cross-combined to obtain 60 spatially distributed regions.

[0039] For each current reference origin, the number of contour points within each of the 60 spatial distribution regions is determined by counting which of the remaining sampled contour points fall into. Specifically, for each of the remaining sampled contour points, the radial distance relative to the current reference origin is calculated. This radial distance is then compared sequentially with the boundary radii of five radial intervals to determine the radial interval to which the radial distance belongs. Simultaneously, the azimuth angle relative to the current reference origin is calculated, and this azimuth angle is compared sequentially with the angle ranges of 12 angle intervals to determine the angle interval to which the azimuth angle belongs. Based on the determined radial and angle intervals, the sampled contour point is assigned to its corresponding spatial distribution region. After traversing all remaining sampled contour points, the number of contour points corresponding to each of the 60 spatial distribution regions is obtained.

[0040] Based on the relative ratios between the number of contour points in each spatial distribution region, a spatial distribution sub-code for the current reference origin is generated. Specifically, the maximum and minimum number of contour points are determined from all 60 spatial distribution regions. The minimum number of contour points in each spatial distribution region is subtracted from the minimum number of contour points to obtain the difference in the number of contour points for each spatial distribution region. Then, the difference in the number of contour points for each spatial distribution region is divided by the difference between the maximum and minimum number of contour points to obtain the normalized ratio for each spatial distribution region.

[0041] The spatial distribution sub-code of the current reference origin is calculated according to the first formula, which is: In the first formula above, This represents the spatial distribution sub-encoded value of the current reference origin. This represents the sum of the number of contour points within the entire spatial distribution area surrounding the current reference origin. The standard deviation represents the number of contour points in all spatial distribution areas around the current reference origin. This standard deviation is obtained as follows: calculate the average number of contour points in all 60 spatial distribution areas, subtract the average number of contour points in each spatial distribution area from the average number of contour points, take the square, add all 60 squared values ​​together, divide by 60, and then take the square root. The resulting value is the standard deviation. Indicates taking and The larger of the two. When and The greater the difference, The closer the value is to 1, the more uneven the distribution of contour points around the current reference origin in various directions of space; when and The smaller the difference, The closer the value is to 0, the more uniform the distribution of contour points around the current reference origin in all directions of space.

[0042] The spatial distribution sub-encoding process is performed on each of the sampled contour points in the contour point sequence as the current reference origin, resulting in a spatial distribution sub-encoding for each sampled contour point. All spatial distribution sub-encodings are then combined sequentially according to the order of their corresponding sampled contour points in the contour point sequence to obtain the initial morphological feature vector for the storyboard area. This initial morphological feature vector is then normalized by determining its maximum and minimum values. Each element in the initial morphological feature vector is subtracted from the minimum value and then divided by the difference between the maximum and minimum values ​​to obtain the normalized value for each element. All normalized values ​​constitute the morphological feature code for the storyboard area.

[0043] Through the above process, a corresponding morphological feature code is obtained for each storyboard area on the page. When a storyboard area is translated on the page, since the spatial distribution sub-code is calculated based on the radial distance and azimuth angle of the contour points relative to the current reference origin, and the radial interval is divided using a logarithmically equal interval method, the translation operation does not change the relative radial distance ratio and relative azimuth angle difference between each contour point, so the morphological feature code remains unchanged. When a storyboard area is rotated on the page, since the angle interval is divided using an equal angle method and the arrangement order of the spatial distribution sub-code is related to the sequence order of the contour points, and the sequence order of the contour points is itself arranged according to the directional angles around the internal reference point, the rotation operation only causes all azimuth angle values ​​to shift by the same amount of angle, and the relative relationship of the angle intervals into which each contour point falls remains unchanged, so the morphological feature code also remains unchanged. The morphological feature code can eliminate the influence of translation and rotation changes on the geometric morphology description of the storyboard area, and only reflects the contour topological attributes of the storyboard area itself.

[0044] In S4, based on the morphological feature encoding of each scene area, the reading weight of each scene area on the page is calculated, and the spatial position of each scene area is re-encoded according to the reading weight to obtain the geometrically corrected positional encoding features, specifically including: The morphological feature codes of each segmentation area are obtained. The morphological feature codes are formed by sequentially combining the spatial distribution sub-codes corresponding to all sampled contour points in the contour point sequence of the segmentation area. Each spatial distribution sub-code corresponds to a sampled contour point, and the value of each spatial distribution sub-code reflects the uniformity of the distribution of the remaining contour points in the surrounding 60 spatial distribution areas when the sampled contour point is taken as the reference origin.

[0045] For each segmentation area, the morphological feature encoding is used to extract the overall closure intensity value and extension amplitude value of the segmentation area. The specific extraction method is as follows: all spatial distribution sub-encodings in the morphological feature encoding are sorted according to the order of their corresponding sampling contour points in the contour point sequence. The values ​​of each spatial distribution sub-encoding are then taken sequentially and divided into 5 levels according to the value size. The division method is to divide the value range of all spatial distribution sub-encodings from the minimum value to the maximum value into 5 equal value intervals. Each value interval corresponds to a level. The levels are recorded as the first level, the second level, the third level, the fourth level, and the fifth level in order of value from low to high. Traverse all sampling contour points corresponding to the storyboard area, count the level to which the spatial distribution sub-encode of each sampling contour point belongs, and obtain the level label sequence. Each element in the level label sequence is an integer from 1 to 5, which represents the level to which the corresponding sampling contour point belongs. At the same time, record the repetition frequency of each level in the level label sequence, determine the level with the highest repetition frequency as the highest frequency level, and determine the level with the lowest repetition frequency as the lowest frequency level.

[0046] Based on the position of each sampled contour point corresponding to the highest frequency level in the contour point sequence, the continuous distribution span of these sampled contour points in the circumferential direction around the internal reference point is determined. Specifically, the contour point sequence is regarded as a ring sequence with its beginning and end connected. From all the sampled contour points corresponding to the highest frequency level, the longest continuous segment that is continuously arranged on the ring sequence and belongs to the highest frequency level is found. The length of the contour point sequence spanned by the longest continuous segment is the continuous distribution span. The larger the continuous distribution span, the more similar the spatial distribution sub-code values ​​of most contour points in the segmentation area are, that is, the curvature of the contour boundary in each direction tends to be consistent, and the higher the degree of closure is. Therefore, the continuous distribution span is divided by the total number of sampled contour points and multiplied by 100. The resulting value is used as the closure intensity value of the segmentation area. The closure intensity value ranges from 0 to 100.

[0047] Based on the position of each sampled contour point corresponding to the lowest frequency level in the contour point sequence, the dispersion span of these sampled contour points in the circumferential direction around the internal reference point is determined. Specifically: the contour point sequence is considered as a ring sequence with its beginning and end connected. From all the sampled contour points corresponding to the lowest frequency level, the number of intervals between any two adjacent sampled contour points belonging to that lowest frequency level in the ring sequence is found. The maximum value among all intervals is taken as the dispersion span. The larger the dispersion span, the more abnormal the spatial distribution sub-code values ​​of the contour points discretely distributed in different directions in the segmentation area. That is, the contour boundary shows a prominent extension or depression in some directions, and the higher the degree of extension, the more the dispersion span is divided by the total number of sampled contour points and multiplied by 100. The resulting value is taken as the extension amplitude value of the segmentation area. The extension amplitude value ranges from 0 to 100. The combination of the closure strength value and the extension amplitude value is used as the geometric morphology analysis result of the segmentation area.

[0048] Based on the closure intensity and extension amplitude values ​​of the storyboard area, the reading weight of that area on the page is calculated. Specifically: the closure intensity value is divided by the sum of the closure intensity and extension amplitude values ​​to obtain the closure intensity percentage; the extension amplitude value is divided by the sum of the closure intensity and extension amplitude values ​​to obtain the extension amplitude percentage. The closure intensity percentage is used as the first weighting coefficient, and the extension amplitude percentage is used as the second weighting coefficient. The first weighting coefficient and the second weighting coefficient are multiplied together, and the product is the reading weight of that storyboard area.

[0049] When the outline of the storyboard area is close to a circle and the boundary is smooth, the closure strength value is high and the extension value is low, the first weighting coefficient is large and the second weighting coefficient is small, and the product of the two is in the middle. When the outline of the storyboard area is long and narrow or has obvious protrusions, the extension value is high and the closure strength value is low, and the product of the first weighting coefficient and the second weighting coefficient changes accordingly. Through this product relationship, the reading weight can comprehensively reflect the degree of visual attention that the storyboard area should receive on the page due to its geometric shape.

[0050] Based on the reading weight of each scene area, the original spatial position of the scene area on the page is corrected. Specifically: the horizontal axis coordinates of all pixels in the boundary contour data of the scene area are obtained, and their average value is calculated as the original center horizontal axis position of the scene area; the vertical axis coordinates of all pixels in the boundary contour data of the scene area are obtained, and their average value is calculated as the original center vertical axis position of the scene area.

[0051] The original center horizontal and vertical axis positions are multiplied by the reading weight of the segmentation area to obtain the corrected center horizontal and vertical axis positions. Simultaneously, the original center horizontal axis position is subtracted from the horizontal axis coordinates of all pixels in the segmentation area boundary contour data to obtain the horizontal axis offset. Each horizontal axis offset is multiplied by the reading weight and then added back to the original center horizontal axis position to obtain the corrected horizontal axis coordinates of each pixel. Similarly, the original center vertical axis position is subtracted from the vertical axis coordinates of all pixels to obtain the vertical axis offset. Each vertical axis offset is multiplied by the reading weight and then added back to the original center vertical axis position to obtain the corrected vertical axis coordinates of each pixel. The corrected horizontal and vertical axis coordinates of all pixels are combined in their original order to form the corrected position data of the segmentation area. This corrected position data is used as the geometrically corrected position encoding feature.

[0052] Through the above corrections, the reading weight of circular or square panel areas with high closure tends to be moderate, and their spatial positions remain basically unchanged; while the spatial positions of narrow or irregularly shaped panel areas are shifted to a corresponding extent because their reading weight deviates from the moderate value, thereby reducing the absolute influence of their spatial coordinates on the ordering in the subsequent narrative connection process.

[0053] In S5, narrative connection paths between different scene areas are constructed based on their location encoding features and associated text content data. These scene areas are then arranged into a logical temporal sequence according to these paths. Finally, the visual and textual features of each scene area are correlated based on this logical temporal sequence, and the classification results are output, including: Obtain the positional encoding features of each scene segment region and the text content data associated with that scene segment region. The positional encoding features include the corrected center horizontal axis position, center vertical axis position, and the corrected horizontal and vertical coordinate values ​​of all boundary pixels of that scene segment region.

[0054] The text content data consists of character encoding sequences extracted from the storyboard area using optical character recognition. For the text content data of each storyboard area, a key pronoun extraction operation is performed: the text content data is divided into several individual sentences according to punctuation marks, and word segmentation is performed on each sentence. The segmentation method is to perform longest match segmentation on the continuous character sequence in the sentence according to a preset common word dictionary to obtain the word sequence corresponding to the sentence.

[0055] Words belonging to surnames, titles, and specific names of individuals are selected from the word sequence as key referents. Key referents include object names and object titles, where the object name is a unique noun referring to a specific object, and the object title is a non-unique appellation referring to the object. The position number of each key referent in the text content data is recorded, where the position number represents the order of the sentence containing the key referent among all sentences. The referent object number pointed to by each key referent is also recorded. Key referents with the same object name are grouped into the same referent object number, and object titles with a known correspondence to the same object name are also grouped into that referent object number.

[0056] Based on the relative arrangement of the positional coding features of each scene area on the page, and combined with the synonymous relationship of key pronouns in the text content data of each scene area, the initial association path between each scene area is determined.

[0057] The specific process is as follows: obtain the corrected center horizontal axis position and the corrected center vertical axis position from the position encoding features of each segmentation area. For any two segmentation areas, calculate the absolute value of the difference between their center horizontal axis positions to obtain the horizontal axis distance, calculate the absolute value of the difference between their center vertical axis positions to obtain the vertical axis distance, add the square of the horizontal axis distance and the square of the vertical axis distance, and take the square root to obtain the center point distance value between the two segmentation areas. The reciprocal of the center point distance value is taken as the proximity degree between the two segmentation areas.

[0058] When two storyboard areas are adjacent on the page, the distance between their center points is small, resulting in a high degree of proximity. Conversely, when two storyboard areas are far apart, the degree of proximity is small. Simultaneously, the referential object numbers of key pronouns in the text content data of each storyboard area are obtained. For any two storyboard areas, the total number of occurrences of key pronouns with the same referential object number in both areas is counted, and this total number of occurrences is used as the referential association strength between the two storyboard areas.

[0059] The initial correlation coefficient between any two scene areas is obtained by multiplying their proximity degree by their referential association strength. For each scene area, its initial correlation coefficient is added to the initial correlation coefficients of all other scene areas on the page to obtain the total correlation value of that scene area. The scene area with the highest total correlation value is determined as the starting node of the initial path. Starting from this starting node, among all other scene areas adjacent to the starting node, the scene area with the highest initial correlation coefficient is selected as the second path node. Then, starting from the second path node, among all other scene areas adjacent to the second path node that have not yet been selected, the scene area with the highest initial correlation coefficient is selected as the third path node, and so on, until all scene areas on the page have been selected. The scene areas are then arranged in the order of selection to obtain the initial correlation path.

[0060] Based on the order in which key pronouns appear on the page in the text content data of each scene segment, the temporal weight of each path node in the initial association path is determined. Specifically: for each scene segment in the initial association path, the position numbers of all key pronouns in its text content data are obtained, the average position number of all position numbers is calculated, and the average position number is used as the temporal weight of that scene segment. The smaller the average position number, the earlier the key pronouns in that scene segment appear on the page.

[0061] The path node with the lowest temporal weight in the initial association path is determined as the path starting point. Starting from this path starting point, according to the original adjacency relationships of the nodes in the initial association path, all path nodes that were previously arranged before the path starting point are moved to the position after the path starting point, while the remaining path nodes maintain their original relative order. This generates the narrative association path for this storyboard area. The order of nodes in the narrative association path corresponds to the order in which key pronouns appear on the page.

[0062] Based on the arrangement order of the scene regions in the narrative association path, each scene region is arranged sequentially into a logical temporal sequence. Each sequence position in the logical temporal sequence corresponds to a scene region, and the order of the sequence positions indicates the reading order of that scene region in the narrative logic. For each scene region in the logical temporal sequence, its visual features are associated with its text content data. The association method is as follows: all pixel data enclosed by the boundary contour data of the scene region is used as the visual feature data of the scene region, and the text content data of the scene region is used as the text feature data corresponding to the visual feature data. The visual feature data and text feature data of each scene region are combined into image-text association data pairs according to the arrangement order in the logical temporal sequence.

[0063] All image-text associated data pairs are input into a pre-constructed classification mapping relationship in logical time sequence. This mapping relationship is a correspondence between the sequence features of the image-text associated data pairs and multiple preset classification labels. This correspondence is pre-established by performing all the above steps on comic book samples with known classification results, extracting sequence features, and statistically matching them with the classification labels. The image-text associated data pairs of each panel area of ​​the current comic book are matched with this classification mapping relationship, and the preset classification label with the highest matching degree is output as the classification result of the comic book, thus completing intelligent classification.

[0064] Please see Figure 3 As shown, a comic strip intelligent classification system combining image and text analysis includes: The data acquisition module collects image data of the comic strip pages and extracts the boundary contour data of each panel area and the text content data associated with each panel area from the page image data. The contour sampling and trajectory generation module samples contour points of equal arc length for each segmentation area based on the boundary contour data, obtains the contour point set corresponding to each segmentation area, and generates a reading trajectory vector to represent the geometric shape of the corresponding segmentation area based on the spatial distribution of the contour points in each contour point set. The morphological feature encoding module constructs a spatial distribution description for each segmentation area based on the reading trajectory vector, with the outline point as the reference origin, and obtains the morphological feature encoding corresponding to each segmentation area. The morphological feature encoding is used to characterize the geometric topological properties of the corresponding segmentation area under translation and rotation. The position weight correction module calculates the reading weight of each segment area on the page based on the morphological feature encoding of each segment area, and re-encodes the spatial position of each segment area according to the reading weight to obtain the position encoding features after geometric morphological correction. The narrative association and classification module constructs narrative association paths between each scene area based on the location encoding features of each scene area and its associated text content data. Based on the narrative association paths, each scene area is arranged into a logical temporal sequence. Based on the logical temporal sequence, the visual features and text features of each scene area are correlated, and the classification results are output.

[0065] The working principle of this invention is as follows: Image data of comic book pages is collected by scanning, and the boundary contour data of each panel area and its associated text content data are extracted. Contour point sets are obtained by sampling the boundary contours of each panel area using equal arc lengths. Based on the spatial distribution of these contour point sets, a reading trajectory vector representing the geometric shape of the panel area is generated. A logarithmic polar coordinate spatial distribution description is constructed based on the reading trajectory vector, with each contour point as the reference origin, generating a morphological feature code with translation and rotation invariance. The closure intensity value and extension amplitude value of the panel area are parsed from the morphological feature code. Based on this, reading weights are calculated, and the spatial position of the panel area is geometrically corrected to obtain the corrected positional coding features. Finally, based on the corrected positional coding features of each panel area and the synonymy and occurrence order of key pronouns in the text content data, a narrative connection path between panel areas is constructed and arranged into a logical temporal sequence. The visual features and textual features of each panel area are correlated, and the classification results are output.

[0066] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for intelligent classification of comic books combining image and text analysis, characterized in that, Includes the following steps: S1: Collect image data of the comic strip page, and extract the boundary contour data of each panel area and the text content data associated with each panel area from the page image data; S2: Based on the boundary contour data, sample contour points of equal arc length for each segmentation area to obtain the contour point set corresponding to each segmentation area, and generate a reading trajectory vector to represent the geometric shape of the corresponding segmentation area based on the spatial distribution of contour points in each contour point set. S3: Based on the reading trajectory vector, construct a spatial distribution description for each segmentation area with the outline point as the reference origin, and obtain the morphological feature code corresponding to each segmentation area. The morphological feature code is used to characterize the geometric topological properties of the corresponding segmentation area under translation and rotation. S4: Based on the morphological feature encoding of each segmentation area, calculate the reading weight of each segmentation area on the page, and re-encode the spatial position of each segmentation area according to the reading weight to obtain the position encoding feature after geometric morphology correction. S5: Based on the positional encoding features of each scene area and its associated text content data, construct the narrative connection path between each scene area, arrange each scene area into a logical temporal sequence according to the narrative connection path, and associate the visual features and text features of each scene area according to the logical temporal sequence to output the classification result.

2. The intelligent classification method for comic books combining image and text analysis according to claim 1, characterized in that, The process of generating the reading trajectory vector is as follows: After obtaining the contour point set of each storyboard area, determine the relative orientation data of each contour point relative to the reference point inside the storyboard area. Based on the relative orientation data, all contour points are arranged in order of direction surrounding the internal reference point to obtain a contour point sequence. Based on the directional change between adjacent contour points in the contour point sequence, extract the geometric morphological feature values ​​of the contour point set. A reading trajectory vector is generated based on geometric morphological feature values. The reading trajectory vector is used to characterize the closed shape and spatial extension of the storyboard area outline.

3. The intelligent classification method for comic books combining image and text analysis according to claim 2, characterized in that, The extraction of geometric morphological feature values ​​of the contour point set specifically includes: Obtain the two direction vectors formed by every three consecutive contour points in the contour point sequence, determine the direction change between each direction vector, and obtain the direction change sequence; Based on the positive and negative signs of the directional change quantities in the sequence, the sequence of directional change quantities is divided into multiple continuous segments of change in the same direction. Determine the number of directional changes contained in each continuous unidirectional change segment, sort the continuous unidirectional change segments in descending order of the number, and select the continuous unidirectional change segments with the highest number of changes. Based on the directional change values ​​of each selected continuous unidirectional change segment, geometric morphological feature values ​​are generated to characterize the curvature of the boundary of the storyboard area.

4. The intelligent classification method for comic books combining image and text analysis according to claim 1, characterized in that, The generation process of the morphological feature code is as follows: Obtain the sequence of contour points corresponding to the reading trajectory vector, and for each current contour point in the sequence, determine the relative distance and relative orientation of the remaining contour points relative to the current contour point. Based on all the relative distances and relative orientations corresponding to each current contour point, the remaining contour points are assigned to the corresponding spatial distribution areas according to the hierarchical assignment of relative distances and the interval assignment of relative orientations. The number of contour points in each spatial distribution area is counted, and a spatial distribution sub-code for each current contour point is generated based on the relative ratio of the number of contour points in each spatial distribution area. The spatial distribution sub-encodes of all current contour points in the contour point sequence are combined according to the arrangement order of the contour point sequence to generate the morphological feature code of the storyboard region.

5. The intelligent classification method for comic books combining image and text analysis according to claim 4, characterized in that, The generation of spatial distribution sub-encoders for each current contour point specifically includes: Obtain the specific values ​​of the number of contour points in each spatial distribution area, and determine the maximum and minimum values ​​from the specific values ​​of all spatial distribution areas. Based on the relative proportion between the number of contour points and the maximum value in each spatial distribution area, the hierarchical classification of each spatial distribution area is determined, and the hierarchical label corresponding to each spatial distribution area is obtained. The hierarchical labels corresponding to each spatial distribution region are combined according to the division order of each spatial distribution region to generate the spatial distribution sub-code of the current contour point. The spatial distribution sub-code is used to characterize the dense distribution features of contour points around the current contour point in various spatial directions.

6. The intelligent classification method for comic books combining image and text analysis according to claim 1, characterized in that, The acquisition of geometrically corrected position coding features specifically includes: Obtain the hierarchical labels of the number of contour points in each spatial distribution area in the morphological feature encoding, and parse out the closure strength value, which represents the degree of closure of the storyboard area, and the extension amplitude value, which represents the extension range of the boundary of the storyboard area, from the hierarchical labels. The first weighting coefficient is determined based on the closure strength value, and the second weighting coefficient is determined based on the extension amplitude value. The product of the first weighting coefficient and the second weighting coefficient is used as the reading weight of the storyboard area. Based on the reading weight of the storyboard area, the horizontal and vertical coordinate values ​​of the original spatial coordinates of the storyboard area are scaled by the same proportion to obtain the corrected horizontal and vertical position data of the storyboard area. The horizontal and vertical position data after correction of the storyboard area are combined into a geometrically corrected position coding feature.

7. The intelligent classification method for comic books combining image and text analysis according to claim 6, characterized in that, The analysis process for the extension amplitude value is as follows: Obtain the order and repetition frequency of each level label in the hierarchical label, and determine the highest and lowest frequency levels based on the repetition frequency of each level label; Based on the continuous distribution span of the spatial distribution area corresponding to the highest frequency level in the circumferential direction around the current contour point, the closure intensity value of the segmentation area where the current contour point is located is determined. The larger the continuous distribution span, the higher the closure intensity value. Based on the scattering span of the spatial distribution area corresponding to the lowest frequency level in the circumferential direction around the current contour point, determine the extension amplitude value of the segment area where the current contour point is located. The larger the scattering span, the higher the extension amplitude value. The closure strength value and extension amplitude value are combined to form the geometric shape analysis result of the storyboard area.

8. The intelligent classification method for comic books combining image and text analysis according to claim 1, characterized in that, The construction of narrative connection paths between various scene areas specifically includes: Obtain the location encoding features of each storyboard area and the associated text content data, and extract key pronouns that refer to the same object from the text content data. Key pronouns include object name and object title. Based on the relative arrangement of the positional coding features of each scene area on the page, and combined with the synonymy of key pronouns in the text content data of each scene area, the initial association path between each scene area is determined. Based on the order in which key pronouns appear on the page in the text content data of each scene area, the temporal weight of each path node in the initial association path is determined. The path node with the highest temporal weight is taken as the starting point of the path, and the remaining path nodes are arranged in sequence along the initial association path to generate the narrative association path.

9. The intelligent classification method for comic books combining image and text analysis according to claim 8, characterized in that, The process of generating the initial association path is as follows: Obtain the positional coding features of each segmentation area, determine the relative orientation relationship between the center points of each segmentation area based on the horizontal and vertical positional data in each positional coding feature, and calculate the proximity degree between each segmentation area based on the relative orientation relationship. Obtain the referents of key pronouns in the text content data of each storyboard area, count the number of times the same referents are shared between any two storyboard areas, and determine the strength of the referential association between each storyboard area based on the number of referent co-occurrences. The proximity degree between each segmentation area is multiplied and summed with the referential association strength to obtain the association weight value between each segmentation area. Based on the association weight value, the storyboard area with the highest association weight value is paired as the initial path backbone, and the remaining storyboard areas are connected to the initial path backbone in descending order of their association weight value with the initial path backbone to generate the initial association path.

10. A comic strip intelligent classification system combining image and text analysis, characterized in that, A method for intelligent classification of comic books combining image and text analysis as described in any one of claims 1-9 includes: The data acquisition module collects image data of the comic strip pages and extracts the boundary contour data of each panel area and the text content data associated with each panel area from the page image data. The contour sampling and trajectory generation module samples contour points of equal arc length for each segmentation area based on the boundary contour data, obtains the contour point set corresponding to each segmentation area, and generates a reading trajectory vector to represent the geometric shape of the corresponding segmentation area based on the spatial distribution of the contour points in each contour point set. The morphological feature encoding module constructs a spatial distribution description for each segmentation area based on the reading trajectory vector, with the outline point as the reference origin, and obtains the morphological feature encoding corresponding to each segmentation area. The morphological feature encoding is used to characterize the geometric topological properties of the corresponding segmentation area under translation and rotation. The position weight correction module calculates the reading weight of each scene area on the page based on its morphological feature encoding, and re-encodes the spatial position of each scene area according to the reading weight to obtain geometrically corrected position encoding features. The narrative association classification module constructs narrative association paths between each scene area based on its position encoding features and associated text content data, arranges each scene area into a logical temporal sequence according to the narrative association paths, and associates the visual features and text features of each scene area according to the logical temporal sequence to output the classification results.

Images