Multi-image adaptive fusion display method based on image analysis

Abstract

The present application relates to industrial visual information fusion display technical field, specifically for the multi-image adaptive fusion display method based on image analysis, comprising: parsing the original mixed data recorded by visual system, automatically separating the image, motion trajectory data and process parameter data therein. According to the trajectory data and image, generate combined data unit, according to process parameters, generate grouped text information, and construct process signal trajectory coordinate set binding process parameters and spatial coordinates. Using this coordinate set driving, the image sequence, parameter text and motion trajectory are precisely aligned and fused in time and space in the display interface, and finally an integrated on-screen display interface is generated, and it is adaptive to different resolution screens. The method solves the problem of difficult separation of original data mixing and multi-source information display fragmentation, and improves the intuitiveness and efficiency of process monitoring and analysis.

Description

Technical Field

[0001] This invention relates to the field of industrial visual information fusion display technology, and in particular to a method for adaptive fusion and simultaneous display of multiple images based on image analysis. Background Technology

[0002] In the field of industrial visual monitoring, the raw data acquired by vision systems typically includes image streams and various types of non-image data recorded simultaneously. In existing technologies, this non-image data, such as equipment motion trajectories and process parameters, is often stored in a mixed, unseparated form within the same data stream or file. This raw, mixed data state makes it difficult to directly extract pure motion path sequences or independent process parameter sequences, hindering independent and in-depth specialized analysis of trajectories or processes. The very first step in data utilization faces an obstacle.

[0003] Traditional multi-information display methods typically employ split windows or view switching to display images, data reports, and trajectory graphs separately. This method physically isolates information with inherent spatiotemporal correlations across different display areas or timelines. Operators cannot intuitively establish a direct correspondence between specific process parameter values ​​and the specific state of the equipment at a particular point in space within a single view, making it difficult to quickly conduct overall process analysis and positioning, and the value of information synergy is not effectively presented.

[0004] The present invention aims to solve the two key technical problems of directly separating specific information from mixed data sources and achieving precise spatiotemporal alignment of multimodal information within a unified display interface. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a multi-image adaptive fusion and simultaneous display method based on image analysis.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multi-image adaptive fusion and simultaneous display method based on image analysis, comprising: The system acquires raw data recorded by the vision system, parses the raw data, and extracts image data and other types of data associated with the image data. The parsed other types of data are processed to identify and separate the trajectory region data and process region data contained in the other types of data. In the initial state, the trajectory region data and process region data are merged together. Based on the trajectory region data and the image data, a corresponding combined data unit is generated; Based on the process area data and the image data, corresponding grouping information is generated; The combined data units and the grouping information are converted into recognizable text-type data; Based on the trajectory region data and process region data, a set of process signal trajectory coordinates is generated; Based on the set of process signal trajectory coordinates, the converted text data, and the image data, fused display data is constructed; The fused display data is adaptively scaled according to the screen resolution parameters of the display terminal. Based on the fused display data after adaptive scaling, a visual simultaneous display interface containing multiple images, text, and coordinate trajectories is generated, ensuring that the visual simultaneous display interface is compatible with displays of various resolutions.

[0007] Preferably, the parsed other types of data are processed to identify and separate the trajectory region data and process region data contained in the other types of data, including: Perform format analysis on the other types of data to identify their data structures; Based on the data structure, the data segments representing location information in the other types of data are located as the regions to be segmented. Analyze the data characteristics of the region to be segmented, including the frequency of data value changes and the distribution density of data values; Based on the frequency of change of the data values, high-frequency change data segments and low-frequency change data segments are divided. Based on the distribution density of the data values, target data segments with a distribution density higher than a preset density threshold are identified in the high-frequency changing data segments as candidates for initial trajectory region data. Based on the distribution density of the data values, target data segments with a distribution density lower than the preset density threshold are identified in the low-frequency changing data segments as candidates for initial process area data. Boundary verification is performed on the initial trajectory region data candidates and the initial process region data candidates to separate the final trajectory region data and the final process region data.

[0008] Preferably, based on the trajectory region data and the image data, a corresponding combined data unit is generated, including: The number of image data to be combined is determined based on the data length of the trajectory region data; The image data is sorted according to the data recording order of the trajectory area data to generate an ordered image data sequence; Each image data in the ordered image data sequence is associated and bound one-to-one with the sub-data at the corresponding data record position in the trajectory region data to generate multiple primary data pairs; Add identification information to each of the primary data pairs; the identification information is used to distinguish different data source categories. All primary data pairs carrying the aforementioned identification information are aggregated to form a set of combined data units. The specific combined data unit that meets the current processing requirements is extracted from the set of combined data units as the final combined data unit.

[0009] Preferably, based on the process area data and the image data, corresponding grouping information is generated, including: Analyze the content of the process area data to identify the process type identifier contained in the process area data; Based on the process type identifier, the process area data is divided into multiple process data subsets; The image data is arranged according to the order of acquisition time or spatial location; The arranged image data is matched with the multiple process data subsets, and an associated image data subsequence is assigned to each process data subset; Based on the allocation results, a grouping label is defined for each subset of process data and its associated image data subsequence; The correspondence between all process data subsets and image data subsequences with the grouping labels is stored to generate structured grouping information.

[0010] Preferably, converting the combined data unit and the grouping information into recognizable text-type data includes: Identify the data content format in the combined data unit; If the data content format in the combined data unit is non-text format, then according to the preset translation mapping rules, the non-text format data content is converted into a predefined text string; Identify the format of the group tag content in the grouping information; If the group tag content format is non-text format, then according to the group type mapping relationship, the non-text format group tag content is converted into the corresponding text description; The converted text string and text description are then processed to unify their encoding formats. The text strings and text descriptions that have undergone unified encoding are integrated, categorized and stored according to their source type, and a recognizable text type data set is generated.

[0011] Preferably, based on the trajectory region data and the process region data, a set of process signal trajectory coordinates is generated, including: Extract a sequence of discrete coordinate points representing the motion path from the trajectory region data; Extract process signal trigger points that are temporally or logically associated with the discrete coordinate point sequence from the process area data; Map each of the process signal trigger points to the discrete coordinate point sequence to find the corresponding mapped coordinate point; Centered on the mapped coordinate points, and combined with the type of process signal, a geometric region with a specific shape is generated; Calculate the boundary coordinates of the geometric region; The discrete coordinate point sequence, the process signal trigger point, the mapped coordinate point, and the boundary coordinates of the geometric region are associated to form a complete trajectory with process annotations; The coordinate information of all trajectories with process annotations is summarized to generate a set of process signal trajectory coordinates.

[0012] Preferably, based on the set of process signal trajectory coordinates, the converted text data, and the image data, fused display data is constructed, including: Create a fused data structure template, which includes an image data storage area, a text data storage area, and a coordinate trajectory data storage area; The image data is stored in the image data storage area according to its logical order in the combined data unit or grouping information; The converted text data is stored in the text data storage area according to its corresponding source category; The coordinate data and associated process annotation information in the process signal trajectory coordinate set are stored in the coordinate trajectory data storage area. Establish a data index relationship between the image data storage area, text data storage area, and coordinate trajectory data storage area. This data index relationship enables images, text, and coordinate trajectories to be correctly associated based on timestamps or event identifiers during display. All stored data and its index relationships are encapsulated to generate the final fused display data package.

[0013] Preferably, the fused display data is adaptively scaled according to the screen resolution parameters of the display terminal, including: Obtain the screen resolution parameters of the display terminal, wherein the screen resolution parameters include the screen width pixel value and the screen height pixel value; Read the original image size of the image data storage area in the fused display data packet; Read the original coordinate range of the coordinate trajectory data storage area in the fused display data packet; Calculate the aspect ratio of the screen display area based on the screen width pixel value and screen height pixel value; Based on the aspect ratio of the screen display area and the original image size, the scaling ratio of the image data is calculated so that all images can fit the screen display area while maintaining the original aspect ratio. Based on the aspect ratio of the screen display area and the original coordinate range, the scaling and translation parameters of the coordinate trajectory data are calculated so that the trajectory graphic can be completely displayed within the specified area of ​​the screen. Based on the calculated scaling ratio of the image data, the image data in the image data storage area is resampled. Based on the scaling and translation parameters of the calculated coordinate trajectory data, the coordinate data in the coordinate trajectory data storage area is transformed. Update the data in the fused display data packet after resampling and transformation.

[0014] Preferably, the scaling ratio of the image data is calculated based on the aspect ratio of the screen display area and the original image size, including: Determine the total number of images that need to be displayed on the same screen; Based on the aspect ratio of the screen display area, the layout of the total number of images on the screen is planned, and the layout includes the number of images displayed in each row and the total number of rows; Based on the layout method, the screen width pixel value, and the screen height pixel value, calculate the width pixel value and height pixel value of the display sub-region allocated to each image; For each image, the ratio of its original image width to the width pixel value of the corresponding display sub-region is calculated as the first candidate scaling ratio; For each image, the ratio of its original image height to the height pixel value of the corresponding display sub-region is calculated as the second candidate scaling ratio; For each image, compare its corresponding first candidate scaling ratio with the second candidate scaling ratio, and select the candidate scaling ratio with the smaller value as the final scaling ratio of the image to ensure that the image can be completely placed in its display sub-region without exceeding the boundary.

[0015] Preferably, the image data in the image data storage area is resampled according to the calculated scaling ratio of the image data, including: Calculate the target width and target height pixel values ​​after image resampling based on the final scaling ratio of each image; For each image in the image data storage area, a bilinear interpolation algorithm is used to calculate the color value of each pixel in the new pixel grid based on the target width pixel value and the target height pixel value; The color values ​​of the new pixels calculated using the bilinear interpolation algorithm are assigned to generate the scaled image; Image filtering algorithms are used to smooth the scaled image to eliminate the jagged effect caused by resampling; The smoothed image data replaces the corresponding original image data in the image data storage area of ​​the fused display data packet.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By parsing the original mixed data stream using specific data processing logic, the implicit motion trajectory information and process parameter information can be identified and separated. This process deconstructs the originally entangled non-visual data into independent, pure structured data sources. Based on the separated data, the system can construct combined units describing motion paths and grouped information describing process states, providing a clear data foundation for subsequent fusion analysis, similar to previous technologies.

[0017] Based on the separated trajectory and process data, a coordinate set is generated that binds process parameters to spatial coordinates and time series. This set serves as the core space-attribute mapping relationship, driving strict alignment and synchronous rendering of image sequences, parameter text, and motion trajectories in screen space when constructing display content. Operators can directly observe the instantaneous association between parameter values ​​and their spatial locations and corresponding images within a single view, achieving an integrated presentation of multimodal information in the spatiotemporal dimensions.

[0018] The generated fused display data can be adaptively scaled according to the terminal screen resolution. This allows complex visualization interfaces containing multiple elements such as images, text, and trajectories to maintain the integrity and readability of the content on display devices of different specifications, enhancing the deployment adaptability and environmental compatibility of the technical solution. Attached Figure Description

[0019] Figure 1 This is a flowchart of the multi-image adaptive fusion and simultaneous display method based on image analysis described in this invention; Figure 2 A flowchart generated for combining data units; Figure 3 A flowchart generated for grouping information; Figure 4 A comparison chart of adaptive scaling parameters at different screen resolutions; Figure 5 This is a dual Y-axis analysis diagram of the Gaussian filtering smoothing stage. Detailed Implementation

[0020] See Figure 1The raw data recorded by the vision system is acquired, including image data and other related data types. The raw data is parsed, and the image data and other data types are extracted. The other data types are processed to identify and separate the trajectory region data and process region data initially merged together. The trajectory region data is combined with the image data to generate corresponding combined data units. The process region data is combined with the image data to generate corresponding grouping information. The combined data units and grouping information are converted into recognizable text-type data. Based on the trajectory region data and process region data, a set of process signal trajectory coordinates is generated. Based on the process signal trajectory coordinate set, the converted text-type data, and the image data, fused display data is constructed. The screen resolution parameters of the display terminal are acquired, and the fused display data is adaptively scaled according to these parameters. Based on the processed fused display data, a visual simultaneous display interface containing multiple images, text, and coordinate trajectories is generated, which is compatible with displays of various resolutions.

[0021] In one embodiment of the present invention, see [reference] Figure 2 After the raw data recorded by the vision system is parsed, other types of data are extracted. Format analysis is performed on these other data types to identify their data structure. Based on this structure, data segments representing location information are located as the regions to be segmented. The data characteristics of these regions are analyzed, including the frequency and density of data value changes. Based on the frequency of data value changes, high-frequency and low-frequency data segments are divided. The frequency of data value changes is calculated using the following formula:

[0022] in: Indicates the frequency of change. Indicates the first data in the region to be segmented The value of each data point This represents the total number of data points in the region to be segmented. Based on the distribution density of data values, target data segments with a distribution density higher than a preset density threshold are identified in high-frequency changing data segments as initial trajectory region data candidates. Based on the distribution density of data values, target data segments with a distribution density lower than a preset density threshold are identified in low-frequency changing data segments as initial process region data candidates. Boundary verification is performed on the initial trajectory region data candidates and the initial process region data candidates to separate the final trajectory region data and the final process region data.

[0023] In specific implementation, the number of image data to be combined is determined based on the data length of the trajectory region data. The image data is sorted according to the data recording order of the trajectory region data to generate an ordered image data sequence. Each image data in the ordered image data sequence is associated and bound one-to-one with the corresponding sub-data in the trajectory region data to generate multiple primary data pairs. Identification information is added to each primary data pair to distinguish different data source categories. All primary data pairs with identification information are summarized to form a combined data unit set. Specific combined data units that meet the current processing requirements are extracted from the combined data unit set as the final combined data unit. In some embodiments, the identification information includes a serial number and an acquisition device identifier. The association and binding of primary data pairs is achieved by establishing a mapping table, where the key is the index of the trajectory region data sub-data, and the value is the storage address of the image data. Optionally, the ordered image data sequence is sorted according to the ascending order of the image data timestamps, and the extraction of the combined data unit set is based on preset filtering conditions, including time range or data source category. It can be understood that the data recording order of the trajectory region data is synchronized with the acquisition order of the image data to ensure the accuracy of the association of the primary data pairs.

[0024] In practical implementation, the analysis of data value change frequency and data value distribution density is performed through a parallel computing module to improve processing efficiency. The division of high-frequency and low-frequency changing data segments adopts an adaptive threshold method, with the threshold dynamically calculated based on the overall statistical characteristics of the data in the region to be segmented. Boundary verification of initial trajectory region data candidates and initial process region data candidates is completed by comparing the similarity of data values ​​in adjacent data segments; the boundary position is confirmed when the similarity is lower than a set value. Optionally, a preset density threshold is loaded during system initialization and updated according to the real-time data stream during operation. The addition of identification information is implemented through a data encapsulation function, which packages image data and trajectory region data sub-data into data blocks of a unified format. It can be understood that the storage of the combined data unit set uses a linked list data structure, facilitating dynamic insertion and deletion operations. Specific combined data units that meet the current processing requirements are retrieved from the combined data unit set through a query function.

[0025] In one embodiment of the present invention, see [reference] Figure 3 The process analyzes the content of the process area data and identifies the process type identifiers contained within it. Based on these identifiers, the process area data is divided into multiple process data subsets. Image data is arranged according to either the acquisition time order or the spatial location order. The arranged image data is then matched against multiple process data subsets, and an associated image data subsequence is assigned to each subset. The matching process calculates the matching degree between the timestamps of the image data and the time windows of the process data subsets.

[0026] in: Indicates the first The degree of matching of a subset of process data Indicates the first The timestamp of each image data Indicates the first The time reference points for each subset of process data are defined. Based on the allocation results, a grouping label is defined for each subset of process data and its associated image data subsequence. The correspondence between all process data subsets and image data subsequences with grouping labels is stored, and structured grouping information is generated. In some embodiments, the process type identifier is represented by a fixed-byte code segment. The content of the code segment is compared with a predefined process type dictionary to complete the identification. The image data arrangement operation is performed in the memory buffer, and the allocation of image data subsequences is based on the matching degree. The numerical values ​​have been selected in descending order. This is understandable; the time reference point... The median of the time field is taken from the internal records of the process data subset. The generation of group labels combines process type identifier and sequence number. The structured grouping information is stored in the form of relational database tables.

[0027] In specific implementation, the data content format in the combined data unit is identified. If the data content format in the combined data unit is non-text, it is converted into a predefined text string according to a preset translation mapping rule. The format of the group label content in the grouping information is identified. If the group label content format is non-text, it is converted into a corresponding text description according to the grouping type mapping relationship. The converted text string and text description are then processed for unified encoding. The unified encoding text string and text description are integrated and categorized according to their source category, ultimately generating a recognizable text type data set. In some embodiments, the preset translation mapping rule is stored in a separate configuration file. The configuration file defines a mapping table from binary code to descriptive statements. Unified encoding converts strings from different sources to UTF-8 encoding. The categorized storage operation uses a hash table data structure, where the key of the hash table is the source category identifier. Optionally, the determination of non-text format is based on the leading byte feature code of the data. The grouping type mapping relationship is a two-level lookup table, where the first-level index is the original encoding of the group label, and the second-level index is the corresponding process classification. It is understandable that text-type data sets are organized into multiple sequential files in physical storage, with each file corresponding to a source category and entries within the file arranged in chronological order.

[0028] In practical implementation, the partitioning of process data subsets is accomplished by parsing delimiters in the process area data. These delimiters are pre-defined special byte sequences. Matching image data with process data subsets involves not only the time dimension but also spatial coordinate calculations. Optionally, the process for recognizing the format of group label content shares the same format parser as the process for recognizing the data content format. The encoding format unification process also includes removing illegal characters and standardizing spaces. It can be understood that the pre-setting process for translation mapping rules is completed during the system deployment phase. Updates to the mapping rules are achieved through a hot-loading mechanism without interrupting system operation. The source category identifiers used during categorized storage originate from the identifier information in the combined data units and the group labels in the grouping information. The generation of text-type data sets marks the transformation from raw binary data to advanced semantic information, providing directly referable text materials for the subsequent fusion and display data construction phase.

[0029] In one embodiment of the present invention, a sequence of discrete coordinate points representing the motion path is extracted from the trajectory region data, and process signal trigger points that are temporally or logically associated with the sequence of discrete coordinate points are extracted from the process region data. Each process signal trigger point is mapped to the sequence of discrete coordinate points, and the corresponding mapped coordinate point is found. The mapping process is completed by calculating the Euclidean distance between the process signal trigger point and each point in the sequence of discrete coordinate points, and the point with the smallest distance is selected as the mapped coordinate point. The formula for calculating the minimum distance is:

[0030] in: Indicates the minimum Euclidean distance. Indicates the coordinates of the process signal trigger point. Represents the first discrete coordinate point in the sequence. The coordinates of each point are used. A geometric region with a specific shape is generated centered on the mapped coordinate point and combined with the type of process signal. The boundary coordinates of the geometric region are calculated. The discrete coordinate point sequence, process signal trigger point, mapped coordinate point, and boundary coordinates of the geometric region are associated to form a complete trajectory with process annotations. The coordinate information of all trajectories with process annotations is summarized to generate a process signal trajectory coordinate set. In some embodiments, the shape of the geometric region is selected from a preset graphic library based on the process signal type. The preset graphic library includes circles, rectangles, and polygons. The boundary coordinates are calculated based on the geometric parameters of the selected graphic and the mapped coordinate points. The association operation is implemented by adding the same trajectory identifier to each data element. It can be understood that the process signal trajectory coordinate set is organized into a tree structure in memory. The root node of the tree represents a complete trajectory, and the child nodes store the discrete coordinate point sequence, process signal trigger point information, and associated geometric region parameters, respectively.

[0031] In specific implementation, a fused data structure template is created, comprising an image data storage area, a text data storage area, and a coordinate trajectory data storage area. Image data is stored in the image data storage area according to its logical order in the combined data units or grouped information. Converted text data is stored in the text data storage area according to its corresponding source category. Coordinate data from the process signal trajectory coordinate set and its associated process annotation information are stored in the coordinate trajectory data storage area. A data index relationship is established between the image data storage area, the text data storage area, and the coordinate trajectory data storage area. This data index relationship enables images, text, and coordinate trajectories to be correctly associated based on timestamps or event identifiers during display. All stored data and their data index relationships are encapsulated and the final fused display data package is generated. In some embodiments, the fused data structure template uses a custom binary format definition. The three storage areas have fixed starting offset and length fields in the data package. The data index relationship is implemented through an independent index area, which stores a mapping table from timestamps to data block pointers in each storage area. Optionally, the logical order refers to the arrangement of image data in the combined data unit or the display order determined by the grouping labels in the grouping information. The source category directly adopts the category identifier already classified in the text type data set. It can be understood that the encapsulation operation includes calculating the data checksum and adding version header information. The generated fused display data package is stored as an independent file or placed in a memory buffer for subsequent steps to read.

[0032] In practical implementation, the generation of geometric regions relies on predefined configuration parameters, which specify the dimensions of the geometric shapes corresponding to different process signal types, such as the radius of a circle or the length and width of a rectangle. Optionally, the process of establishing data index relationships parses the timestamp field common to image data, text data, and coordinate trajectory data. When the timestamp difference is within a set tolerance range, it is determined to be an associable item. It can be understood that the encapsulation format of the fused display data packet takes into account network transmission requirements, supports chunked transmission and integrity verification, and the timestamp or event identifier is globally unique within the system as an association key. The coordinate trajectory data storage area stores not only the original coordinate points but also the calculated coordinates of the geometric region vertices and the region fill style descriptor. The image data storage area stores parsed and formatted image pixel data or reference links to image files. The text data storage area stores a collection of text data with a unified encoding format, which is convenient for the display engine to render directly.

[0033] In one embodiment of the present invention, the screen resolution parameters of the display terminal are obtained, including the screen width pixel value and the screen height pixel value. The original image size of the image data storage area in the fused display data package is read, and the original coordinate range of the coordinate trajectory data storage area in the fused display data package is read. The aspect ratio of the screen display area is calculated based on the screen width pixel value and the screen height pixel value.

[0034] in: Indicates the aspect ratio of the screen display area. This represents the screen width in pixels. This represents the screen height in pixels. It is determined based on the aspect ratio of the screen display area. The scaling ratio of the image data is calculated based on the original image size, so that all images can fit the screen display area while maintaining the original aspect ratio. The scaling and translation parameters of the coordinate trajectory data are calculated based on the original coordinate range, ensuring that the trajectory graphic is displayed completely within the specified area of ​​the screen. In some embodiments, the original coordinate range is determined by traversing all coordinate points in the coordinate trajectory data storage area and finding the maximum and minimum values ​​of the horizontal and vertical coordinates. The calculation of the scaling and translation parameters ensures that the transformed trajectory graphic is displayed centered within the specified area of ​​the screen without touching the area boundaries.

[0035] In practice, the image data in the image data storage area is resampled according to the calculated scaling ratio, and the coordinate data in the coordinate trajectory data storage area is transformed according to the calculated scaling and translation parameters. The resampled and transformed data in the fused display data package is then updated. The coordinate data transformation applies a linear transformation formula to the coordinates of each point in the coordinate trajectory data storage area. and

[0036] in: Represents the transformed coordinates. Represents the original coordinates. and This represents the scaling parameter. and This represents the translation parameters. In some embodiments, the resampling process is performed by calling graphics processing library functions. The function input parameters include the original image data, the target width in pixels, and the target height in pixels. The transformation process updates the coordinate trajectory data storage area while retaining a backup of the original coordinate data. Optionally, scaling parameters... and The calculation is based on the pixel size of the specified area on the screen and the actual span of the original coordinate range, with translation parameters... and The calculation ensures that the center point of the transformed coordinate range coincides with the center point of the specified area on the screen. It can be understood that the data update process involves writing the newly generated image data block from the resampled data into the original location of the image data storage area in the fused display data package, and overwriting the original content of the coordinate trajectory data storage area with the transformed coordinate data. After the update, the version information of the fused display data package is incremented.

[0037] In practice, screen resolution parameters are obtained by calling the application programming interface provided by the operating system. The original image size is recorded in pixels in the header information of the image data storage area. The planning of the screen display area considers a fixed proportion of functional areas, such as allocating different screen spaces for image display, trajectory display, and text information bar. Refer to Table 1, which shows an exemplary comparison of key parameter values ​​when performing adaptive scaling calculations for two different screen resolutions.

[0038] Table 1: Adaptive scaling parameters for different screen resolutions

[0039] It is understood that the sizes of the image display sub-region and the trajectory display region are static values ​​pre-calculated based on screen resolution parameters and a fixed layout ratio. The image scaling ratio in the table above is a hypothetical value used to illustrate the calculation relationship. Optional, scaling parameters for the coordinate trajectory data. and They may be equal or unequal, depending on the matching relationship between the aspect ratio of the trajectory display area and the aspect ratio of the original coordinate range.

[0040] See Figure 4 This is a comparison chart of adaptive scaling parameters at different screen resolutions, illustrating the relationship between the display sub-region size and image scaling ratio of a multi-image adaptive fusion display system at different resolutions. The chart visually verifies the system's adaptive scaling capability at different resolutions; the sub-region size increases linearly with resolution, while the scaling ratio remains constant, ensuring the consistency and reliability of multi-image simultaneous display. The fixed scaling ratio (0.8) indicates that the system's computational load during image resampling is relatively stable and does not increase significantly with resolution increases, which is of significant reference value for the system's real-time performance and resource consumption control. The consistency of the sub-region ratio provides a clear benchmark for subsequent UI layout optimization, helping engineers to further optimize the space allocation of each region while ensuring display quality.

[0041] In one embodiment of the present invention, the total number of images to be displayed simultaneously is determined, and a layout of the total number of images on the screen is planned according to the aspect ratio of the screen display area. The layout includes the number of images displayed in each row and the total number of rows. Based on the layout, screen width pixel value, and screen height pixel value, the width and height pixel values ​​of the display sub-region allocated to each image are calculated. For each image, the ratio of its original image width to the width pixel value of its corresponding display sub-region is calculated as a first candidate scaling ratio, and the ratio of its original image height to the height pixel value of its corresponding display sub-region is calculated as a second candidate scaling ratio. For each image, the first and second candidate scaling ratios are compared, and the candidate scaling ratio with the smaller value is selected as the final scaling ratio for the image, ensuring that the image can be completely placed within its display sub-region without exceeding the boundary.

[0042] In practice, the target width and height pixel values ​​after image resampling are calculated based on the final scaling ratio of each image. For each image in the image data storage area, a bilinear interpolation algorithm is used to calculate the color value of each pixel in the new pixel grid based on the target width and height pixel values. The color values ​​of the new pixels calculated using the bilinear interpolation algorithm are assigned to generate a scaled image. An image filtering algorithm is used to smooth the scaled image to eliminate the jagged effect caused by resampling. The smoothed image data replaces the corresponding original image data in the image data storage area of ​​the fused display data package. The bilinear interpolation algorithm performs interpolation calculations based on the color values ​​of four adjacent pixels in the original image and their relative position weights. The image filtering algorithm uses Gaussian filtering or mean filtering, and the size of the filter kernel is dynamically adjusted according to the scaling ratio. The replacement operation updates the data content of the image data storage area in the fused display data package, and simultaneously updates the field recording the image size in the header information of the image data storage area.

[0043] In practical implementation, the layout planning considers screen space utilization efficiency and visual comfort, setting constraints on the number of images displayed in each row and the spacing between images. When calculating the width and height pixel values ​​of the display sub-region, the preset layout margins and image gaps are subtracted from the total width and height of the screen display area, and the remaining space is then evenly divided according to the number of rows and columns. Optionally, the comparison between the first and second candidate scaling ratios is implemented through conditional statements; when the two candidate scaling ratios are equal, one is selected as the final scaling ratio of the image. It can be understood that the execution of the bilinear interpolation algorithm calls specialized graphics computing library functions to improve processing speed, and image filtering is performed immediately after interpolation assignment, forming a continuous processing pipeline. The final scaling ratio of the image may vary depending on the differences between the original image size and the uniform display sub-region size; the scaling and update operations of each image in the image data storage area are performed independently.

[0044] See Figure 5 This is a dual Y-axis analysis graph of the Gaussian filtering smoothing stage, showing the relationship between the filter kernel size and image smoothness and edge preservation. As the filter kernel size increases from 3 to 19, both image smoothness (blue curve) and edge preservation (red curve) show a significant decreasing trend. This phenomenon is consistent with the basic principle of Gaussian filtering: the larger the filter kernel, the higher the degree of image blurring and the better the smoothing effect, but at the same time, more edge details are lost. The synchronous decrease of the curves indicates that in image smoothing processing, improving the smoothing effect inevitably comes at the cost of sacrificing edge details; there is a trade-off between the two. The rate of decrease in edge preservation is significantly faster than that of smoothness, indicating that edge information is more sensitive to changes in the filter kernel size. In scenarios of adaptive image fusion display, the filter kernel size needs to be dynamically adjusted according to specific requirements.

Claims

1. A multi-image adaptive fusion on-screen display method based on image analysis, characterized in that, include: The system acquires raw data recorded by the vision system, parses the raw data, and extracts image data and other types of data associated with the image data. The parsed other types of data are processed to identify and separate the trajectory region data and process region data contained in the other types of data. In the initial state, the trajectory region data and process region data are merged together. Based on the trajectory region data and the image data, a corresponding combined data unit is generated; Based on the process area data and the image data, corresponding grouping information is generated; The combined data units and the grouping information are converted into recognizable text-type data; Based on the trajectory region data and process region data, a set of process signal trajectory coordinates is generated; Based on the set of process signal trajectory coordinates, the converted text data, and the image data, fused display data is constructed; The fused display data is adaptively scaled according to the screen resolution parameters of the display terminal. Based on the fused display data after adaptive scaling, a visual simultaneous display interface containing multiple images, text, and coordinate trajectories is generated, ensuring that the visual simultaneous display interface is compatible with displays of various resolutions.

2. The method of claim 1, wherein, The parsed other types of data are processed to identify and separate the trajectory region data and process region data contained in the other types of data, including: Perform format analysis on the other types of data to identify their data structures; Based on the data structure, the data segments representing location information in the other types of data are located as the regions to be segmented. Analyze the data characteristics of the region to be segmented, including the frequency of data value changes and the distribution density of data values; Based on the frequency of change of the data values, high-frequency change data segments and low-frequency change data segments are divided. Based on the distribution density of the data values, target data segments with a distribution density higher than a preset density threshold are identified in the high-frequency changing data segments as candidates for initial trajectory region data. Based on the distribution density of the data values, target data segments with a distribution density lower than the preset density threshold are identified in the low-frequency changing data segments as candidates for initial process area data. Boundary verification is performed on the initial trajectory region data candidates and the initial process region data candidates to separate the final trajectory region data and the final process region data.

3. The method of claim 1, wherein the method further comprises: Based on the trajectory region data and the image data, a corresponding combined data unit is generated, including: The number of image data to be combined is determined based on the data length of the trajectory region data; The image data is sorted according to the data recording order of the trajectory area data to generate an ordered image data sequence; Each image data in the ordered image data sequence is associated and bound one-to-one with the sub-data at the corresponding data record position in the trajectory region data to generate multiple primary data pairs; Add identification information to each of the primary data pairs; the identification information is used to distinguish different data source categories. All primary data pairs carrying the aforementioned identification information are aggregated to form a set of combined data units. The specific combined data unit that meets the current processing requirements is extracted from the set of combined data units as the final combined data unit.

4. The method of claim 1, wherein the method further comprises: Based on the process area data and the image data, corresponding grouping information is generated, including: Analyze the content of the process area data to identify the process type identifier contained in the process area data; Based on the process type identifier, the process area data is divided into multiple process data subsets; The image data is arranged according to the order of acquisition time or spatial location; The arranged image data is matched with the multiple process data subsets, and an associated image data subsequence is assigned to each process data subset; Based on the allocation results, a grouping label is defined for each subset of process data and its associated image data subsequence; The correspondence between all process data subsets and image data subsequences with the grouping labels is stored to generate structured grouping information.

5. The method of claim 1, wherein the method further comprises: Converting the combined data units and the grouping information into recognizable text-type data includes: Identify the data content format in the combined data unit; If the data content format in the combined data unit is non-text format, then according to the preset translation mapping rules, the non-text format data content is converted into a predefined text string; Identify the format of the group tag content in the grouping information; If the group tag content format is non-text format, then according to the group type mapping relationship, the non-text format group tag content is converted into the corresponding text description; The converted text string and text description are then processed to unify their encoding formats. The text strings and text descriptions that have undergone unified encoding are integrated, categorized and stored according to their source type, and a recognizable text type data set is generated.

6. The method of claim 1, wherein the method further comprises: Based on the trajectory region data and process region data, a set of process signal trajectory coordinates is generated, including: Extract a sequence of discrete coordinate points representing the motion path from the trajectory region data; Extract process signal trigger points that are temporally or logically associated with the discrete coordinate point sequence from the process area data; Map each of the process signal trigger points to the discrete coordinate point sequence to find the corresponding mapped coordinate point; Centered on the mapped coordinate points, and combined with the type of process signal, a geometric region with a specific shape is generated; Calculate the boundary coordinates of the geometric region; The discrete coordinate point sequence, the process signal trigger point, the mapped coordinate point, and the boundary coordinates of the geometric region are associated to form a complete trajectory with process annotations; The coordinate information of all trajectories with process annotations is summarized to generate a set of process signal trajectory coordinates.

7. The method of claim 1, wherein the method further comprises: Based on the set of process signal trajectory coordinates, the converted text data, and the image data, fused display data is constructed, including: Create a fused data structure template, which includes an image data storage area, a text data storage area, and a coordinate trajectory data storage area; The image data is stored in the image data storage area according to its logical order in the combined data unit or grouping information; The converted text data is stored in the text data storage area according to its corresponding source category; The coordinate data and associated process annotation information in the process signal trajectory coordinate set are stored in the coordinate trajectory data storage area. Establish a data index relationship between the image data storage area, text data storage area, and coordinate trajectory data storage area. This data index relationship enables images, text, and coordinate trajectories to be correctly associated based on timestamps or event identifiers during display. All stored data and its index relationships are encapsulated to generate the final fused display data package.

8. The method of claim 1, wherein the method further comprises: Based on the screen resolution parameters of the display terminal, the fused display data undergoes adaptive scaling processing, including: Obtain the screen resolution parameters of the display terminal, wherein the screen resolution parameters include the screen width pixel value and the screen height pixel value; Read the original image size of the image data storage area in the fused display data packet; Read the original coordinate range of the coordinate trajectory data storage area in the fused display data packet; Calculate the aspect ratio of the screen display area based on the screen width pixel value and screen height pixel value; Based on the aspect ratio of the screen display area and the original image size, the scaling ratio of the image data is calculated so that all images can fit the screen display area while maintaining the original aspect ratio. Based on the aspect ratio of the screen display area and the original coordinate range, the scaling and translation parameters of the coordinate trajectory data are calculated so that the trajectory graphic can be completely displayed within the specified area of ​​the screen. Based on the calculated scaling ratio of the image data, the image data in the image data storage area is resampled. Based on the scaling and translation parameters of the calculated coordinate trajectory data, the coordinate data in the coordinate trajectory data storage area is transformed. Update the data in the fused display data packet after resampling and transformation.

9. The multi-image adaptive fusion display method based on image analysis according to claim 8, characterized in that, Based on the aspect ratio of the screen display area and the original image size, the scaling ratio of the image data is calculated, including: Determine the total number of images that need to be displayed on the same screen; Based on the aspect ratio of the screen display area, the layout of the total number of images on the screen is planned, and the layout includes the number of images displayed in each row and the total number of rows; Based on the layout method, the screen width pixel value, and the screen height pixel value, calculate the width pixel value and height pixel value of the display sub-region allocated to each image; For each image, the ratio of its original image width to the width pixel value of the corresponding display sub-region is calculated as the first candidate scaling ratio; For each image, the ratio of its original image height to the height pixel value of the corresponding display sub-region is calculated as the second candidate scaling ratio; For each image, compare its corresponding first candidate scaling ratio with the second candidate scaling ratio, and select the candidate scaling ratio with the smaller value as the final scaling ratio of the image to ensure that the image can be completely placed in its display sub-region without exceeding the boundary.

10. The method of claim 8, wherein the method further comprises: Based on the calculated scaling ratio of the image data, the image data in the image data storage area is resampled, including: Calculate the target width and target height pixel values ​​after image resampling based on the final scaling ratio of each image; For each image in the image data storage area, a bilinear interpolation algorithm is used to calculate the color value of each pixel in the new pixel grid based on the target width pixel value and the target height pixel value; The color values ​​of the new pixels calculated using the bilinear interpolation algorithm are assigned to generate the scaled image; Image filtering algorithms are used to smooth the scaled image to eliminate the jagged effect caused by resampling; The smoothed image data replaces the corresponding original image data in the image data storage area of ​​the fused display data packet.

Images