Intelligent image-text integrated processing system and method

Abstract

The invention discloses an intelligent image-text integrated processing system and method, and relates to the field of engineering. The system provides a vector graphic processing platform, supports the processing of texts, vector graphics, grid graphics, animations, videos and other multimedia data, and can simultaneously complete the processing and mixed typesetting of the multimedia data without calling an external program. The method comprises the following steps of: constructing an image-text large model, providing intelligent conversion and automatic typesetting capabilities for texts and multimedia data, and realizing description and typesetting of multimedia texts such as images and texts automatically generated by vector / grid graphs according to semantic association of the images and texts, or automatically generating the vector / grid graphs by inputting a natural language and intelligently generating a thematic document with both images and texts. According to the method, a large model for image-text processing and generation is constructed, interactive and intelligent image-text generation, processing and typesetting in an automatic mode are considered, and the method is particularly suitable for the fields of engineering and the like which need a large amount of image-text data and have association relations between images and texts, and replaces traditional text type office software.

Description

Technical Field

[0001] The present invention relates to the field of engineering graphics and text, and in particular to an intelligent graphics and text integrated processing system and an intelligent graphics and text integrated processing method. Background Art

[0002] Graphics and text editing software has always played a crucial role in our daily lives, providing immense convenience. Traditional graphic and text processing software includes Microsoft's Office series and China's Kingsoft WPS series. Computer-aided design (CAD) software includes AutoCAD and ZWCAD. Geographic Information System (GIS) software includes Esri's ArcGIS and SuperMap. Internet-based graphic and text processing software includes Baidu and Tencent's smart dashboards.

[0003] Traditional image and text processing software is based on text processing platforms and systems. When processing vector or raster graphics, it is usually necessary to complete it through external program calls (such as Microsoft OLE technology) or screenshots or screen captures in text processing systems (such as WPS). This method is cumbersome and requires the operation of multiple software programs. After data is shared across terminals and systems, the lack of external call programs may lead to the inability to display or process external graphics properly, becoming one of the obstacles affecting the application of image and text systems in the Internet era. In addition, screenshots and screen captures of vector graphics will lose the description of the spatiotemporal relationship of the graphics and related attribute information, which is not conducive to intelligent analysis and image-text conversion.

[0004] Artificial intelligence technology is currently booming, and traditional office and mapping software can also provide some general AI support (such as style configuration, unified typesetting, text images or videos, etc.). However, due to the lack of key attributes such as spatial relationships and professional graphic and text spatiotemporal models, especially the lack of intelligent understanding of vector graphics and graphic-text conversion functions, it is impossible to quickly and automatically generate professional spatiotemporal intelligent analysis or engineering documents about graphics in the engineering field. These are mainly completed through interactive manual operations. Professionals are inefficient when compiling highly professional reports in engineering and other fields (such as engineering papers or reports such as GIS / CAD).

[0005] Although the smart panels of general Internet companies provide simple vector editing functions such as graffiti, formulas, flow charts, and mind maps, they lack mixed text and image layout, and have no intelligent analysis and processing functions for vector engineering graphics. Summary of the Invention

[0006] In view of the above problems, the present invention proposes an intelligent image-text integrated processing system and an intelligent image-text integrated processing method to overcome the shortcomings of the existing technology.

[0007] The embodiment of the present invention provides an intelligent integrated graphics and text processing system, comprising: a vector graphics processing platform, a graphics and text large model module, a graphics and text typesetting module, and a graphics and text document generation module; The vector graphics processing platform has a built-in multimedia data processing engine including text, vector graphics, raster graphics, animation, and video, which is used to directly complete graphic editing, text processing, and mixed typesetting of multimedia data through a unified graphical interactive interface; The graphic typesetting module is used to provide manually customized graphic processing and typesetting based on the vector graphics processing platform through graphical processing and interaction; and is used to provide automatic graphic processing and typesetting based on the graphic large model module, automatically generating vector graphics or raster graphics and layout corresponding to the content based on the text content according to templates or intelligent analysis, or automatically generating text content and layout corresponding to the vector graphics or raster graphics based on the vector graphics or raster graphics, wherein the templates include: those provided by the graphic large model module or those provided by a third party; The graphic document generation module is used to automatically generate a professional document report with both pictures and text based on the graphic typesetting module, in combination with a given theme or the template, through semantic analysis and association analysis of key attribute data and GIS / CAD spatial relationship data, combined with the graphic large model module and spatiotemporal reasoning function. The GIS / CAD spatial relationship data includes topological relationship data.

[0008] Optionally, the vector graphics processing platform includes: a graphic and text rendering module, a graphic and text interactive editing module, and a graphic and text linkage module; Through the graphic rendering module, graphic interactive editing module and graphic linkage module, the vector graphics processing platform has the ability to edit vector and raster graphics, and also provides multimedia data editing and processing capabilities on the graphics processing base or platform; The graphic rendering module, graphic interactive editing module, and graphic linkage module provide a unified interactive interface for the intelligent graphic integrated processing system. Vector and raster graphic editing, text processing, and mixed typesetting can be completed without calling external programs. Seamless switching and mutual assistance eliminate system-level data exchange links, avoiding the loss of key information including attributes and topological relationships. The hybrid typesetting refers to: performing unified operations on vector graphics, raster graphics, and multimedia data, supporting tiled sequential typesetting of office software through built-in global offset attributes of graphic primitives, or supporting stacked placeholder typesetting through fixed position attributes of graphic primitives; The unified interactive interface does not require calling external programs, which means that the rendering, editing, typesetting and linkage updating of the multimedia data are respectively completed by the graphic rendering module, the graphic interactive editing module and the graphic linkage module without the help of external programs; the rendering area of ​​the unified interactive interface dynamically updates the graphic elements through preset or dynamically calculated critical values, has stepless zooming capabilities, and supports browsing and interactive editing at multiple view granularities.

[0009] Optionally, the graphic rendering module is used to load the vector and raster graphics and display the graphic elements through a memory-mapped graphics processor (GPU); The basic elements of vector graphics include: points, nodes, lines, arcs, circles, ellipses, polylines, curves, surfaces, bodies, complex bodies, as well as point symbols, line symbols, and fill symbols formed by setting symbols for common elements; The basic elements of raster graphics include: ordinary pictures and image pictures. The ordinary pictures include: PNG, BMP, JPEG, and TIFF. The image pictures are ordinary pictures stored by establishing a pyramid.

[0010] Optionally, the graphic and text interactive editing module updates the graphic and text elements in the intelligent graphic and text integrated processing system by receiving parameter calls triggered by external input; The editing capabilities of the graphic-text interactive editing module include: creation, modification and deletion of elements; The multimedia data includes text, vector graphics, raster graphics, animation, audio, and video.

[0011] Optionally, the graphic-text linkage module is used to complete the update of associated graphic elements based on built-in attributes of graphic elements, including: updating through associated call of built-in unique identifiers, updating through linkage of graphic element styles that support inheritance or combination, and active unified updating after matching one or more attributes; The update of the image-text linkage module supports automatic generation based on the image-text big model, including: analyzing and understanding the context of the image-text elements through the image-text big model, automatically generating and updating other multimedia elements including graphics or text associated with them; inputting the modification description of the local context in natural language, and automatically updating the context-related multimedia elements including graphics or text through the intention recognition and typesetting operations of the image-text big model.

[0012] Optionally, the method for constructing the large image and text model includes: S1: collecting the graphic data and the engineering document data and performing preprocessing to obtain the multimedia data and a parameterized description of the content of the multimedia data, which includes: a content outline, graphic content parameters, and typesetting parameter definitions, association relationships, and parameterized processing commands; S2: extracting features of the parameterized processing command and understanding its semantics, associating and aligning the parameterized graphic data and the parameterized text description, and establishing a mapping relationship between the two; S3: Based on the mapping relationship, a large-scale image and text model for parameterized description is constructed. The large-scale model framework is established by combining Transformer and graph neural network. The parameterized data and the data of vector graphics and raster graphics are used for pre-training. The pre-trained model is fine-tuned to obtain an optimized large-scale image and text model so that it can accurately understand and execute parameterized processing commands. The parameterized data includes: the image and text content parameters and the layout parameters. S4: Based on the optimized large-scale image and text model, it provides services for the large-scale image and text model, recognizes the user's natural language input, generates and updates the corresponding text content or multimedia content or layout including vector and raster graphics, or generates and updates the corresponding text content based on multimedia content including vector graphics and raster graphics.

[0013] Optionally, the manually customized image and text processing and typesetting is achieved by actively calling the image and text interactive editing module and passively triggering the image and text linkage module; The automatic graphic and text processing and typesetting is achieved by using the graphic and text large model module and the template as input or intelligent analysis to automatically generate graphics, images and realize mixed typesetting; the intelligent graphic and text integrated processing system is also based on the graphic and text large model and relies on preset graphics or other multimedia data to establish a new generation of knowledge base and expert library with intelligent deduction functions, construct application scenario simulation of the entire process of each profession, industry report standard verification, and expert review functions.

[0014] Optionally, the GIS spatial relationship and the CAD relationship both include: topological spatial relationship, metric spatial relationship and sequential spatial relationship; The key attribute data also includes: built-in professional attribute data that can be identified by the graphic and text model; The semantic analysis and association analysis include: automatic annotation and topology verification; The spatiotemporal reasoning functions include: historical data backtracking and trend prediction; The automatic generation of a professional analysis report with both pictures and texts refers to: analyzing the structured parameters generated by specific picture and text content in the intelligent picture and text integrated processing system through the picture and text large model, combining the professional attributes of the picture and text with the GIS spatial relationship and the CAD relationship, calling the system's built-in service to generate content and calling the picture and text typesetting module to complete typesetting, thereby generating a project special report that conforms to the preset document template format; The built-in services include: vector map services, raster map services, query services, and scalable professional services; The vector map service includes: creation, opening, content addition, deletion, modification and query; The grid map services include: cutting map, shortest path, and disaster avoidance route; The query service refers to efficient query based on spatial index of spatial elements; The scalable professional services include: surveying and mapping, ecological environment, resource management thematic map generation, pie chart generation, bar chart generation, circuit diagram generation, building and mechanical equipment design drawing generation, mining thematic map generation, road and bridge thematic map generation, water conservancy thematic map generation, as well as customized professional vector graphics processing service components written by personnel according to their own engineering professional needs.

[0015] Optionally, the graphic and text model module is also used to record the user's writing style and continuously iterate and upgrade it to ensure that the style of the generated professional report is consistent with the user's current writing style.

[0016] An embodiment of the present invention provides an intelligent image-text integrated processing method, which is applied to any of the above-mentioned intelligent image-text integrated processing systems, and includes: Through the vector graphics processing platform in the intelligent integrated graphics and text processing system, basic text, vector graphics, raster graphics and other multimedia data editing and processing are provided, providing manual customized graphics and text processing and typesetting; Through the image and text model generation service in the intelligent image and text integrated processing system, for scenes with graphical presentations, a graphics drawing or visualization script is automatically generated in combination with the context, and the intelligent image and text integrated processing system executes the script to generate the corresponding vector graphics or raster graphics; or the vector graphics or raster graphics are combined to automatically generate the corresponding text content; Through the typesetting service of the large graphic and text model in the intelligent graphic and text integrated processing system, combined with the specified theme or the template, the layout position or adjustment position script of the document content is automatically generated, and the intelligent graphic and text integrated processing system completes the typesetting after execution, forming a professional document report with both graphics and text in the user's writing style.

[0017] The intelligent image and text integrated processing system proposed in the present invention includes: The present invention uses a professional vector graphics processing platform as its base, and provides a unified mixed arrangement of multimedia data such as text, pictures, and tables, which can truly complete the unified processing of text and graphics on a single platform. At the same time, combined with the intelligent processing requirements for text and graphics processing, based on a large text and graphics model with professional analysis capabilities, the conversion between text and graphics and the generation and printing of special documents can be easily completed within the intelligent text and graphics integrated processing system, providing intelligent support for applications such as engineering text and graphics processing and report writing. The proposed intelligent text and graphics integrated processing system supports intelligent automatic text and graphics generation and typesetting, and is particularly suitable for engineering and other fields that require a large amount of text and graphics data and have a correlation between text and graphics. It replaces traditional text-based office software and has broad application prospects and high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings: Figure 1 This is an architectural diagram of an intelligent integrated image and text processing system proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention, are only part of the embodiments of the present invention, not all of the embodiments, and are not intended to limit the present invention.

[0020] Traditional graphic processing software typically requires processing vector or raster graphics within a text processing system (e.g., WPS) by calling external programs (e.g., Microsoft OLE technology) or taking screenshots. This method is cumbersome, requiring the use of multiple software programs. Furthermore, after data is shared across devices and systems, the lack of external call programs can lead to problems with displaying or processing external graphics. This has become a major obstacle to the application of graphic systems in the internet age.

[0021] For example, a user needs to create a power supply design report, using AutoCAD to create the power supply diagram and WPS to create the final report. The content of the power supply diagram needs to be inserted in multiple places in the report (the data exists as pictures or OLE objects), and the corresponding content must be supplemented with text descriptions or analysis reports. These descriptions or reports are usually interrelated with the content of the power supply diagram. If the power supply diagram changes, the user can only take a screenshot and copy and paste it into WPS. If the properties of the power supply diagram content change (for example, the load of a certain device changes), the corresponding text description or analysis report in the report must also be modified, making the overall process more cumbersome.

[0022] To address the above problems, the inventors creatively propose an intelligent image-text integrated processing system and an intelligent image-text integrated processing method of the present invention. The technical solution of the present invention is explained and illustrated in detail below.

[0023] The present invention proposes an intelligent image and text integrated processing system, referring to Figure 1 The schematic diagram shown includes: a vector graphics processing platform, a graphic and text model module, a graphic and text typesetting module, and a graphic and text document generation module; The vector graphics processing platform features a built-in multimedia data processing engine, including text, vector graphics, raster graphics, animation, and video. This platform uses a unified graphical interface to directly perform graphic editing, text processing, and mixed typesetting of multimedia data. The vector graphics processing platform can process not only data from platforms like GIS and CAD (including GIS graphics and CAD drawings), but also multimedia data, including text, vector graphics, raster graphics, animation, and video.

[0024] The graphic and text large model module is used to use general graphic and text resources and engineering field document resources as data sets, combine multimedia data including vector graphics, parametric descriptions of multimedia data content, and the typesetting associations between them, to build a graphic and text large model with the ability to intelligently understand, process and typeset multimedia data.

[0025] The graphic and text typesetting module is used to provide manually customized graphic and text processing and typesetting through graphical processing and interaction based on the vector graphics processing platform; and is used to provide automatic graphic and text processing and typesetting based on the graphic and text large model module, automatically generating vector graphics or raster graphics and layout corresponding to the content based on the text content according to templates or intelligent analysis, or automatically generating text content and layout corresponding to the vector graphics or raster graphics based on the vector graphics or raster graphics. The template includes: provided by the graphic and text large model module or provided by a third party.

[0026] The graphic document generation module is used to automatically generate professional document reports with both pictures and texts based on the graphic typesetting module, combined with a given theme or template, through semantic analysis and association analysis of key attribute data and GIS spatial relationship data, CAD spatial relationship data, combined with the graphic large model module and time-space reasoning function. GIS spatial relationship data and CAD spatial relationship data both include topological relationship data.

[0027] In one embodiment of the present invention, the vector graphics processing platform includes: a graphic rendering module, a graphic-text interactive editing module, and a graphic-text linkage module; these modules support and implement the functions of the vector graphics processing platform.

[0028] Through the graphic rendering module, graphic interactive editing module and graphic linkage module, the vector graphics processing platform has the ability to edit vector and raster graphics, and also provides multimedia data layout and processing capabilities on the graphics processing base.

[0029] Through the graphic rendering module, graphic interactive editing module and graphic linkage module, a unified interactive interface is provided for the intelligent graphic integrated processing system. Vector and raster graphic editing, text processing and mixed typesetting can be completed without calling external programs. Seamless switching and mutual assistance can be achieved, eliminating system-level data exchange links and avoiding the loss of shielded information including attributes and topological relationships.

[0030] Among them, the so-called mixed typesetting refers to: unified operation of vector and raster graphics and multimedia data, supporting tiled sequential typesetting of office software through the built-in global offset properties of the graphics elements, or supporting stacked placeholder typesetting through the fixed position properties of the graphics elements.

[0031] The so-called unified interactive interface, which does not require the calling of external programs, means that the rendering, editing, typesetting, and linkage update of multimedia data, including vector and raster graphics, are completed by the graphic rendering module, the graphic interactive editing module, and the graphic linkage module respectively, without the aid of external programs; the rendering area of ​​the unified interactive interface dynamically updates graphic elements through preset or dynamically calculated critical values, has stepless scaling capabilities, and supports browsing and interactive editing at multiple view granularities.

[0032] In one embodiment of the present invention, a graphic rendering module is used to load vector and raster graphics and display graphic elements through a memory-mapped graphics processor (GPU). The basic elements of vector graphics include: points, nodes, lines, arcs, circles, ellipses, polylines, curves, surfaces, bodies, and complexes, as well as point symbols, line symbols, and fill symbols formed by setting symbols for common elements. The basic elements of raster graphics include: common pictures and image pictures. The common pictures include: PNG, BMP, JPEG, and TIFF. The image pictures are common pictures stored by establishing a pyramid.

[0033] Preferably, the graphic and text interactive editing module can update the graphic and text elements within the intelligent graphic and text integrated processing system by receiving parameter calls triggered by external input; the editing capabilities of the graphic and text interactive editing module include: creation, modification and deletion of elements; among them, multimedia data includes: text, vector graphics, raster graphics, animation, audio, and video.

[0034] In one embodiment of the present invention, a graphic-text linkage module is used to complete the update of associated graphic elements based on the built-in attributes of graphic-text elements, including: updating through associated calls with built-in unique identifiers, updating through graphic element style linkage that supports inheritance or combination, and active unified updating after matching one or more attributes.

[0035] The update of the graphic-text linkage module supports automatic generation based on the graphic-text big model, including: analyzing and understanding the context of graphic and text elements through the graphic-text big model, automatically generating and updating other multimedia elements including graphics or text associated with them; inputting modification descriptions of local context in natural language, and automatically updating context-related multimedia elements including graphics or text through the intention recognition and typesetting operations of the graphic-text big model.

[0036] In one embodiment of the present invention, a method for constructing a large image and text model includes: S1: Collect and pre-process graphic and text data and engineering document data to obtain multimedia data including vector graphics, raster graphics, etc., and parameterized description of the multimedia data content, including: content outline, graphic content parameters, and typesetting parameter definitions, association relationships, and parameterized processing commands; S2: Extract the features of the parameterized processing commands and understand their semantics, associate and align the parameterized graphic data and parameterized text descriptions, and establish a mapping relationship between the two; S3: Based on the mapping relationship, a large-scale image and text model for parameterized description is constructed. The large-scale model framework is established by combining Transformer and graph neural network. The parameterized data and the data of vector graphics and raster graphics are used for pre-training. The pre-trained model is fine-tuned to obtain an optimized large-scale image and text model, which accurately understands and executes parameterized processing commands. The parameterized data includes: the image and text content parameters and layout parameters in step S1; S4: Based on the optimized large-scale image and text model, it provides services for the large-scale image and text model, recognizes the user's natural language input, generates and updates the corresponding text content or multimedia content or layout including vector and raster graphics, or generates and updates the corresponding text content based on multimedia content including vector graphics and raster graphics.

[0037] In one embodiment of the present invention, the manually customized image and text processing and typesetting is achieved by actively calling the image and text interactive editing module and passively triggering the image and text linkage module. For example: When the user creates a ventilation system diagram, the graphics and text interactive editing module in the vector graphics processing platform creates ventilation node objects, ventilation branch objects, and text or multi-line text objects. Ventilation nodes and ventilation branches are professional ventilation objects, which contain ventilation professional attribute information, associated object information, and control point information. These objects are created through mouse interaction. After the user selects a professional object, the control point information of the professional object is displayed through the rendering module, and the attribute information of the professional object is displayed through the interactive editing module property dialog box. The user can move the position or perform other geometric transformations of the ventilation professional object by dragging the control point as needed, and modify the attributes of the professional object by setting the attribute value in the attribute box. When the attributes are modified, the geometric display of the object can also change. The user double-clicks the text or multi-line text to directly edit it in the WYSIWYG editor. The graphics and text interactive editing module completes the integrated editing of vector graphics and text without calling any external programs. When the position of a ventilation node changes, the graphic-text linkage module searches for its associated ventilation branch object in the loaded ID-ID mapping table and calls the specialized ventilation algorithm through a virtual interface to update the position of the ventilation branch object. Ventilation node number information can be stored as fields in text or multi-line text. When the ventilation system diagram is updated and the node number changes, the graphic-text linkage module searches the loaded field-ID mapping to find the text or multi-line text and aligns the field content for update. The graphic-text association module passively triggers the update of vector data and text position and content, thereby completing typesetting.

[0038] Automatic graphic and text processing and typesetting utilizes the graphic and text macromodel module, using templates (provided by the macromodel or third-party sources) as input or intelligent analysis to automatically generate graphics and images, as well as implement mixed typesetting. The intelligent integrated graphic and text processing system, based on the graphic and text macromodel and relying on pre-set graphics or other multimedia data, establishes a next-generation knowledge base and expert database with intelligent deduction capabilities. This system also features full-process application scenario simulation for various disciplines, industry report standard verification, and expert review.

[0039] The GIS and CAD spatial relationships involved in the above description include topological spatial relationships, metric spatial relationships, and ordinal spatial relationships. In addition to spatial relationships, key attribute data also includes built-in professional attribute data that can be recognized by the large-scale graphic model.

[0040] Semantic analysis and association analysis include automatic annotation and topology verification. Spatiotemporal reasoning functions include historical data backtracking and trend prediction.

[0041] Automatically generating professional analysis reports with both text and images means: analyzing the structured parameters generated by specific graphic content in the intelligent graphic and text integrated processing system through a large graphic and text model, combining the spatial relationship and professional attributes of the graphic and text, calling the system's built-in services, and generating a professional analysis report that conforms to the preset document template format.

[0042] Built-in services include vector map services, raster map services, query services, and scalable professional services. Vector map services include creation, opening, content addition, deletion, modification, and querying; raster map services include map slicing, shortest path, and disaster avoidance routes; and query services enable efficient spatial indexing queries based on spatial elements. Scalable professional services include surveying and mapping, ecological environment, resource management thematic map generation, pie chart generation, bar chart generation, circuit diagram generation, building and mechanical equipment design drawings, mining thematic maps, road and bridge thematic maps, and water conservancy thematic maps, as well as customized professional vector graphics processing service components developed by engineers based on their own engineering expertise.

[0043] In one embodiment of the present invention, the graphic and text model module is also used to record the user's writing style and continuously iterate and upgrade it to ensure that the style of the generated professional report is consistent with the user's current writing style.

[0044] Based on the above-mentioned intelligent image-text integrated processing system, the present invention further provides an intelligent image-text integrated processing method, which is applied to any of the above-mentioned intelligent image-text integrated processing systems, and includes: Step T1: Using the vector graphics processing platform in the intelligent integrated graphics and text processing system, basic text, vector graphics, raster graphics, and other multimedia data are edited and processed, providing manual customized graphics and text processing and typesetting; Step T2: Using the large-scale graphic model generation service in the intelligent graphic-text integrated processing system, for scenes with graphical displays, a graphic drawing or visualization script is automatically generated based on the context. The intelligent graphic-text integrated processing system then executes the generated vector or raster graphics; or, alternatively, the corresponding text content is automatically generated based on the vector or raster graphics. Step T3: The typesetting service of the large graphic and text model in the intelligent graphic and text integrated processing system, combined with the specified theme or template, automatically generates the layout position or adjustment position script of the document content, which is executed by the intelligent graphic and text integrated processing system to complete the typesetting and form a professional document report with both graphics and text in the user's writing style.

[0045] In order to better understand the intelligent integrated graphics and text processing system and method proposed in the present invention, combined with the aforementioned modules, a specific example is used to describe how the intelligent integrated graphics and text processing system specifically typesets and how it automatically generates a professional analysis report.

[0046] For example, a user may use the intelligent integrated graphics and text processing system provided by the present invention to typeset and produce a professional power supply design report.

[0047] Manual typesetting: (1) The user calls an existing power supply system diagram into the intelligent integrated graphics and text processing system or generates a power supply system diagram immediately. The system diagram usually includes: ① Power supply equipment and power supply cable electromechanical professional objects. This type of object is displayed in the system as a vector graphic complex, which is composed of points, line segments, text, etc. This type of object contains its professional attributes (such as voltage, current, etc.); This type of object contains spatial association information, for example, the power supply equipment contains the cable data information associated with it, and the power supply cable also stores the associated power supply equipment information; This type of object contains control point information, for example, the power supply equipment contains the center point control point for translation; ② Legend information, which is displayed as raster graphics in the system. The vector graphics processing platform unifies the vector graphics, raster graphics, and text in the system diagram into a single system, rendering them through the graphics rendering module without relying on any third-party software. Vector graphics are rendered in the GPU using point elements, line elements, and triangle elements, raster graphics are rendered in the GPU using texture elements, text is rendered in the GPU using triangle elements or texture elements, and control points are rendered in the GPU using triangle elements.

[0048] (2) The user clicks on the specified position of the system diagram to insert text or multi-line text in the vector graphic to form the text content of the report. Both text and multi-line text contain text format information, and multi-line text also contains paragraph information for setting paragraph format; the text or multi-line text interactive editor has a WYSIWYG visual effect; after the text or text is input, it can be re-entered into the editing state by double-clicking to modify the content; after the vector graphic in the system diagram is selected, the rendering module will display its control point information, and the user can edit the vector graphic in whole or in part by selecting and dragging. For example, after the center control point of the power supply equipment is selected, the user can complete the translation of the entire power supply equipment object by dragging, and after selecting a control point of the ordinary polyline in the graphic, the user can move the corresponding point of the control point by dragging; after the professional object in the intelligent graphic and text integrated processing system diagram is selected, the interactive editing module displays the attribute information of the object through the attribute dialog box, and the user can complete the attribute modification by editing the attribute content in the attribute dialog box. For example, the user modifies the length of the power supply cable object through the attribute dialog box, the attribute of the object is modified, and the cable length in the corresponding complex of the object also changes accordingly, and finally notifies the rendering module to refresh and display the new text. Therefore, the vector graphics processing platform completes the mixed editing of vector graphics and text through a unified graphic and text interactive editing module.

[0049] (3) In the power supply system diagram, when the user moves the position of a power supply device, the associated nodes of the power supply cable associated with it need to change accordingly with the power supply device. The image-text association module stores the association relationship between objects in the form of ID-ID mapping. This association relationship is formed when the power supply device and the power supply cable are created and associated when the system diagram is drawn or generated. When the user moves the power supply device, the image-text association module reads the ID of the object moved by the user, queries the ID of the power supply cable associated with it through mapping, and then calls the professional update algorithm through the virtual interface to complete the update of the power supply cable object. In the power supply system diagram, the user inserts the length of a power supply cable in a paragraph in the form of a multi-line text field. The image-text association module stores the association relationship between the field and the object in the form of field type-ID mapping. This association relationship is formed when the field is created. When the user sets the length of the power supply cable, the image-text association module traverses the field type in the system diagram, matches the ID of the power supply cable, and updates the field value in the multi-line text. Therefore, the vector graphics processing platform completes the update of the associated graphic element based on the built-in attributes of the graphic element through the image-text association module.

[0050] (4) When the user needs to fine-tune the power supply cables in the power supply system diagram, the user can use stepless zooming to adjust the view to a scale that is convenient for interactive editing.

[0051] Automatic typesetting: (1) Collecting general power industry graphic and document resources as a dataset, we obtain power supply-related graphic and multimedia data and their layout relationships. We then parameterize the graphic and multimedia data, for example, parameterizing the power supply equipment and power cable information in the graphic data using JSON strings. Based on this, we train and construct a large-scale graphic and text model capable of intelligently understanding, processing, and layout graphics, text, and multimedia.

[0052] (2) The user transmits the text description information of the power supply system diagram to the graphic and text large model module in the graphic and text integrated processing system in a natural language. The graphic and text large model module calculates the Json string description information of the power supply system diagram based on the training results. The graphic and text media processing engine creates objects, adds attributes, and establishes related information based on the professional objects or text, multi-line text, and raster images in the Json string. Professional objects include electromechanical equipment objects and electromechanical cable objects. Text or multi-line text contains the report body content. Raster images include legend information in the system diagram. The graphic and text typesetting module automatically generates vector graphics or raster graphics corresponding to the text content, and further automatically typesets all created objects based on the typesetting information in the Json string. Users can upload a customized power supply design report template as needed, and the graphic and text typesetting module can also generate and typeset content based on this template.

[0053] Automatically generate power supply design report: In automatic typesetting, users can use natural language as parameters to call the graphic model module to obtain vector graphics or raster graphics and complete typesetting. In order to further generate a professional power supply design report with both graphics and text, the following operations are required: (1) Use the attribute information generated by automatic typesetting to perform measurement analysis. For example, use the built-in electromechanical service algorithm to count the setting value information of all high and low voltage switches, obtain the tree structure diagram of the power supply model, and return it to the system in SVG format; at the same time, the electromechanical service algorithm analyzes the coordination relationship of each switch, makes a prediction on "over-tripping", and makes a decision on the value of the switch setting value, and returns it to the system in multiple lines of text; (2) Use the associated information generated by automatic typesetting to perform topological analysis of power supply equipment and cables. Use the associated information of all power supply equipment and power cables as parameters to call the built-in electromechanical service algorithm to calculate the topological structure information of the power supply system diagram and return it to the system in multiple lines of text; (3) In the power supply design report template or graphic model generation template uploaded by the user, the preset fields are replaced with the attribute information generated by automatic typesetting; and the various formulas in the template are calculated by calling the built-in electromechanical service algorithm to obtain the results. The results are converted into SVG through the built-in cutting service and returned to the system; (4) In the power supply design report template or graphic model generation template uploaded by the user, for the preset graphic content, the built-in local drawing service is called to generate SVG or Json format and returned to the system for filling; the professional analysis description of the graphic content is obtained by calling the built-in electromechanical service algorithm and returned to the system in multiple lines of text; (5) After the user obtains the automatically generated power supply design report, he or she can also make manual fine-tuning.

[0054] (6) The generated power supply design report is circulated into the large graphic and text model for iterative training and upgrading, so that the graphic and text integrated processing system gradually has the ability to generate reports in the user's writing style.

[0055] The intelligent integrated graphics and text processing system proposed in the present invention uses a professional graphics processing platform as its base, provides a unified mixed arrangement of multimedia data such as text, pictures, and tables, and can truly complete the unified processing of graphics and text on a single platform. At the same time, combined with the intelligent processing requirements of graphics and text processing, based on a large graphics and text model with professional analysis capabilities, the conversion between graphics and text and the generation and printing of special documents can be conveniently completed within the intelligent integrated graphics and text processing system, providing intelligent support for applications such as engineering graphics and text processing and report writing. The proposed intelligent integrated graphics and text processing system supports intelligent automatic graphics and text generation and typesetting, and is particularly suitable for engineering and other fields that require a large amount of graphics and text materials and have correlations between graphics and texts. It replaces traditional text-based office software and has broad application prospects and high practicality.

[0056] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0057] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0058] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. An intelligent image and text integrated processing system, characterized in that: include: Vector graphics processing platform, graphic and text large model module, graphic and text typesetting module, graphic and text document generation module; The vector graphics processing platform has a built-in multimedia data processing engine including text, vector graphics, raster graphics, animation, and video, which is used to directly complete graphic editing, text processing, and mixed typesetting of multimedia data through a unified graphical interactive interface; The graphic and text large model module is used to use general graphic and text resources and engineering document resources as data sets, combined with multimedia data including text, vector graphics, raster graphics, animation, video, parameterized descriptions of multimedia data content, and the typesetting associations between them, to construct a graphic and text large model with the ability to intelligently understand, process, and typeset multimedia data; The graphic typesetting module is used to provide manually customized graphic processing and typesetting through graphical processing and interaction based on the vector graphics processing platform; and, for providing automatic text and image processing and typesetting based on the text and image macromodel module, automatically generating corresponding vector graphics or raster graphics and layout based on text content, or automatically generating corresponding text content and layout based on vector graphics or raster graphics, based on templates or intelligent analysis, wherein the templates include: those provided by the text and image macromodel module or those provided by a third party; The graphic document generation module is used to automatically generate a professional document report with both pictures and text based on the graphic typesetting module, in combination with a given theme or the template, through semantic analysis and association analysis of key attribute data and GIS spatial relationship data, CAD spatial relationship data, combined with the graphic large model module and time-space reasoning function. The GIS spatial relationship data and the CAD spatial relationship data both include topological relationship data.

2. The intelligent image and text integrated processing system according to claim 1, characterized in that: The vector graphics processing platform includes: a graphic rendering module, a graphic interactive editing module and a graphic linkage module; Through the graphic rendering module, graphic interactive editing module and graphic linkage module, the vector graphics processing platform has the ability to edit vector graphics and raster graphics, and simultaneously provides multimedia data editing and processing capabilities on the graphics processing base or platform; The graphic rendering module, graphic interactive editing module, and graphic linkage module provide a unified interactive interface for the intelligent graphic integrated processing system. Vector and raster graphic editing, text processing, and mixed typesetting can be completed without calling external programs. Seamless switching and mutual assistance eliminate system-level data exchange links, avoiding the loss of key information including attributes and topological relationships. The hybrid typesetting refers to: performing unified operations on vector graphics, raster graphics, and multimedia data, supporting tiled sequential typesetting of office software through built-in global offset attributes of graphic primitives, or supporting stacked placeholder typesetting through fixed position attributes of graphic primitives; The unified interactive interface does not require calling external programs, which means that the rendering, editing, typesetting and linkage updating of the multimedia data are respectively completed by the graphic rendering module, the graphic interactive editing module and the graphic linkage module without the help of external programs; the rendering area of ​​the unified interactive interface dynamically updates the graphic elements through preset or dynamically calculated critical values, has stepless zooming capabilities, and supports browsing and interactive editing at multiple view granularities.

3. The intelligent image-text integrated processing system according to claim 2, characterized in that: The graphic rendering module is used to load the vector and raster graphics and display the graphic elements through a memory-mapped graphics processor (GPU); The basic elements of vector graphics include: points, nodes, lines, arcs, circles, ellipses, polylines, curves, surfaces, bodies, and complex bodies, as well as point symbols, line symbols, and fill symbols formed by setting symbols for the basic elements. The basic elements of raster graphics include: ordinary pictures and image pictures. The ordinary pictures include: PNG, BMP, JPEG, and TIFF. The image pictures are ordinary pictures stored by establishing a pyramid.

4. The intelligent image-text integrated processing system according to claim 2, characterized in that: The graphic and text interactive editing module updates the graphic and text elements in the intelligent graphic and text integrated processing system by receiving parameter calls triggered by external input; The editing capabilities of the graphic-text interactive editing module include: creation, modification and deletion of elements; The multimedia data includes text, vector graphics, raster graphics, animation, audio, and video.

5. The intelligent image-text integrated processing system according to claim 2, characterized in that: The graphic-text linkage module is used to update associated graphic elements based on built-in attributes of graphic elements, including: updating through associated calls with built-in unique identifiers, updating through graphic element styles that support inheritance or combination, and actively updating after matching one or more attributes; The update of the image-text linkage module supports automatic generation based on the image-text big model, including: analyzing and understanding the context of the image-text elements through the image-text big model, automatically generating and updating other multimedia elements including graphics or text associated with them; inputting the modification description of the local context in natural language, and automatically updating the context-related multimedia elements including graphics or text through the intention recognition and typesetting operations of the image-text big model.

6. The intelligent image and text integrated processing system according to claim 1, characterized in that: The method for constructing the large graphic and text model includes: S1: collecting the graphic data and the engineering document data and performing preprocessing to obtain the multimedia data and a parameterized description of the content of the multimedia data, which includes: a content outline, graphic content parameters, and typesetting parameter definitions, association relationships, and parameterized processing commands; S2: extracting features of the parameterized processing command and understanding its semantics, associating and aligning the parameterized graphic data and the parameterized text description, and establishing a mapping relationship between the two; S3: Based on the mapping relationship, a large-scale image and text model for parameterized description is constructed. The large-scale model framework is established by combining Transformer and graph neural network. The parameterized data and the data of vector graphics and raster graphics are used for pre-training. The pre-trained model is fine-tuned to obtain an optimized large-scale image and text model so that it can accurately understand and execute parameterized processing commands. The parameterized data includes: the image and text content parameters and the layout parameters. S4: Based on the optimized large-scale image and text model, it provides services for the large-scale image and text model, recognizes the user's natural language input, generates and updates the corresponding text content or multimedia content or layout including vector and raster graphics, or generates and updates the corresponding text content based on multimedia content including vector graphics and raster graphics.

7. The intelligent image and text integrated processing system according to claim 2, characterized in that: The manual customization of image and text processing and typesetting is achieved by actively calling the image and text interactive editing module and passively triggering the image and text linkage module; The automatic graphic and text processing and typesetting is achieved by using the graphic and text large model module and the template as input or intelligent analysis to automatically generate graphics, images and realize mixed typesetting; the intelligent graphic and text integrated processing system is also based on the graphic and text large model and relies on preset graphics or other multimedia data to establish a new generation of knowledge base and expert library with intelligent deduction functions, construct application scenario simulation of the entire process of each profession, industry report standard verification, and expert review functions.

8. The intelligent image-text integrated processing system according to claim 1, characterized in that: The GIS spatial relationship and the CAD spatial relationship both include: topological spatial relationship, metric spatial relationship and sequential spatial relationship; The key attribute data also includes: built-in professional attribute data that can be identified by the graphic and text model; The semantic analysis and association analysis include: automatic annotation and topology verification; The spatiotemporal reasoning functions include: historical data backtracking and trend prediction; The automatic generation of a professional analysis report with both pictures and texts refers to: analyzing the structured parameters generated by specific picture and text content in the intelligent picture and text integrated processing system through the picture and text large model, combining the professional attributes of the picture and text and the relationship between GIS space and CAD space, calling the system's built-in service to generate content and calling the picture and text typesetting module to complete typesetting, thereby generating a project special report that conforms to the preset document template format; The built-in services include: vector map services, raster map services, query services, and scalable professional services; The vector map service includes: creation, opening, content addition, deletion, modification and query; The grid map services include: cutting map, shortest path, and disaster avoidance route; The query service refers to efficient query based on spatial index of spatial elements; The scalable professional services include: surveying and mapping, ecological environment, resource management thematic map generation, pie chart generation, bar chart generation, circuit diagram generation, building and mechanical equipment design drawing generation, mining thematic map generation, road and bridge thematic map generation, water conservancy thematic map generation, as well as customized professional vector graphics processing service components written by personnel according to their own engineering professional needs.

9. The intelligent image-text integrated processing system according to claim 1, characterized in that: The graphic and text model module is also used to record the user's writing style and continuously iterate and upgrade it to ensure that the style of the generated professional report is consistent with the user's current writing style.

10. An intelligent image and text integrated processing method, characterized in that: The intelligent image-text integrated processing method is applied to the intelligent image-text integrated processing system according to any one of claims 1 to 9, and comprises: Through the vector graphics processing platform in the intelligent integrated graphics and text processing system, basic text, vector graphics, raster graphics and other multimedia data editing and processing are provided, providing manual customized graphics and text processing and typesetting; Through the image and text model generation service in the intelligent image and text integrated processing system, for scenes with graphical presentations, a graphics drawing or visualization script is automatically generated in combination with the context, and the intelligent image and text integrated processing system executes the script to generate the corresponding vector graphics or raster graphics; or the vector graphics or raster graphics are combined to automatically generate the corresponding text content; Through the typesetting service of the large graphic and text model in the intelligent graphic and text integrated processing system, combined with the specified theme or the template, the layout position or adjustment position script of the document content is automatically generated, and the intelligent graphic and text integrated processing system completes the typesetting after execution, forming a professional document report with both graphics and text in the user's writing style.

Images