Specification generation method and device, electronic equipment, storage medium and computer program product
By combining image and text information in a multimodal generation method and using a trained product instruction manual generation model, the problem of low efficiency in writing medical device instruction manuals in existing technologies is solved. The generated instruction manuals are closer to the quality of human writing, thus improving writing efficiency and accuracy.
Patent Information
- Application Number
- CN202411129832.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2026-03-06
AI Technical Summary
In existing technologies, the generated product manuals cannot reach the level of those written by humans, especially in terms of the inaccurate description of information such as the structure, materials, and scope of application of medical devices, resulting in low writing efficiency.
By acquiring image and text information of the target product, and using a trained product manual generation model, combined with a product information database and R&D process documents, a multi-dimensional and related manual is generated, including a combination of images and text, for multimodal generation.
The generated instruction manuals are closer to the level of human writing, saving writing time and improving writing efficiency, especially in the field of medical devices where they are more professional and accurate.
Smart Images
Figure CN121615604A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and specifically to a method, apparatus, electronic device, storage medium, and computer program product for generating a specification. Background Technology
[0002] In related technologies, for product manual writing, big data can be used to retrieve relevant manuals, extract and generate corresponding manual titles, and then ChatGPT (Chat Generative Pre-trained Transformer) can be used to iteratively generate the corresponding manual content.
[0003] This method of writing instruction manuals relies on the generation capabilities of a large model to generate an instruction manual. However, the model does not know the structure, materials, and applicable scope of the current medical device, so the generated instruction manual cannot reach the level of a manually written instruction manual. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, electronic device, storage medium, and computer program product for generating instruction manuals, which can make the generated instruction manuals closer to the level of human-written instruction manuals, thereby further saving instruction manual writing time and improving the efficiency of instruction manual writing.
[0005] According to a first aspect of the embodiments of this application, a method for generating a specification is provided, comprising:
[0006] Obtain the first prompt information; the first prompt information includes images and text about the target product;
[0007] The first prompt information is input into the trained product manual generation model, so that the product manual generation model outputs the manual for the target product; some text in the manual is generated based on the image.
[0008] In one embodiment, the product manual includes at least one chapter, and each chapter includes a chapter title and chapter content;
[0009] Before obtaining the first prompt information, the process also includes:
[0010] Obtain query information input by the user; wherein, the query information includes first identification information, first keyword, and second keyword; the first identification information is the identification information of the target product; the first keyword includes second identification information of the target product in each development stage of the product development process, the second identification information is determined based on the chapter title and the correspondence between the chapter title and the development stage identifier, the chapter title is determined based on the product manual writing specification information; the second keyword is determined based on the necessary content in the chapter content;
[0011] Based on the first identification information, a query is performed in the product information database to determine the target product file; the product information database includes the correspondence between product identifications and product files; the target product file includes R&D process documents recorded during the product development process of the target product.
[0012] Based on the first keyword and the second keyword, a query is performed in the target product file to obtain target information that matches the query information; the target information is used in conjunction with the product information of the target product and the product instruction manual writing specifications to obtain the first prompt information; the R&D process document includes the phase introduction content of each R&D stage and the correspondence between the R&D stage identifier and the phase introduction content, and the phase introduction content includes the necessary content.
[0013] In one embodiment, the product manual writing specification information includes a general title and at least one of the aforementioned chapters, wherein the general title includes the product name;
[0014] The first prompt information includes the product name information of the target product; the product information of the target product includes the product name information.
[0015] In one embodiment, the target product is a medical device; the product information also includes medical device type information and scope of application information, the medical device type information includes first type information and second type information, the first type information is one of active medical devices and passive medical devices, and the second type information is one of Class I medical devices, Class II medical devices and Class III medical devices.
[0016] In one implementation, at least one of the chapters includes a first type of chapter, the chapter title of which is generated based on the text about the target product in the first prompt information; the first prompt information includes a first sub-prompt information, which is a prompt information written based on the product information, the product instruction manual writing specifications, and the R&D process documents for the chapter title of the first type of chapter.
[0017] In one implementation, at least one of the chapters includes a second type of chapter, the chapter title of which is generated based on the image of the target product in the first prompt information; the first prompt information includes a second sub-prompt information, which is a prompt information written for the chapter title of the second type of chapter based on the product instruction manual writing specification information, the image, and the chapter title corresponding to the image.
[0018] In one implementation, before inputting the first prompt information into the trained product manual generation model, the method further includes:
[0019] The first training dataset is used to incrementally train the pre-trained large language model to obtain the first intermediate model; the first training dataset includes R&D process documents and reference product manuals recorded during product development; the first training dataset includes multiple first training data, the first training data is text data, and the first training data is the R&D process documents or the reference product manual.
[0020] The first intermediate model is subjected to supervised fine-tuning using a second training dataset to obtain a second intermediate model; the second training dataset includes second training data for multiple specified tasks; the specified tasks are supervised fine-tuning tasks for the content of the instruction manual of the target product;
[0021] The second intermediate model is trained using a third training dataset to obtain the trained product manual generation model; wherein, the third training dataset includes multiple third training data, which are training data for the image and text alignment task.
[0022] In one embodiment, the target product is a medical device; the product instruction manual includes at least one chapter, and each chapter includes a chapter title and chapter content;
[0023] The R&D process documents include clinical trial data and testing data of the medical device; the reference product instruction manual is the medical device instruction manual, including reference chapter titles and reference chapter content;
[0024] The second training data consists of key-value pairs between the reference chapter titles and the reference chapter content;
[0025] The third training data includes images of medical device components and corresponding text descriptions. The text descriptions include content descriptions of the medical device component images and questions based on the medical device component images. The medical device component images and corresponding text descriptions are stored in key-value pairs. The product manual generation model includes a mapping relationship from medical device component images to the word vector space of a language model.
[0026] According to a second aspect of the embodiments of this application, an apparatus for generating a specification is provided, comprising:
[0027] The acquisition module is configured to acquire first prompt information; the first prompt information includes images and text about the target product.
[0028] The processing module is configured to input the first prompt information into a trained product manual generation model, so that the product manual generation model outputs a manual for the target product; some text in the manual is generated based on the image.
[0029] According to a third aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the method described above.
[0030] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that, when the executable computer program in the storage medium is executed by a processor, the above-described method can be implemented.
[0031] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method.
[0032] Compared with existing technologies, the beneficial effects of this application are as follows: By acquiring first prompt information, which includes images and text about the target product, and inputting the first prompt information into a trained product manual generation model, the product manual generation model outputs a manual for the target product. Some text in the manual is generated based on images. Thus, since the first prompt information input into the product manual generation model includes not only text but also image information, the model can acquire more dimensions of target product information. Consequently, the generated manual contains multi-dimensional correlation information about the target product, ultimately making the generated manual closer to the level of a manually written manual, further saving manual writing time and improving the efficiency of manual writing. Attached Figure Description
[0033] Figure 1 This is a flowchart illustrating a method for generating a specification according to an exemplary embodiment.
[0034] Figure 2 This is a flowchart illustrating a method for generating a specification according to another exemplary embodiment.
[0035] Figure 3 This is a flowchart illustrating a method for generating a specification according to another exemplary embodiment.
[0036] Figure 4 This is a block diagram illustrating a specification generation apparatus according to an exemplary embodiment.
[0037] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0038] Unless otherwise defined, the technical or scientific terms used in this specification and claims shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. Specific embodiments of the invention will be described below with reference to the accompanying drawings. It should be noted that, in order to provide a concise description, this specification cannot provide a detailed description of all features of the actual embodiments. Without departing from the spirit and scope of the invention, those skilled in the art can make modifications and substitutions to the embodiments of the invention, and the resulting embodiments are also within the protection scope of the invention.
[0039] In related technologies, with the development of science and technology, the field of artificial intelligence has seen rapid development. Among them, the large-scale multimodal interaction technology has gradually demonstrated amazing creativity, improving image understanding capabilities and the ability to read audio and video. However, the inventors of this application have found that current multimodal applications are all question-and-answer based on images and audio / video. Currently, in the application scenarios of large-scale multimodal models, there is no evidence of applying multimodal technology to the field of specification generation.
[0040] In related technologies, there are two methods for writing product manuals: Method 1 is manual writing, with personnel from different departments dividing the work and each writing a part of the manual; Method 2 is to use big data to retrieve relevant manuals, extract and generate corresponding manual titles, and then use ChatGPT (Chat Generative Pre-trained Transformer) to iteratively generate the corresponding manual content.
[0041] The following difficulties exist under the first writing method: 1. Technical personnel need to refer to a large number of laws and standards when writing the manual; 2. Finding and reading laws and standards takes a lot of time; 3. Technical personnel need to take turns writing their own modules and cannot refer to other unfinished content.
[0042] The second method involves generating an instruction manual based on the large model's generation capabilities. However, the model does not know the structure, materials, or applicable scope of the medical device, so the generated instruction manual cannot reach the level of a manually written manual.
[0043] To address the aforementioned technical problems, this application proposes a method, apparatus, electronic device, storage medium, and computer program product for generating a user manual, which can make the generated user manual closer to the level of a manually written user manual, further saving user manual writing time and improving the efficiency of user manual writing.
[0044] One embodiment of this application provides a method for generating a specification. This method can be applied to electronic devices with information processing capabilities, such as computers, tablet computers, and mobile phones. Please see [link to relevant documentation]. Figure 1 The method for generating this instruction manual may include the following steps 101 to 102:
[0045] Step 101: Obtain the first prompt information; the first prompt information includes images and text about the target product.
[0046] Step 102: Input the first prompt information into the trained product manual generation model so that the product manual generation model outputs the manual for the target product; some text in the manual is generated based on images.
[0047] In this embodiment, the first prompt information can be a message with a first format. This first prompt information may include multiple fields, some of which are used to fill in text about the target product, and others are used to fill in the storage address of an image of the target product. When the first prompt information is input into a trained product manual generation model, the product manual generation model can obtain an image of the target product based on the image's storage address. Then, the product manual generation model outputs a manual for the target product based on the first prompt information, wherein some text in the manual is generated based on the image of the target product.
[0048] In another embodiment, the first prompt information may include a prompt instruction and an image of the target product; the prompt instruction is a message with a second format, which may include multiple fields, wherein some fields are used to fill in text about the target product, and other fields are used to instruct the product manual generation model to combine the text about the target product with the input image of the target product to output a product manual. Therefore, the step of inputting the first prompt information into the trained product manual generation model may include: first, inputting the image of the target product into the trained product manual generation model, and then inputting the prompt instruction into the trained product manual generation model.
[0049] After the first prompt information is input into the trained product manual generation model, the trained product manual generation model obtains text about the target product based on the prompt instructions, and processes the text about the target product with the input image about the target product to output the product manual.
[0050] In this embodiment, the target product may be a medical device. The product instruction manual may include a general title and at least one chapter. The general title may include the product name. The product instruction manual may include 16 chapters. It should be noted that the number of chapters is not limited to the 16 mentioned in this application. Each chapter includes a chapter title and chapter content.
[0051] In this embodiment, all sections of the product manual are plain text. All sections of the product manual can be divided into two categories: Category 1 and Category 2.
[0052] The chapter titles for the first type of chapters are generated based on the text about the target product in the initial prompt information. This text, related to the target product, guides the product manual generation model in generating the chapter titles and content for the first type of chapters in the product manual. The text about the target product in the initial prompt information may include the chapter titles for the first type of chapters, which are used to guide the product manual generation model in generating the chapter titles and content for the first type of chapters in the product manual.
[0053] The chapter titles for the second category are generated based on the images of the target product provided in the initial prompt. For each image of the target product, this image guides the product manual generation model in generating the corresponding chapter titles and content.
[0054] In this embodiment, the instruction manual for the target product output by the product instruction manual generation model is plain text and does not include images. Users can adjust the instruction manual output by the product instruction manual generation model and insert corresponding images at appropriate locations in the second type of chapters. Since the content of the second type of chapters is generated based on the corresponding images, users only need to insert the corresponding images at appropriate locations, eliminating the need to write lengthy chapter content based on the images. This saves instruction manual writing time and improves the efficiency of instruction manual writing.
[0055] In this embodiment, since the first prompt information input to the product manual generation model includes not only text information but also image information, the product manual generation model can obtain more dimensions of target product information. Consequently, the generated product manual can contain multi-dimensional related information of the target product. Ultimately, the generated manual can be closer to the level of a human-written manual, which can further save manual writing time and improve the efficiency of manual writing.
[0056] Another embodiment of this application provides a method for generating a specification. Based on the above embodiments, this embodiment further includes obtaining reference information for writing the first prompt information before step 101. Please refer to... Figure 2 The method for generating this instruction manual may include the following steps 201 to 203 before step 101:
[0057] Step 201: Obtain the query information input by the user; wherein, the query information includes first identification information, first keyword and second keyword; the first identification information is the identification information of the target product; the first keyword includes the second identification information of the target product in each development stage of the product development process, the second identification information is determined based on the chapter title and the correspondence between the chapter title and the development stage identifier, the chapter title is determined based on the product manual writing specification information; the second keyword is determined based on the necessary content in the chapter content.
[0058] In this embodiment, before writing the first prompt message, the user needs to obtain reference information for writing the first prompt message. This reference information may include product information of the target product, product manual writing specifications, and target information, as described below. The target information may be necessary information in the target product's manual. The target information can be obtained by querying the target product file and is used in conjunction with the target product information and product manual writing specifications to obtain the first prompt message. The target product file includes development process documents recorded during the product development process of the target product.
[0059] In this embodiment, the electronic device first obtains the query information input by the user. The query information includes first identification information, first keywords, and second keywords.
[0060] The first identification information may be the identification information of the target product. The identification information of the target product may be the product name information of the target product.
[0061] The first keyword may include second identification information of the target product at each stage of the product development process. This second identification information can be the stage name of the development phase. The second identification information is determined based on the chapter titles of the instruction manual and the correspondence between these chapter titles and the development phase identifiers. The chapter titles of the instruction manual can be determined based on the product instruction manual writing specifications.
[0062] The determination of the second identifier information based on the chapter titles in the instruction manual and the correspondence between the chapter titles and the R&D stage identifiers can be done manually by the user or through an electronic device. When determined through an electronic device, the electronic device stores the correspondence between the chapter titles and the R&D stage identifiers. The electronic device can query the correspondence between the chapter titles and the R&D stage identifiers based on the chapter titles entered by the user to obtain the second identifier information.
[0063] The chapter titles in a product manual can be determined manually by the user or by an electronic device, based on the manual's writing specifications. When determined by an electronic device, the device can analyze the manual's writing specifications and extract the chapter titles.
[0064] The second keyword is determined based on the essential content of each chapter in the instruction manual. For example, for a chapter titled "Indications," which must include information about the medical conditions and diseases, the information about the medical conditions and diseases is essential content for that chapter, and the medical conditions and diseases are the second keywords determined based on this essential content.
[0065] Step 202: Based on the first identification information, query the product information database to determine the target product file; the product information database includes the correspondence between product identification and product files; the target product file includes the R&D process documents recorded during the product development process of the target product.
[0066] In this embodiment, the product information database can contain product files for multiple products and a mapping between product identifiers and product files. The electronic device can search the product information database based on the first identifier information in the query information to determine the target product file. The target product file may include development process documents recorded during the product development process of the target product.
[0067] Step 203: Based on the first keyword and the second keyword, query the target product file to obtain target information that matches the query information; the target information is used in conjunction with the product information of the target product and the product manual writing specifications to obtain the first prompt information; the R&D process documents include the phase introduction content of each R&D stage and the correspondence between the R&D stage identifier and the phase introduction content, and the phase introduction content includes necessary content.
[0068] In this embodiment, the R&D process document of the target product includes a phased introduction of each R&D stage and a correspondence between the R&D stage identifier and the phased introduction. Each phased introduction includes the corresponding necessary content.
[0069] In this embodiment, the electronic device can search the target product file based on the first keyword and the second keyword in the query information to obtain target information matching the query information. For example, the electronic device can locate the corresponding phase introduction content based on the first keyword, and locate the necessary content associated with the second keyword and the related content of the necessary content within the located phase introduction content based on the second keyword. The target information includes the necessary content in each chapter and the related content of the necessary content, such as text or images. The necessary content in each chapter is the necessary information in the instruction manual.
[0070] In this embodiment, the target information is used to guide the generation of chapter content in the product manual generation model. The target information may include images and text about the target product.
[0071] In this embodiment, the target information is used in conjunction with the product information of the target product and the product manual writing specifications to obtain the first prompt information. After obtaining the target information, the electronic device outputs the target information for user reference. The user can manually edit the appropriate first prompt information based on the target information, the product information of the target product, and the product manual writing specifications.
[0072] Of course, in other embodiments, the electronic device can automatically generate and output the first prompt information based on the target information, the product information of the target product, and the specification information of the product manual. The electronic device can receive prompt adjustment information input by the user and update the first prompt information based on the prompt adjustment information. This can improve the efficiency of writing prompt information.
[0073] In this embodiment, the product manual writing specification information includes a general title and at least one chapter. The general title includes the product name, and each chapter includes a chapter title and chapter content. For example, the product manual may include a general title and 16 chapters, and include chapter titles for all 16 chapters. It should be noted that the number of chapters is not limited to the 16 mentioned in this application.
[0074] In this embodiment, the first prompt information includes product information of the target product.
[0075] In this embodiment, the target product is a medical device. Product information includes product name information, medical device type information, and scope of application information. The medical device type information includes first type information and second type information. The first type information indicates whether the medical device is an active or passive medical device. The second type information indicates whether the medical device is a Class I, Class II, or Class III medical device.
[0076] In this embodiment, all chapters of the product manual can be divided into a first category of chapters and a second category of chapters. The chapter titles of the first category of chapters are generated based on the text about the target product in the first prompt information. The chapter titles of the second category of chapters are generated based on the images of the target product in the first prompt information.
[0077] In this embodiment, the R&D process documents include a phased introduction to each stage of the medical device's R&D process. The user can determine the chapter titles of each section of the instruction manual based on the product instruction manual's writing specifications, and establish a correspondence between chapter titles and R&D stage identifiers. Then, based on the chapter titles and the correspondence between chapter titles and R&D stage identifiers, the user can determine the corresponding R&D stage for each chapter title, and then determine the corresponding phased introduction content based on the R&D stage corresponding to the chapter title. Of course, in other embodiments, the target product documents may include a correspondence between chapter titles and phased introduction content. The electronic device can determine the phased introduction content corresponding to each chapter title based on the chapter titles and the correspondence between chapter titles and phased introduction content.
[0078] In this embodiment, the first prompt information includes a first sub-prompt information and a second sub-prompt information. The first sub-prompt information is a prompt information written based on product information, product manual writing specifications, and R&D process documents for the chapter titles of the first type of chapters. The second sub-prompt information is a prompt information written based on product manual writing specifications, images, and the corresponding chapter titles for the chapter titles of the second type of chapters.
[0079] For the first sub-prompt information, the first reference prompt information can be edited first. Then, based on the R&D process documents, the first reference prompt information can be refined to generate prompt information for the chapter titles of the first category of chapters, which serves as the first sub-prompt information. The first sub-prompt information is used to guide the product instruction manual generation model in generating the chapter content of the first category of chapters. For example, for contraindications, the corresponding first reference prompt information can be edited, and then, based on the medical device R&D process documents, the first reference prompt information can be refined to generate contraindication prompt information, which serves as the first sub-prompt information. A similar method can also be used to obtain the first sub-prompt information for indications.
[0080] Regarding the second sub-prompt information, users can edit it based on images of medical device components, corresponding chapter titles, and product instruction manual writing guidelines. This second sub-prompt information guides the product instruction manual generation model in generating chapter titles and content for the second type of chapters. Chapter titles for the second type of chapters could, for example, include sections on structural principles and usage instructions.
[0081] In this embodiment, the query information input by the user is obtained, and the product information database is searched according to the query information to obtain the target information. The target information is used in conjunction with the product information and the product manual writing specification information to obtain the first prompt information. In this way, more suitable first prompt information can be written, which can more accurately guide the product manual generation model to generate the manual of the target product and improve the processing efficiency of the product manual generation model.
[0082] Another exemplary embodiment of this application also provides a method for generating a product manual. In this embodiment, a method for training a product manual generation model is provided. For example... Figure 3 As shown, based on the above embodiments, before step 102, the method for generating this specification may further include the following steps 301 to 303:
[0083] Step 301: Incrementally train the pre-trained large language model using the first training dataset to obtain the first intermediate model. The first training dataset includes R&D process documents and reference product manuals recorded during product development; the first training dataset includes multiple first training data sets, where each first training data set is text data and consists of R&D process documents or reference product manuals.
[0084] In this embodiment, the target product is a medical device. The first training dataset includes development process documents recorded during the medical device product development process and reference product manuals. The first training dataset includes multiple first training data sets, which are text data, such as TXT format files. One portion of the first training data consists of development process documents, and another portion consists of reference product manuals.
[0085] The R&D process documents include clinical trial data and testing data for the medical device. Testing data may include, for example, material testing data and drug testing data. The reference product instructions are the medical device instruction manuals, including the reference section titles and content. The medical device instruction manual can be a publicly available instruction manual from within the medical device industry.
[0086] In this embodiment, the pre-trained large language model can be ChatGLM, Llama, or Vicuna, but is not limited to these.
[0087] In this embodiment, the pre-trained large language model is incrementally trained using the first training dataset to obtain the first intermediate model.
[0088] Step 302: Supervised fine-tuning of the first intermediate model using the second training dataset to obtain the second intermediate model. The second training dataset includes training data for multiple specified tasks, with the specified task being supervised fine-tuning of the content of the instruction manual for the target product.
[0089] In this embodiment, the second training dataset includes training data for multiple specified tasks, specifically supervised fine-tuning of the content of instructions for use with medical devices. The second training dataset is used to supervise and fine-tune the first intermediate model to obtain the second intermediate model, which allows the output of the second intermediate model to better align with responses within the medical device domain.
[0090] In this embodiment, the second training data is a key-value pair of reference chapter titles and reference chapter contents from the product manual, and the second training data can be stored in JSON format.
[0091] In this embodiment, fine-tuning can be performed using large model fine-tuning methods such as lora (Low-Rank Adaptation), Qlora (Quantized Low-Rank Adaptation), or P-tuning-V2, but is not limited to these methods.
[0092] Step 303: Train the second intermediate model using the third training dataset to obtain the trained product manual generation model. The third training dataset includes multiple third training data sets, which are the training data for the image-text alignment task.
[0093] In this embodiment, the third training dataset includes multiple third training data sets, which are training data for the image-to-text alignment task. The second intermediate model is trained using the third training dataset to obtain a trained product manual generation model.
[0094] In this embodiment, the third training data includes images of medical device components and corresponding text descriptions. The text descriptions include content descriptions of the medical device component images and questions based on the images. The medical device component images and corresponding text descriptions are stored in key-value pairs. For example, the third training data is stored in JSON format. The product manual generation model includes a mapping relationship from the medical device component images to the word vector space of the language model.
[0095] For example, when the pre-trained large language model is Vicuna, the visual encoder of the electronic device uses CLIP (Contrastive Language-Image Pre-train) to extract features from the image, uses a second intermediate model to generate text embeddings, and aligns the image embeddings and text embeddings so that the second intermediate model learns to construct a mapping from images of medical device parts to the word vector space of the language model.
[0096] In an exemplary embodiment, the pre-trained large language model is Vicuna. To make the product instruction manual generation model more professional in the field of medical devices, Vicuna can be incrementally pre-trained using the R&D process documents in the medical device R&D process. Then, the language model Vicuna can be SFT (supervised fine-tuning) using LoRa, so that the model can generate the specified modules of the instruction manual according to the instructions and make the generated content more in line with our expectations. Then, CLIP is used to extract features from the image, and the image embedding is aligned to the semantic space of the text embedding through training.
[0097] In this embodiment, the pre-trained large language model is first incrementally trained using R&D process documents and reference product manuals as training data. Then, the large language model is supervisedly fine-tuned using training data from a supervised fine-tuning task targeting the content of the product's manual, making the model's output more aligned with the product's technical field. Next, the large language model is trained using training data from an image-text alignment task, enabling it to perform image-text alignment. This allows the final product manual generation model to output a manual that better reflects the product's technical field, and it can generate partial text from the manual based on images of the target product in the initial prompt information, thus realizing the application of a multimodal large model in the field of manual generation.
[0098] In this embodiment, the instruction manual generation model is a large language model for text generation. It can generate instruction manuals for medical devices based on the research and development process information of medical devices. In this way, the research and development process information of medical devices can be fully utilized, avoiding users spending extra time on information collection when writing instruction manuals. This can better meet user needs and save a lot of instruction manual writing time.
[0099] The training method for the product manual generation model has been introduced above. Before training the model, there are data collection and training data preparation stages. The following section, focusing on medical devices as the target product, will describe the work involved in each stage.
[0100] I. Data Collection Phase
[0101] 1. Collect publicly available reference product manuals from within and outside the user's company, as well as R&D process documents recorded during product development, such as clinical trial data and testing data. These reference product manuals and R&D process documents include both text and image data.
[0102] 2. Clean the text data in the collected data. Specifically, separate the reference chapter titles and content from the product manual and store them separately in a MySQL database.
[0103] 3. Clean the image data contained in the collected data. The specific method is to first download the images, and then use text to describe the image content. The image content includes, but is not limited to, components, functions, structural descriptions, materials, etc., which are also organized into key-value pairs and saved as JSON data to obtain the initial medical device component images and corresponding text descriptions.
[0104] II. Training Data Processing Phase
[0105] 1. Text-based training data preparation
[0106] The product manual and R&D process documents recorded during product development were converted into txt format to obtain the first training dataset.
[0107] Then, based on the characteristics of each chapter title in the instruction manual and the product instruction manual writing specifications, corresponding second prompt messages were created for each chapter title. The second prompt messages included the chapter titles from the MySQL database. The data in the MySQL database was organized into key-value pairs and saved as JSON format. Then, based on the second prompt messages and the JSON format data, a second training dataset was obtained.
[0108] Adding a summary of chapter titles, product manual writing guidelines, and R&D process documents related to product development to the second prompt information allows the large language model to learn the correlation between chapter titles, product manual writing guidelines, R&D process documents, and results (product manuals).
[0109] 2. Preparation of training data for image-text alignment
[0110] The image content is described in more detail, and questions about the image are generated. Based on these descriptions and questions, the initial medical device component images and their corresponding text descriptions are updated, resulting in updated medical device component images and their corresponding text descriptions. The updated text descriptions provide a more detailed description of the medical device component images and include questions based on them. Then, the third training dataset is obtained based on the updated medical device component images and their corresponding text descriptions.
[0111] Another exemplary embodiment of this application also provides an apparatus for generating a specification. For example... Figure 4 As shown, in this embodiment, the apparatus for generating the specification includes:
[0112] The acquisition module 41 is configured to acquire first prompt information; the first prompt information includes images and text about the target product;
[0113] Processing module 42 is configured to input the first prompt information into the trained product manual generation model so that the product manual generation model outputs the manual for the target product; some text in the manual is generated based on images.
[0114] In this embodiment, since the first prompt information input to the product manual generation model includes not only text information but also image information, the product manual generation model can obtain more dimensions of target product information. Consequently, the generated product manual can contain multi-dimensional related information of the target product. Ultimately, the generated manual can be closer to the level of a human-written manual, which can further save manual writing time and improve the efficiency of manual writing.
[0115] Embodiments of this application also provide an electronic device, including a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the specification generation method of any of the above embodiments.
[0116] Embodiments of this application also propose a computer-readable storage medium that, when an executable computer program in the storage medium is executed by a processor, enables the generation method of the specification of any of the above embodiments.
[0117] Embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements a driving method for the universal serial bus interface of any of the above embodiments.
[0118] Regarding the apparatus in the above embodiments, the specific manner in which the processor performs the operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0119] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, electronic device 900 may be provided as a server. (Refer to...) Figure 5 The device 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions, such as application programs, that can be executed by the processing component 922. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the aforementioned generation method for the specification.
[0120] Device 900 may also include a power supply component 926 configured to perform power management of device 900, a wired or wireless network interface 950 configured to connect device 900 to a network, and an input / output (I / O) interface 958. Device 900 can operate on an operating system stored in memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.
[0121] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, which can be executed by a processing component 922 of the device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0122] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.
[0123] The above description of the embodiments is intended to enable those skilled in the art to understand and apply this application. It will be apparent to those skilled in the art that various modifications can be easily made to these embodiments, and the general principles described herein can be applied to other embodiments without creative effort. Therefore, this application is not limited to the embodiments described herein, and any improvements and modifications made by those skilled in the art based on the disclosure of this application without departing from the scope and spirit of this application are within the scope of this application.
Claims
1. A method of generating a specification, characterized by, The method comprises the following steps: obtaining first prompt information; the first prompt information comprises an image and text about a target product; inputting the first prompt information into a trained product description generation model, so that the product description generation model outputs a description of the target product; part of the text in the description is generated based on the image.
2. The generation method of a specification according to claim 1, characterized in that, The product description comprises at least one chapter, and each chapter comprises a chapter title and chapter content. Before obtaining the first prompt information, the method further comprises the following steps: obtaining query information input by a user; wherein the query information comprises first identification information, first keywords and second keywords; the first identification information is identification information of the target product; the first keywords comprise second identification information of each research and development stage of the target product in the product research and development process, the second identification information is determined based on the chapter title and the corresponding relationship between the chapter title and the research and development stage identification, and the chapter title is determined based on product description writing specification information; the second keywords are determined based on necessary content in the chapter content; querying in a product information library based on the first identification information to determine a target product file; the product information library comprises a corresponding relationship between product identification and product files; the target product file comprises research and development process files recorded in the product research and development process of the target product; querying in the target product file based on the first keywords and the second keywords to obtain target information matching the query information; the target information is used to obtain the first prompt information in cooperation with product information of the target product and the product description writing specification information; the research and development process file comprises stage introduction content of each research and development stage and a corresponding relationship between research and development stage identification and stage introduction content, and the stage introduction content comprises the necessary content.
3. The generation method of a specification according to claim 2, characterized in that, In the product description writing specification information, the product description comprises a total title and at least one chapter, and the total title comprises a product name; the first prompt information comprises product name information of the target product; and the product information of the target product comprises product name information.
4. The method of generating a specification according to claim 3, wherein, The target product is a medical device; the product information further comprises medical device type information and application range information, the medical device type information comprises first type information and second type information, the first type information is one of active medical devices and passive medical devices, and the second type information is one of class I medical devices, class II medical devices and class III medical devices.
5. The method of generating a specification according to claim 3, wherein, At least one chapter comprises a first type of chapter, the chapter title of the first type of chapter is generated based on the text about the target product in the first prompt information; the first prompt information comprises first sub-prompt information, and the first sub-prompt information is prompt information written for the chapter title of the first type of chapter based on the product information, the product description writing specification information and the research and development process file.
6. The method of generating a specification of claim 3, wherein, At least one of the chapters includes a second type of chapter, and a chapter title of the second type of chapter is generated based on the image about the target product in the first prompt information; the first prompt information includes second sub-prompt information, and the second sub-prompt information is prompt information for writing a chapter title of the second type of chapter based on the product specification writing specification information, the image, and the chapter title corresponding to the image.
7. The method of generating a specification of claim 1, wherein, Before the first prompt information is input into the trained product specification generation model, the method further includes: The pre-trained large language model is incrementally trained using a first training data set to obtain a first intermediate model; the first training data set includes development process files recorded in a product development process and reference product specifications; the first training data set includes a plurality of first training data, and the first training data is text data, and the first training data is the development process file or the reference product specification; The first intermediate model is supervised fine-tuned using a second training data set to obtain a second intermediate model; the second training data set includes a plurality of second training data of specified tasks; the specified task is a supervised fine-tuning task for the content of the specification of the target product; The second intermediate model is trained using a third training data set to obtain the trained product specification generation model; wherein the third training data set includes a plurality of third training data, and the third training data is training data of an image and text alignment task.
8. The method of generating a specification of claim 7, wherein, The target product is a medical device; the product specification includes at least one chapter, and each chapter includes a chapter title and chapter content; The development process file includes clinical experiment data and detection data of a medical device; the reference product specification is a medical device specification, including a reference chapter title and reference chapter content; The second training data is a key-value pair of the reference chapter title and the reference chapter content; The third training data includes a medical device component image and a corresponding text description, and the text description includes a content description of the medical device component image and a question based on the medical device component image; the medical device component image and the corresponding text description are stored in the form of a key-value pair; the product specification generation model includes a mapping relationship from a medical device component image to a language model word vector space.
9. An apparatus for generating a specification, characterized by Comprise: An acquisition module configured to acquire first prompt information; The first prompt information includes an image and text about a target product; A processing module configured to input the first prompt information into a trained product specification generation model to cause the product specification generation model to output a specification of the target product; and part of the text in the specification is generated based on the image.
10. An electronic device, comprising: A memory and a processor, the memory is used to store computer programs executable by the processor; the processor is used to execute the computer programs in the memory to realize the method of any one of claims 1-8.
11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program embodied in the storage medium, when executed by a processor, is capable of implementing the method according to any one of claims 1-8.
12. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-8.