Method for processing identification result and related device

Abstract

The invention relates to a method for processing an identification result and a related device. The invention discloses a method for processing an identification result. The method comprises the following steps: acquiring a species image from a user and characteristic information of the user; identifying image features of the species image, and extracting content information from a content database according to the identified image features; inputting the content information and the characteristic information into a large language model to generate an identification result page from the content information based on the characteristic information; and displaying the recognition result page provided by the large language model on a user interface.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and more specifically, to a method for processing recognition results, an electronic device, a non-transitory storage medium, and a computer program product, and also to a maintenance system. Background Art

[0002] Currently, there are many applications (APPs) for identifying objects, such as applications for identifying plants, etc. These applications usually receive images input from users, and identify objects in the images through recognition models established based on artificial intelligence technology to obtain recognition results, and present the recognition results to users on the user interface. Summary of the invention

[0003] A brief overview of the disclosure is given below in order to provide a basic understanding of some aspects of the disclosure. However, it should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify the key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to give some concepts of the disclosure in a simplified form as a prelude to a more detailed description given later.

[0004] According to a first aspect of the present disclosure, a method for processing recognition results is provided, comprising: acquiring a species image from a user and characteristic information of the user; identifying image features of the species image, and extracting content information from a content database based on the identified image features; inputting the content information and characteristic information into a large language model (LLM) to generate a recognition result page from the content information based on the characteristic information; and displaying the recognition result page provided by the large language model on a user interface.

[0005] In some embodiments, the characteristic information includes first characteristic information obtained through historical data of the user.

[0006] In some embodiments, the first characteristic information includes at least one of the following: attribute characteristic information, the attribute characteristic information includes maintenance level information; operation characteristic information, the operation characteristic information includes maintenance history information.

[0007] In some embodiments, the characteristic information includes second characteristic information obtained through user interaction data.

[0008] In some embodiments, the second characteristic information includes demand characteristic information, and the demand characteristic information includes one or more of focus content information, detail preference information, typesetting preference information, and reading habit information.

[0009] In some embodiments, the method further includes: obtaining one or more of the user's location information, time information, weather information, and climate information; inputting the species image, the one or more information, and the content information into a multimodal model to adjust the content information based on the species image and the one or more information; and inputting the adjusted content information and feature information into a large language model to obtain a recognition result page.

[0010] In some embodiments, the method also includes: displaying interactive questions about identifying species images on a user interface; receiving user input including responses to the interactive questions, the user input including at least one of images, text, audio, or video; inputting the species image, user input, and content information into a multimodal model to adjust the content information based on the species image and user input; and inputting the adjusted content information and feature information into a large language model to obtain a recognition result page.

[0011] In some embodiments, the method further includes: inputting the species image, content information and a preset content framework into an artificial intelligence generated content (AIGC) model to supplement the content information, wherein the supplemented content information includes the content that is missing compared to the preset content framework; and inputting the supplemented content information and feature information into a large language model to obtain a recognition result page.

[0012] In some embodiments, the feature information includes focus content information, and the method includes: inputting the focus content information, species images, and content information into the AIGC model to supplement the content information, wherein the supplemented content information includes the content information that is missing compared to the focus content information; and inputting the supplemented content information and feature information into the large language model to obtain a recognition result page.

[0013] In some embodiments, the feature information includes detail preference information, and the method includes: inputting the detail preference information, species image and content information into the AIGC model to regenerate the content information, the regenerated content information has a detail level that conforms to the detail preference information; and inputting the regenerated content information and feature information into the large language model to obtain a recognition result page.

[0014] In some embodiments, the recognition results page includes one or more content modules.

[0015] In some embodiments, the one or more content modules are divided according to themes, and the recognition result page further includes a primary dividing line between every two adjacent content modules in the one or more content modules.

[0016] In some embodiments, the recognition result page further includes a first level title located before each content module of the one or more content modules.

[0017] In some embodiments, the first-level title is determined based on a summary of the content module, or based on the main points of the content module.

[0018] In some embodiments, each of the one or more content modules includes one or more paragraphs divided according to contextual relationships, and the recognition result page further includes a secondary dividing line between every two adjacent paragraphs of the one or more paragraphs.

[0019] In some embodiments, the recognition result page further includes a secondary heading located before each of the one or more paragraphs.

[0020] In some embodiments, the secondary heading is determined based on a summary of the paragraph, or based on the main points of the paragraph.

[0021] In some embodiments, key words and / or key sentences in one or more paragraphs are highlighted.

[0022] In some embodiments, one or more paragraphs are arranged in an ordered list or an unordered list.

[0023] In some embodiments, the recognition result page does not include a first-level title before a content module located at the front of the recognition result page among the one or more content modules.

[0024] In some embodiments, the large language model is trained using first training data, the first training data including a combination of a first text indicating content information and a second text indicating feature information as a sample, and the first training data also including a recognition result page of a label as a sample.

[0025] In some embodiments, the multimodal model is trained with second training data, the second training data including a species image as a sample, a first text indicating content information as a processing object, and a combination of data of any one or more modalities indicating reference information as a processing reference, and the second training data also includes a second text indicating content information as a processing result as a label of the sample.

[0026] In some embodiments, the AIGC model is trained with third training data, the third training data including a combination of a species image as a sample, a first text indicating content information as a processing object, and a second text indicating reference information as a processing reference, and the third training data also includes a third text indicating content information as a processing result as a label of the sample.

[0027] In some embodiments, the method includes: generating a maintenance plan based on the identification result page, the maintenance plan including one or more pairs, each of the one or more pairs including one or more maintenance tasks and an identification of a maintenance device for performing the one or more maintenance tasks; and displaying the maintenance plan on a user interface.

[0028] In some embodiments, the method includes: controlling the corresponding maintenance device in each of the one or more pairs in the maintenance plan according to an identification of the maintenance device to complete one or more maintenance tasks in the pair.

[0029] According to the second aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory storing computer executable instructions, which, when executed by the one or more processors, enable the one or more processors to execute the method for processing recognition results described in any embodiment of the first aspect of the present disclosure.

[0030] According to a third aspect of the present disclosure, there is provided a non-volatile storage medium having computer executable instructions stored thereon, which, when executed by a computer, enables the computer to execute the method for processing recognition results described in any embodiment of the first aspect of the present disclosure.

[0031] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product includes instructions, and when the instructions are executed by a processor, the method for processing a recognition result according to any embodiment of the first aspect of the present disclosure is implemented.

[0032] According to a fifth aspect of the present disclosure, a maintenance system is provided, comprising: an electronic device, the electronic device comprising a processor and a memory coupled to the processor and storing instructions, the instructions, when executed by the processor, causing the processor to: obtain a species image from a user and characteristic information of the user, identify image features of the species image, and extract content information from a content database based on the identified image features, input the content information and the characteristic information into a large language model to generate a recognition result page from the content information based on the characteristic information, generate a maintenance plan based on the recognition result page, the maintenance plan comprising one or more pairs, each of the one or more pairs comprising one or more maintenance tasks and an identification of a maintenance device for performing the one or more maintenance tasks, transmit a command to a corresponding maintenance device based on the identification of the maintenance device in each of the one or more pairs in the maintenance plan to control the corresponding maintenance device to complete the one or more maintenance tasks in the pair; and a maintenance device communicatively coupled to the electronic device, the maintenance device being configured to perform a maintenance task in response to receiving a command from the electronic device.

[0033] In some embodiments, the maintenance device is configured to transmit execution data to the electronic device in response to the execution of a maintenance task, wherein the instructions include instructions that, when executed by the processor, cause the processor to perform the following operations: updating the user's characteristic information based on the execution data received from the maintenance device.

[0034] In some embodiments, the maintenance system includes: a camera communicatively coupled to an electronic device, the camera being configured to capture species images and transmit the captured species images to the electronic device, wherein the instructions include instructions that, when executed by the processor, cause the processor to perform the following operations: generating the identification result page based on the species image received from the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The foregoing and other features and advantages of the present disclosure will become apparent from the following description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are incorporated herein and form a part of the specification, and are further used to explain the principles of the present disclosure and enable those skilled in the art to make and use the present disclosure. Among them:

[0036] Figure 1 A flowchart of a method for processing recognition results according to some embodiments of the present disclosure is shown;

[0037] Figure 2 A schematic diagram showing an exemplary user interface in which the method for processing recognition results according to some embodiments of the present disclosure is applied;

[0038] Figure 3 A schematic diagram exemplarily shows a recognition result page obtained by applying the method for processing recognition results according to some embodiments of the present disclosure;

[0039] Figure 4 The training data used for training a large language model according to some embodiments of the present disclosure are exemplarily shown;

[0040] Figure 5 The following exemplarily shows the training data used for training the multimodal model according to some embodiments of the present disclosure;

[0041] Figure 6 The training data used to train the AIGC model according to some embodiments of the present disclosure are exemplarily shown;

[0042] Figure 7 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown;

[0043] Figure 8A schematic block diagram showing a computer system on which some embodiments of the present disclosure may be implemented;

[0044] Fig. 9 A schematic block diagram of a maintenance system according to some embodiments of the present disclosure is shown.

[0045] Note that in the embodiments described below, sometimes the same reference numerals are used in common between different drawings to represent the same parts or parts with the same functions, and their repeated descriptions are omitted. In some cases, similar numbers and letters are used to represent similar items, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0046] For ease of understanding, the position, size, range, etc. of each structure shown in the drawings and the like may not represent the actual position, size, range, etc. Therefore, the present disclosure is not limited to the position, size, range, etc. disclosed in the drawings and the like. DETAILED DESCRIPTION

[0047] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0048] The following description of at least one exemplary embodiment is in fact merely illustrative and is in no way intended to limit the present disclosure and its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will appreciate that they merely illustrate exemplary ways of the present disclosure that can be implemented, rather than exhaustive ways. In addition, the drawings need not be drawn to scale, and some features may be enlarged to illustrate the details of specific components.

[0049] In addition, technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0050] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0051] The page that presents the recognition results in the user interface (hereinafter referred to as the "recognition result page") usually contains a large amount of text information and image information. For users, it often takes a long time to completely read the large amount of information in the recognition result page in order to obtain the information they are concerned about (for example, the type of species (such as plants) and the recognition results of diseases), and they are unable to quickly grasp the key points. In addition, some applications can highlight part of the information presented on the recognition result page to remind users. However, the types of these highlighted key information are often fixed, and the key content that different users are concerned about is often different. Therefore, prompting the same key information cannot meet the personalized needs of users.

[0052] To this end, the present disclosure provides a method for processing recognition results, which personalizes the presentation of content information of the recognition results through a large language model based on the user's characteristic information, so that the recognition result page presented to the user can be customized according to the user's personal characteristics, and the content that the user should pay attention to and the content that the user is interested in can be filtered out from a large amount of information, helping the user to quickly grasp the key points, improve the efficiency of information transmission, and improve the user experience.

[0053] The method for processing the recognition result according to various embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. It is understood that the actual method for processing the recognition result may also include other steps, but in order to avoid blurring the main points of the present disclosure, these other steps are not discussed herein and are not shown in the accompanying drawings.

[0054] Figure 1 FIG. 1 is a flow chart of a method 100 for processing recognition results according to some embodiments of the present disclosure. Figure 1 As shown, the method 100 includes:

[0055] At step S102, a species image and characteristic information of the user are obtained from the user;

[0056] At step S104, image features of the species image are identified, and content information is extracted from a content database according to the identified image features;

[0057] At step S106, the content information and the feature information are input into the large language model to generate a recognition result page from the content information based on the feature information;

[0058] At step S108 , the recognition result page provided by the large language model is displayed on the user interface.

[0059] Specifically, identifying the species in the species image may include, for example, identifying the type and / or disease of the species in the species image. Accordingly, the content information extracted from the content database based on the identified image features may include content related to identifying the type and / or disease of the species in the species image. In addition, before identifying the image features of the species image, the species image may be preprocessed. The preprocessing may include normalization, brightness adjustment, or noise reduction. As a non-limiting example, the following may be provided: Figure 2 The user interface 200 shown includes a dialog box 210, an input box 220, and an input button 230, wherein the input box 220 is used to receive text input by a user, and the input button 230 is used to receive images, audio, video, etc. input by a user.

[0060] For example, content information may be pre-stored in a content database, so that in response to a species image uploaded by a user, corresponding content information may be extracted from the content database. For example, the name of the species, the conservation information of the species, the disease of the species, and the corresponding treatment and prevention methods, etc. may be stored in the content database. Such content information about the species may be stored in the content database in association with the image features about the species. Content information may be extracted based on the degree of matching between the image features of the identified species image and the image features of the species stored in the content database, for example, when the degree of matching falls within a preset range. As a non-limiting example, the cosine similarity of a first vector representing the image features of the identified species image and a second vector representing the image features of the species stored in the content database may be calculated, and when the calculated cosine similarity exceeds a preset threshold, it is considered that the image features of the identified species image match the image features of the species stored in the content database, and the content information stored in the content database in association with the matched image features is extracted.

[0061] In addition, for example, a recognition model can be established based on a neural network (such as a deep convolutional neural network) or a deep residual network, and the species image can be recognized by a pre-trained (or "trained") recognition model to obtain image features. After the recognition model recognizes the image features of the species image, content information related to the species image can be extracted from the content database based on the image features.

[0062] In some embodiments, the characteristic information of the user includes the first characteristic information obtained through the historical data of the user. As a non-limiting embodiment, the historical data of the user can be stored in the user database. For example, when the user uploads the species image to the application, he is usually in the logged-in state, so the account information of the user is known, so the historical data of the user can be called from the user database according to the account information of the user to obtain the first characteristic information of the user. Exemplarily, the first characteristic information may include attribute characteristic information and / or operation characteristic information. The attribute characteristic information may include maintenance level information, etc. For example, if the user has maintained a variety of plants, and the growth status of the plants is very good, it means that the user's maintenance level is high. On the contrary, if the user has never maintained a plant or the growth status of the maintained plants is very bad, it means that the user's maintenance level is low. Operation characteristic information may include maintenance history information, etc. For example, when the user was maintaining a certain plant, after the plant was sick, he only watered the plant and caused the plant to wither, indicating that the user's operation was incorrect and untimely. For another example, when the user was caring for a certain plant, he took preventive measures in time before the plant entered a disease-prone period, thereby enabling the plant to grow healthily, indicating that the user's operation was correct and timely.

[0063] In some embodiments, the user's characteristic information includes second characteristic information obtained through the user's interaction data. In some examples, the interaction data may be historical interaction data reflecting the user's historical interaction operations, or current interaction data reflecting the user's current interaction operations. Exemplarily, the second characteristic information may include demand characteristic information. The demand characteristic information may include one or more of the following: attention content information, detail preference information, typesetting preference information, reading habit information, etc. Figure 2 The user interface 200 shown, for example, for the current interactive data, the user can input his own characteristic information through the input box 220. For example, the user can input instructions such as "please provide the conservation precautions of this species", "please omit the species details information of this species", "please bold the diagnosis results of the disease of this species and the key points of the treatment measures" in the form of text or voice so that the large language model can adjust the presented recognition result page according to the user's characteristic information. For another example, for historical interactive data, the user has deleted some content modules that he is not interested in in the historically presented recognition result page. Therefore, when the recognition result page is currently presented, those content modules are automatically hidden to avoid manual deletion by the user again, thereby improving the user experience. Of course, the user's interactive operations can also include various interactive operations such as clicking, graffiti, and annotation.

[0064] After obtaining the content information about the species image input by the user and the characteristic information about the user, the content information and the characteristic information can be input into the large language model to generate a personalized recognition result page for the user. In some cases, the content information extracted from the content database can be input into the large language model as it is. In other cases, the content information extracted from the content database can be preprocessed before being input into the large language model so as to better meet the needs of the user.

[0065] In some aspects, more user-related information can be introduced through a multimodal model to obtain more accurate and relevant content information, thereby improving the effective information density and information transfer efficiency of the recognition result page generated based on the adjusted content information.

[0066] A multimodal model refers to a machine learning model that can process and understand information in multiple modalities (such as text, images, audio, video, etc.). Multimodal models improve understanding and generation capabilities by fusing different types of data. They can capture the correlation between different modalities, thereby providing richer context and more accurate output. Non-limiting examples of multimodal models include CLIP, DALL-E, GPT-4, etc.

[0067] For example, take the processing of two modal data, image and text, as an example. In some embodiments, the multimodal model may include only a multimodal large language model for processing image features and text features. In other embodiments, the multimodal model may include (one, or multiple serial or parallel) visual models for extracting image features and a multimodal large language model for processing image features and text features coupled to the downstream of the visual model. The expressiveness of image features can be improved by a special visual model, thereby optimizing the performance of the multimodal model on visual tasks. The visual model can be based on various suitable neural network architectures, such as ResNet, DenseNet, etc. Exemplarily, the visual model can be a convolutional neural network (CNN) transformer model, such as but not limited to a ConvNext model. In order to improve the expressiveness of such visual models in specific application fields (e.g., plant identification fields), image-text pairs (e.g., plant image-text pairs) can be used to train them through contrastive learning, which will be described in more detail later. For example, the CNN Transformer model can be pre-trained with image-text pairs through contrastive learning, and then jointly trained with the downstream multimodal large language model using multimodal data. This is conducive to improving the overall performance of the multimodal model. Of course, it is also possible to pre-train the CNN Transformer model separately first, and then fix the parameters of the CNN Transformer model when training the multimodal model with multimodal data and only update the parameters of the multimodal large language model, which can speed up training and reduce the computing and storage resources consumed by training.

[0068] It can be understood that the same is true for other multimodal data. For example, if audio data needs to be processed, the multimodal model can be provided by using a multimodal large language model for processing audio features, image features, and text features alone or in combination with one or both of a visual model for extracting image features and an auditory model for extracting audio features.

[0069] For example, assume that the multimodal model receives image data (I) and text data (T), although there may be data in other modalities in other cases. Image data may include, but is not limited to, images input by the user (e.g., species images), images from a content database similar to the images input by the user, template images for recognition results (e.g., recognized species or diseases, etc.). Text data may include, but is not limited to, various data input by the user such as location data, time data, weather data, climate data, etc. It is understandable that the historical data and interaction data mentioned above can also be received by the multimodal model in various modalities.

[0070] After receiving the above-mentioned data of multiple modalities, the multimodal model first needs to convert these different types of data into feature representations that are compatible within the model so as to fuse these different types of data. This is usually achieved by the encoders of different modalities.

[0071] Specifically, the multimodal model can convert image data and text data into corresponding feature representations through an image encoder fI (such as a convolutional neural network (CNN), a recurrent neural network (RNN), etc.) and a text encoder fT (such as a Word2Vec model, a bag of words model (BoW), etc.), namely,

[0072] VI = fI (I),

[0073] VT=fT(T),

[0074] Among them, VI represents image feature representation, VT represents text feature representation, I represents image data, and T represents text data.

[0075] After converting different types of data into corresponding feature representations, feature fusion can be performed to fuse the feature representations of different modalities into a unified feature representation for input into the model for further processing or prediction. Fusion can be achieved by simple concatenation, weighted average, and / or more complex fusion mechanisms (such as attention-based fusion mechanisms, etc.). Specifically, for example, Vfused = β × VI + (1-β) × VT, where Vfused represents a unified feature representation and β represents a parameter for adjusting the importance of VI and VT in the fusion process. It should be understood that the specific fusion method can be selected according to the actual situation to obtain the best performance.

[0076] In some embodiments, method 100 includes: obtaining one or more of the user's location information, time information, weather information, and climate information; inputting the species image, the one or more of the information, and the content information into a multimodal model to adjust the content information based on the species image and the one or more of the information; and inputting the adjusted content information and feature information into a large language model to obtain a recognition result page.

[0077] As a non-limiting embodiment, it is assumed that after identification by the recognition model, it is determined that the species in the species image input by the user is a tomato plant, and the tomato plant has flowers. It can be understood that for different times (e.g., time units such as seasons and months) and different climates, the ripening time of tomatoes is different. For example, in March and April, the temperature is low, and it takes 50 to 60 days for tomatoes to bloom and bear fruit. In May and June, the temperature is suitable, and it takes 45 to 50 days for tomatoes to bloom and bear fruit. In July and August, the temperature is high, and it takes about 40 days for tomatoes to bloom and bear fruit. When the user uploads the image of the tomato in March, the content information extracted from the content database may include information on the ripening time required for tomatoes from blooming to bearing fruit in three situations. This means that if all the content information extracted from the content database is presented as it is in the recognition result page, the information that the user needs to read contains a large amount of "noise" information, resulting in low efficiency in the user's acquisition of target information. In this embodiment, the multimodal model can further adjust the content information based on the user's time information (e.g., March) so that the adjusted content information only contains information about the maturity time required for tomatoes from flowering to fruiting in March, so that the recognition result page finally provided by the large language model can simply present "it takes 50 to 60 days for tomatoes to grow from flowering to fruiting". Therefore, for the user, the content presented on the recognition result page is clear at a glance. It can be understood that although the present disclosure uses plant images as species images to describe the embodiments, this is only exemplary and not restrictive, and the species image can also be an image of any other species, such as an animal image.

[0078] In some embodiments, method 100 includes: displaying interactive questions about identifying species images on a user interface; receiving user input including responses to the interactive questions, the user input including at least one of an image, text, audio, or video; inputting the species image, user input, and content information into a multimodal model to adjust the content information based on the species image and the user input; and inputting the adjusted content information and feature information into a large language model to obtain a recognition result page.

[0079] For example, when the recognition model fails to converge, the confidence of the output result is not enough, the precision of the output result is not enough, etc., it can be considered that the recognition model cannot provide image features sufficient to accurately extract content information from the content database. This may be due to a variety of reasons, such as there are many species in the species image and it is not known which species the target species is, the species image is too blurred, etc. In addition, content information may be extracted from the content database based on the degree of matching with the image features. When there are multiple content information matching the image features, they can all be extracted, but only a part of them may be what the user really needs. For these situations, more information (especially highly distinguishable information) can be obtained by asking interactive questions to generate accurate and reliable content information. For example, interactive questions may include requests for voice / text descriptions of the target species, close-up images or videos of one or more characteristic parts of the target species, etc. Assuming that the species in the species image input by the user has a disease, the interactive questions may include one or more of requesting close-up images of one or more diseased parts of the species, the shooting time of the species image, the shooting location of the species image, and the maintenance details of the species.

[0080] Furthermore, a chain of interactive questions can be designed, and then interactive questions can be output in sequence along the chain until the information obtained is sufficient to provide content information about identifying the species in the species image. For example, for identifying the disease of the species in the species image, you can first ask for a close-up image of the diseased part of the species. If the accuracy of the content information still does not meet expectations, you can further ask for the shooting time and location of the species image.

[0081] For non-limiting illustrative purposes, for example, reference Figure 2 . The user provides a plant image. While the type of plant in the image is determined based on the plant image provided by the user, since it is also recognized that the plant seems to have a disease, the user is requested to provide photos of the plant from other angles, such as close-up photos of the fruit. In addition, since the diagnosis of plant diseases requires consideration of multiple factors to accurately diagnose whether the plant is sick and what the disease is, the user is requested to provide the time and place where the photo was taken. After the user answers that it was taken in place Y in June of this year, the multimodal model uses the collected user input to adjust the content information extracted from the content database, and then presents the user with a recognition result page generated by the large language model based on the adjusted content information.

[0082] It is understandable that in addition to responding to interactive questions, users can also actively provide user input, which can also be used by the multimodal model to adjust the content information.

[0083] In some aspects, the AIGC model can be used to enhance / optimize content information. The AIGC model can use artificial intelligence technology to automatically generate various types of content, including text, images, audio, video, etc. Such models are usually based on complex algorithms and deep learning architectures, and can complete tasks such as creative generation and content creation in various applications. The main types of AIGC models include text generation models, image generation models, audio generation models, video generation models, etc. or any combination thereof. The AIGC model can use general models such as GPT, Llama, Gemini, Wenyan Yixin, etc., or it can use self-developed models.

[0084] In some embodiments, method 100 includes: inputting species images, content information and a preset content framework into the AIGC model to supplement the content information, the supplemented content information including the content that is missing compared to the preset content framework; and inputting the supplemented content information and feature information into the large language model to obtain a recognition result page.

[0085] As a non-limiting implementation, for example, the species in the species image uploaded by the user is a diseased plant, and the preset content framework for the identification result page of the diseased plant is assumed to include three parts: plant species, disease type, and treatment measures, but the content information extracted from the content database only includes plant species and disease type, but does not include treatment measures. Therefore, after the species image, content information, and preset content framework are input into the AIGC model, the AIGC model supplements the content information so that the supplemented content information further includes treatment measures.

[0086] Since the information stored in the content database is relatively fixed and limited, the content information extracted from the content database may be missing in some cases. Therefore, the introduction of the AIGC model can improve the integrity of the content information used to generate the recognition result page.

[0087] In some embodiments, when the feature information includes focus content information, method 100 includes: inputting the focus content information, species images, and content information into the AIGC model to supplement the content information, the supplemented content information including the content that is missing compared to the focus content information; and inputting the supplemented content information and feature information into the large language model to obtain a recognition result page.

[0088] As a non-limiting implementation, the species in the species image uploaded by the user is a diseased plant, and the content information extracted from the content database includes plant species, disease type, and treatment measures, but the user is also concerned about preventive measures. Therefore, after the species image, content information, and the concerned content information indicating "concern for preventive measures" are input into the AIGC model, the AIGC model supplements the content information so that the supplemented content information further includes preventive measures, thereby enabling the recognition result page generated by the large language model to present information that meets the user's concern preferences, further optimizing the user experience.

[0089] In some embodiments, when the feature information includes detail preference information, method 100 includes: inputting the detail preference information, species image and content information into the AIGC model to regenerate the content information, the regenerated content information has a detail level that conforms to the detail preference information; and inputting the regenerated content information and feature information into the large language model to obtain a recognition result page.

[0090] As a non-limiting embodiment, the species in the species image uploaded by the user is a diseased plant, and the content information extracted from the content database includes plant species, symptom type, treatment measures and preventive measures. For the plants that they maintain, users usually know the species of the plant and therefore do not need to know the species information of the plant. In contrast, users are very concerned about the symptom type and treatment measures of the plant, and hope to learn about the preventive measures of the disease by the way. Therefore, after the species image, content information and the detailed preference information indicating "not mentioning the plant species, detailing the symptom type, detailing the treatment measures and briefly describing the preventive measures" are input into the AIGC model, the AIGC model can regenerate the content information so that it includes detailed content about the symptom type and treatment measures of the plant and summary content about the preventive measures of the disease of the plant, so that the recognition result page generated by the large language model can present information that meets the user's detailed preference, further optimizing the user's experience.

[0091] It can be understood that the content information extracted from the content database can be input into the large language model as it is, or it can be processed by one or both of the multimodal model and the AIGC model in any desired order according to specific needs, and then input into the large language model after processing to generate a recognition result page.

[0092] Compared with directly presenting the content information in the recognition result page, the recognition result page generated from the content information by the large language model based on the user's characteristic information can have a specific format and content, which significantly improves the user experience.

[0093] In some embodiments, the recognition result page includes one or more content modules. For example, one or more content modules can be divided according to themes. The recognition result page can also include a primary dividing line between each adjacent two content modules in the one or more content modules to help users easily distinguish content modules of different themes visually.

[0094] refer to Figure 3 , which shows a schematic diagram of a recognition result page 300 obtained by applying some embodiments of the present disclosure. Figure 3 As shown, the recognition result page 300 includes four content modules, namely, a content module 3020 for introducing the diagnosis report at the beginning, a content module 3022 for explaining the causes of tomato cracking, a content module 3024 for explaining the edibility of cracked tomatoes, and a content module 3026 for explaining the preventive measures for tomato cracking. Figure 3 In the figure, there is a primary dividing line depicted as a dotted line between content module 3020 and content module 3022, between content module 3022 and content module 3024, and between content module 3024 and content module 3026. To simplify the drawings, only the primary dividing line between content module 3024 and content module 3026 is identified by reference numeral 306. It is understood that the primary dividing line and the secondary dividing line mentioned later can take any appropriate style, including but not limited to the illustrated dotted line, blank line, etc.

[0095] In certain embodiments, the recognition result page also includes a first level title before each content module in one or more content modules. For example, the first level title can be determined based on the summary of the content module, or determined based on the main points of the content module.

[0096] As a non-limiting embodiment, continue to refer to Figure 3 , the recognition result page 300 includes a first-level title “Causes of tomato cracking: water, poor skin toughness” before the content module 3022, a first-level title “Cracked tomatoes are stilledible” before the content module 3024, and a first-level title “Preventcracking: change the watering time. Increase potassium and calciumfertilizers” before the content module 3026. To simplify the drawings, only the first-level title before the content module 3024 is identified by the reference numeral 308.

[0097] The large language model does not necessarily refine the first-level title into a more concise traditional title (such as "Causes", "Edibility", "Precautions"), but retains the flexibility of the first-level title while ensuring the accuracy of the first-level title of each content module, so that the first-level title itself can convey the core idea of ​​the corresponding content module so that users can immediately judge the content focus of the content module through the first-level title. For example, a short expression such as "Cracked tomatoes are stilledible" can convey the main idea of ​​the content itself, and can quickly and accurately convey more information to users compared to traditional titles such as "Edibility".

[0098] Of course, it is understandable that if the content module has too much content and the first-level title cannot express the main points of the content module, the first-level title can also be refined into a traditional title.

[0099] In addition, the recognition result page may not include a first-level title before the content module located at the front. Figure 3 As shown, there is no first-level title before the content module 3020. The title of the opening introduction content is not necessary, and in some cases, not adding a title may make it more natural for users to read.

[0100] In some embodiments, each of the one or more content modules of the recognition result page includes one or more paragraphs divided according to contextual relationships. For example, the recognition result page may also include a secondary dividing line between each two adjacent paragraphs in the one or more paragraphs to avoid the appearance of large paragraphs of continuous text, thereby reducing the reading burden of the user.

[0101] As a non-limiting embodiment, continue to refer to Figure 3 Taking the content module 3022 as an example, the content module 3022 includes a first paragraph and a second paragraph, wherein the first paragraph is used to describe one reason for tomato cracking - water, and the second paragraph is used to describe another reason for tomato cracking - poor skin toughness. In addition, the recognition result page 300 includes a secondary dividing line 310 depicted as a blank line between the first paragraph and the second paragraph of the content module 3022.

[0102] In some embodiments, the recognition result page further includes a secondary title located before each paragraph in the one or more paragraphs. For example, the secondary title can be determined based on the summary of the paragraph, or based on the main points of the paragraph.

[0103] As a non-limiting embodiment, continue to refer to Figure 3, the recognition result page 300 also includes a secondary title "1. Water" before the first paragraph of the content module 3022 and a secondary title "2. Poor skin toughness" before the second paragraph of the content module 3022. To simplify the drawings, only the secondary title before the first paragraph of the content module 3022 is identified by the reference numeral 312. Through the secondary titles, users can quickly know the main content of each paragraph, thereby improving the efficiency of information transmission.

[0104] exist Figure 3 In the example, the first-level title 308 of the content module 3022 includes the second-level titles of two paragraphs of the content module. Therefore, when the large language model determines the first-level title of the content module, it can refer to the second-level titles of the paragraphs included in the content module. Conversely, when the large language model determines the second-level titles of the paragraphs included in the content module, it can also refer to the first-level title of the content module.

[0105] By setting the first-level title and the second-level title, the key information in the recognition result page can be succinctly expressed, so that users no longer have to bother summarizing what the recognition result page expresses, thereby improving the user experience. In some cases, especially when the first-level title and the second-level title are determined based on the key points of the related content, users may even be able to understand the main idea of ​​the entire article by just looking at the title without browsing the specific content of each part, or they can directly locate the content of interest through the title and then read it carefully.

[0106] In some embodiments, one or more paragraphs are arranged in an ordered list. As a non-limiting embodiment, continue to refer to Figure 3 Taking content module 3022 as an example, it can be seen that the respective secondary titles of the first paragraph and the second paragraph mentioned above contain ordered serial numbers "1" and "2". Because the first paragraph and the second paragraph belong to two parallel arguments of "reasons for tomato cracking", the first paragraph and the second paragraph can be arranged in an ordered list. Of course, in the case where one or more paragraphs have other textual relationships such as a progressive relationship, a general-specific relationship, a general-specific relationship, etc., an ordered list can be used to arrange one or more paragraphs. In other embodiments, one or more paragraphs are arranged in an unordered list. Since the paragraphs using an ordered list may also contain one or more paragraphs for further developing the discussion, these paragraphs can also be arranged in an unordered list.

[0107] In some embodiments, keywords and / or key sentences in one or more paragraphs are highlighted. For example, keywords / key sentences can be words and sentences that directly affect and / or guide the user's judgment and / or operation, words and sentences that answer questions that the user is concerned about, words and sentences that need to attract the user's attention, etc. Figure 3For example, the key sentence 314 "change the watering time from the afternoon to the morning" in the paragraph in the content module 3026 is bolded to highlight the key sentence 314, which directly affects and guides the user's watering operation. Figure 3 As shown, since the first-level title 308 and the second-level title 312 are determined based on the key points, they can also be considered as keywords / key sentences, and thus are also bolded for highlighting. Highlighting keywords and / or key sentences helps users quickly locate key information from a large amount of information, thereby improving information transmission efficiency and user experience.

[0108] For illustrative purposes, a non-limiting example application of method 100 may include diagnosis of plant diseases. In this example, the species image input by the user is an image of a diseased plant. Image features are identified by inputting the image into a segmentation attention model. Since the segmentation attention model is specifically optimized for image segmentation, the segmentation attention model is able to identify and accurately segment key structures and disease areas in the image. Through efficient segmentation technology, the segmentation attention model is able to generate an accurate image mask containing pathological information as the identified image features, which can provide a good basis for subsequent diagnostic analysis. Content information can be extracted from a content database (specifically, a disease database) based on the image features identified by the segmentation attention model.

[0109] After the image segmentation is completed, the multimodal model correctly understands and processes the image and content information, such as a description of the segmentation results, an indication of the characteristics of the disease, and possible diagnostic problems. In addition, the prompt information entered by the user can help the multimodal model focus on the key information in the image and combine this information with the content information extracted from the content database (specifically, the disease database). The multimodal model can identify specific pathological features in the image, combine these features with text descriptions, and classify the disease or make diagnostic suggestions alone or in combination with the large language model and / or AIGC model. The multimodal model is used to adjust and / or supplement the content information by using its ability to process different types of data and its ability to fuse this information to make an accurate diagnosis. In addition, the adjusted and / or supplemented content information is integrated into a detailed diagnostic report through the large language model and presented to the user. This diagnostic report not only includes possible diagnostic results, but also describes in detail the diagnostic basis, the identified pathological features, and the recommended follow-up treatment plan. The report displays key image areas and segmentation results in a visual way, allowing users to quickly understand the analysis process and conclusions of the model and provide support for the user's treatment decision.

[0110] A dedicated training data set may be set for each model of the present disclosure so as to train the model to have different processing capabilities.

[0111] In some embodiments, the large language model is trained using first training data, the first training data including a combination of a first text indicating content information and a second text indicating feature information as a sample, and the first training data also including a recognition result page of a label as a sample.

[0112] In some examples, the recognition result page of the label as a sample can be an expert's full score answer created by an expert based on the corresponding first text and second text. The large language model can be trained using the first training data until the output of the large language model meets the requirements. For example, the error between the recognition result page generated by the large language model and the recognition result page as the label can be determined; in response to the error being not less than a predetermined error threshold range, the parameters of the large language model are updated based on the error; and in response to the error being less than a predetermined error threshold range or in response to the number of updates exceeding a predetermined number of thresholds, the training of the large language model is terminated and a trained large language model is obtained. The trained large language model can understand the user's characteristic information and content information so as to process the content information according to the user's characteristic information to obtain a recognition result page that meets the user's characteristics. The style of the recognition result page output by the large language model can be adjusted by adjusting the first training data.

[0113] refer to Figure 4 , exemplarily showing the first training data for the large language model. It can be seen that the first text in the sample is the content information as the processing object, which is expressed as a long string of continuous text, which will cause a huge reading burden to the user. The second text in the sample indicates the user's characteristic information, which tells the large language model that the user pays attention to plant care skills, likes ordered lists, and does not like long paragraphs. The recognition result page as the label of this sample can divide the long paragraph into two content modules-the opening introduction module and the maintenance skills introduction module, and divide the maintenance skills introduction module into multiple paragraphs arranged in an ordered list. In addition, each paragraph of the recognition result page as the label of this sample also has highlighted words and sentences to prompt the key information of the corresponding paragraph.

[0114] In some embodiments, the multimodal model is trained with second training data, the second training data including a species image as a sample, a first text indicating content information as a processing object, and a combination of data of any one or more modalities indicating reference information as a processing reference, and the second training data also includes a second text indicating content information as a processing result as a label of the sample.

[0115] In some examples, the second text as a label of the sample can be an expert's full score answer created by an expert based on the corresponding species image, the first text, and the modal data. The multimodal model can be trained using the second training data until the output of the multimodal model meets the requirements. For example, the error between the second text generated by the multimodal model and the second text as the label can be determined; in response to the error being not less than a predetermined error threshold range, the parameters of the multimodal model are updated based on the error; and in response to the error being less than a predetermined error threshold range or in response to the number of updates exceeding a predetermined number of thresholds, the training of the multimodal model is terminated and a trained multimodal model is obtained. The trained multimodal model can understand the needs of the user and process the content information according to the needs of the user to obtain content information that meets the user's needs. The content information output by the multimodal model can be adjusted by adjusting the second training data.

[0116] Exemplarily, the training process of the multimodal model may include a pre-training phase and a fine-tuning training phase, and accordingly, the second training data may include a pre-training data set and an instruction fine-tuning data set.

[0117] For example, still taking the multimodal model processing two modal data of image and text as an example. The pre-training data set may include a large amount of image-text pair data from the application field. For example, in the field of plant variety identification and / or disease diagnosis, it may include a pair of species images and text for describing the variety of the plant in the species image, and / or a pair of species images and text for describing the disease diagnosis results of the plant in the species image. These image-text pairs may be manually annotated and processed.

[0118] For example, the aforementioned CNN Transformer model can adopt contrastive learning based on plant image-text pairs in the pre-training phase. The specific training process can be as follows. For the plant image-text pairs input to the CNN Transformer model, the image encoder extracts image features, and the text encoder extracts text features. Assuming that a training batch contains N plant image-text pairs, the image encoder will extract N image features, and the text encoder will also extract N text features. The N image features and N text features are combined in pairs to obtain N 2 samples. For each image feature, there is 1 positive sample and (N-1) negative samples; for each text feature, there is 1 positive sample and (N-1) negative samples. In general, there are N positive samples and (N 2-N) negative samples. The training goal can be to maximize the similarity of N positive samples (for example, the cosine similarity of text features and image features can be directly calculated). The training process is equivalent to a multi-classification task, and the cross entropy loss can be calculated. Through contrastive learning, the visual feature expression ability of the CNN Transformer model can be enhanced, thereby improving the performance of the multimodal large language model on visual tasks, and ultimately improving the plant recognition ability of the entire multimodal model.

[0119] Through pre-training, the word segmentations encoded by the visual model in the multimodal model and the word segmentations encoded by the multimodal large language model in the multimodal model can be aligned in the semantic space, so that the multimodal large prediction model can also acquire the ability to see pictures (i.e., visual ability).

[0120] The instruction fine-tuning dataset can be set according to specific tasks. Specifically, basic tasks in multiple fields can be defined first. For example, for the plant field, basic tasks may include the diagnosis of plant diseases. Then, the user inputs the key points / requirements / questions of concern and basic information about the plant image, and obtains a set of multi-round dialogue data by constructing instructions. Similarly, all related questions in the plant field are constructed into multi-round dialogues, thereby forming an instruction fine-tuning dataset for fine-tuning training of the multimodal model, so that the multimodal model has the ability to complete various questions in the plant field, so that the input information fed back by the user can be integrated into the content information. Through the above-mentioned interactive form, the multimodal model can use prior knowledge, additional information and reasoning capabilities on the basis of the original recognition capability to achieve improved recognition accuracy and reliability.

[0121] refer to Figure 5 , exemplarily showing the second training data for the multimodal model. It can be seen that the first text in the sample is the content information as the processing object, which describes the three common causes of calcium deficiency in tomato fruit in the order of ABC, where part A is used to describe soil calcium deficiency, part B is used to describe the inability of tomato fruit to utilize calcium, and part C is used to describe insufficient watering, and preventive measures are described in the order of DEF, where part D is used to describe calcium supplementation during the critical period, part E is used to describe reasonable control of fertilization, and part F is used to describe controlled watering. The multimodal data used as a processing reference in the sample includes a watering video uploaded by the user and text indicating the climate information of the user's location, which indicates that the climate in the user's location is dry and the user waters very little. The sample also includes species images. The second text as the label of the sample can adjust the order of the text relative to the first text, moving part C to the first common cause (i.e., describing the three common causes of calcium deficiency in tomato fruit in the order of CAB), and moving part F to the first preventive measure (i.e., describing the preventive measures in the order of FDE).

[0122] In some embodiments, the AIGC model is trained with third training data, the third training data including a combination of a species image as a sample, a first text indicating content information as a processing object, and a second text indicating reference information as a processing reference, and the third training data also includes a third text indicating content information as a processing result as a label of the sample.

[0123] In some examples, the third text used as a label of the sample can be an expert's full score answer created by an expert based on the corresponding species image, the first text, and the second text. The AIGC model can be trained using the third training data until the output of the AIGC model meets the requirements. For example, the error between the third text generated by the AIGC model and the third text used as a label can be determined; in response to the error being not less than a predetermined error threshold range, the parameters of the AIGC model are updated based on the error; and in response to the error being less than a predetermined error threshold range or in response to the number of updates exceeding a predetermined number of thresholds, the training of the AIGC model is terminated and a trained AIGC model is obtained. The trained AIGC model can understand the needs of the user and process the content information according to the needs of the user to obtain content information that meets the needs of the user. The content information output by the AIGC model can be adjusted by adjusting the third training data.

[0124] refer to Figure 6 , exemplarily showing the third training data for the AIGC model. It can be seen that the first text in the sample is the content information as the processing object, which includes the G part for describing the disease and the H part for describing the cause of the disease. The second text in the sample as a processing reference includes a preset content framework for the disease diagnosis and recognition result page, which includes the disease, the cause of the disease, and the measures to prevent the disease. The sample also includes a species image. The third text as the label of the sample can supplement the I part for describing the measures to prevent the disease relative to the first text.

[0125] For example, the training method of the autoregressive language model can be used to train multimodal models, large language models, and AIGC models. The training process of the autoregressive language model is based on a large amount of text data to calculate the probability distribution of each word given a previous word sequence. Specifically, the model regards each word in the training data as a discrete random variable, and then uses the maximum likelihood estimation method to calculate the conditional probability distribution of each word given a previous word sequence, which can be used to generate and predict text sequences. Specifically, the model regards the text sequence as a random variable sequence X1, X2, ..., X T , where each random variable represents a word. The model assumes that the word at the current moment is only related to a finite number of previous words, that is, the word X at the current moment tOnly with the previous word sequence X1, X2, ..., X t-1 For (t=1, 2, ..., T), this is the Markov hypothesis. According to Bayes' theorem, the next occurrence of word x can be t+1 The probability P(x t+1 |X) is expressed as: P(x t+1 |X)=P(x t+1 |X1, X2, …, Xt). Since the probability of occurrence of each word in a text sequence is affected by the previous words, the above formula can be further expanded: P(x t+1 |X)=P(x t+1 |x t , x t-1 , …, x1), this formula means that the probability of the next word depends on the occurrence of the previous word, that is, if the previous word sequence is known, then the probability of the next word can be predicted based on the conditional probability.

[0126] For example, for a multimodal model, as described above, after receiving data of multiple modalities, the multimodal model first needs to convert these different types of data into compatible feature representations so as to fuse these different types of data. For the second training data for the multimodal model, in addition to the first text, the species image and any one or more modal data indicating the reference information can also be converted into text-like inputs, and then fused with the first text to obtain a fused text sequence. Next, the multimodal model can encode the words in the fused text sequence to map each word to a vector representation of a fixed length, model the encoded words to obtain output (for example, when the multimodal model is built based on RNN, the encoded word sequence can be input into RNN for modeling to obtain the output of RNN; when the multimodal model is built based on Transformer model, the multi-head self-attention mechanism can be used to model the encoded word sequence to capture the dependency between different positions, thereby obtaining the output of Transformer) and convert the output into the probability distribution of the next word through a related function (such as softmax function). During training, the multimodal model can utilize a cross entropy loss function so that the prediction result of the multimodal model is as close as possible to the text sequence of the second text as the label.

[0127] It can be understood that the large language model and the AIGC model can adopt a training process similar to the training process described above for the multimodal model that draws on the idea of ​​the autoregressive language model, and will not be elaborated here.

[0128] After obtaining the recognition result page, the associated tasks can be generated and displayed to the user based on the content of the recognition result page, especially the key information therein (e.g., highlighted keywords or key sentences, etc.). If a device capable of executing the task is connected, the corresponding device can also be automatically controlled to execute the task.

[0129] Specifically, in some embodiments, method 100 may include: generating a maintenance plan based on the identification result page, the maintenance plan including one or more pairs, each of the one or more pairs including one or more maintenance tasks and an identification of a maintenance device for performing the one or more maintenance tasks; and displaying the maintenance plan on a user interface.

[0130] In some examples, the maintenance plan can be generated based on key information in the recognition result page. For example, the maintenance plan can be generated based on one or more of the title, keywords and / or key sentences in the recognition result page.

[0131] The maintenance plan may include a daily maintenance plan, a therapeutic maintenance plan, etc. or a combination thereof. For example, when the identification result page does not involve the disease of the species, the daily maintenance plan of the species may be output; when the identification result page involves the disease of the species, the therapeutic maintenance plan of the species may be output, and optionally, the matching daily maintenance plan may also be output.

[0132] For illustrative purposes, a non-limiting example application of method 100 may include diagnosing a disease of a tomato plant maintained by a user. In this example, the leaves and fruits of the tomato plant in the species image input by the user begin to rot, so the recognition result page ultimately generated based on the species image may include a symptom module, a cause module, a treatment measure module, and a preventive measure module for tomatoes with rotten fruits. In order to further help the user maintain the tomato, a maintenance task may be generated based on the content in the recognition result page, especially the content highlighted in the treatment measure module (such as specific treatment timing, treatment drugs, treatment frequency, etc.). After the maintenance task is generated based on the recognition result page, since the content database also stores the identifier of the maintenance device that is communicatively coupled to the user terminal, the identifier of the maintenance device associated with the maintenance task can also be determined from the content database based on the maintenance task, thereby generating a maintenance plan. After obtaining the maintenance plan, such as Figure 2As shown, the maintenance plan can be displayed on the user interface. Thus, the user can know what kind of maintenance device (such as an irrigation device, a fertilizing device, a pruning device, a drug delivery device, a light control device, a temperature control device, a humidity control device, etc. or a combination thereof) should be used and what kind of maintenance task should be implemented to maintain the tomatoes. Maintenance tasks may include, for example, watering, spraying, fertilizing, pruning, weeding, rotating pots, sunshine, shading, adjusting temperature, adjusting humidity, applying pesticides, and applying fungicides. Specifically, the maintenance task may also include various parameters of the task, such as the time, interval, and amount of watering, the dosage, time, and interval of fertilization, the location of pruning, the dosage and location of pesticide spraying, etc.

[0133] In some embodiments, the method 100 may include: controlling the corresponding maintenance device in each of the one or more pairs in the maintenance plan according to the identification of the maintenance device to complete one or more maintenance tasks in the pair.

[0134] Since the maintenance device usually has a communication function, commands can be transmitted to the maintenance device (e.g., via Bluetooth protocol, Zigbee protocol, etc.). For example, in the above example, the maintenance plan may include two pairs of {fruit pruning task and leaf pruning task, pruning device identification} and {pesticide spraying task, drug delivery device identification}. According to the identification, commands instructing the execution of pruning tasks and spraying tasks are sent to the corresponding pruning devices and spraying devices respectively, so as to control the pruning devices and spraying devices to automatically complete the pruning tasks and spraying tasks, thereby reducing the maintenance burden of the user and improving the maintenance efficiency.

[0135] In addition to automatically executing the maintenance plan, in some embodiments, after displaying the maintenance plan, the user may be further asked whether to confirm the execution of the maintenance plan, and after the user confirms the execution of the maintenance plan, the corresponding maintenance device is controlled according to the maintenance plan to complete the corresponding maintenance task. Exemplarily, after the user confirms the execution of the maintenance plan, the user's characteristic information, such as maintenance level information, maintenance history information, etc., may be updated based on the maintenance plan. It is also possible to update the user's characteristic information, such as maintenance level information, maintenance history information, etc., based on the execution data returned by the maintenance device after the maintenance plan is executed. In this way, when the user's species image is acquired again, the recognition result page and the corresponding maintenance plan can be generated in combination with the updated user's characteristic information.

[0136] The present disclosure also provides an electronic device in another aspect. Figure 7 , which shows a schematic block diagram of an electronic device 400 according to some embodiments of the present disclosure. Figure 7As shown, the electronic device 400 includes (one or more) processors 402 and a memory 404 storing computer executable instructions, which, when executed by (one or more) processors 402, cause (one or more) processors 402 to perform the method for processing recognition results described in any of the foregoing embodiments of the present disclosure. (One or more) processors 402 may be, for example, a central processing unit (CPU) of the electronic device 400. (One or more) processors 402 may be any type of general-purpose processor, or may be a processor specifically designed to process recognition results, such as an application-specific integrated circuit ("ASIC"). The memory 404 may include various computer-readable media accessible by (one or more) processors 402. In various embodiments, the memory 404 described herein may include volatile and non-volatile media, removable and non-removable media. For example, the memory 404 may include any combination of the following: random access memory ("RAM"), dynamic RAM ("DRAM"), static RAM ("SRAM"), read-only memory ("ROM"), flash memory, cache memory, and / or any other type of non-transient computer-readable media. The memory 404 may store instructions that, when executed by the processor 402, enable the processor 402 to perform the method for processing the recognition result according to any of the foregoing embodiments of the present disclosure. In some embodiments, the electronic device 400 may be implemented as a smart phone, a smart camera, a computer, and the like.

[0137] The present disclosure also provides a non-transitory storage medium having computer executable instructions stored thereon, which, when executed by a computer, enables the computer to execute the method for processing recognition results according to any of the aforementioned embodiments of the present disclosure.

[0138] The present disclosure also provides a computer program product, which may include instructions, and when the instructions are executed by a processor, the method for processing recognition results described in any of the foregoing embodiments of the present disclosure may be implemented. The instructions may be any set of instructions to be executed directly by one or more processors, such as machine code, or any set of instructions to be executed indirectly, such as a script. The instructions may be stored in an object code format for direct processing by one or more processors, or stored in any other computer language, including a script or collection of independent source code modules that are interpreted on demand or compiled in advance.

[0139] Figure 8A schematic block diagram of a computer system 500 on which embodiments of the present disclosure may be implemented is shown. The computer system 500 includes a bus 502 or other communication mechanism for transmitting information, and a processing device 504 coupled to the bus 502 for processing information. The computer system 500 also includes a memory 506 coupled to the bus 502 for storing instructions to be executed by the processing device 504, which may be a random access memory (RAM) or other dynamic storage device. The memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processing device 504. The computer system 500 also includes a read-only memory (ROM) 508 or other static storage device coupled to the bus 502 for storing static information and instructions for the processing device 504. A storage device 510 such as a magnetic disk or optical disk is provided and coupled to the bus 502 for storing information and instructions. The computer system 500 may be coupled to an output device 512 via the bus 502 for providing output to a user, such as, but not limited to, a display (such as a cathode ray tube (CRT) or a liquid crystal display (LCD)), a speaker, etc. An input device 514 such as a keyboard, a mouse, a microphone, etc. is coupled to the bus 502 for transmitting information and command selections to the processing device 504. The computer system 500 may perform embodiments of the present disclosure. Consistent with certain implementations of the present disclosure, the results are provided by the computer system 500 in response to the processing device 504 executing one or more sequences of one or more instructions contained in the memory 506. Such instructions may be read into the memory 506 from another computer-readable medium such as the storage device 510. The execution of the sequence of instructions contained in the memory 506 causes the processing device 504 to perform the methods described herein. Alternatively, hard-wired circuitry may be used instead of software instructions or in combination with software instructions to implement the present teachings. Therefore, implementations of the present disclosure are not limited to any specific combination of hardware circuitry and software. In various embodiments, computer system 500 can be connected to one or more other computer systems like computer system 500 across a network via network interface 516 to form a networked system. The network can include a private network or a public network such as the Internet. In a networked system, one or more computer systems can store data and supply data to other computer systems. The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to processing device 504 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks or disks such as storage device 510. Volatile media include dynamic memories such as memory 506. Transmission media include coaxial cables, copper wires, and optical fibers, including wiring that includes bus 502.Common forms of computer-readable media or computer program products include, for example, floppy disks, flexible disks, hard disks, tapes, or any other magnetic media, CD-ROMs, digital video disks (DVDs), Blu-ray disks, any other optical media, thumb drives, memory cards, RAMs, PROMs and EPROMs, flash EPROMs, any other memory chips or boxes, or any other tangible media from which a computer can read. Various forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to processing device 504 for execution. For example, instructions may initially be carried on a disk of a remote computer. The remote computer may load instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 500 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector coupled to bus 502 may receive the data carried in the infrared signal and place the data on bus 502. Bus 502 carries the data to memory 506, and processing device 504 retrieves instructions from memory 506 and executes the instructions. For example, instructions received by memory 506 may be stored on storage device 510 either before or after execution by processing device 504 .

[0140] Fig. 9 A schematic block diagram of a maintenance system 600 according to some embodiments of the present disclosure is shown. The maintenance system 600 includes an electronic device 610, which includes a processor 612 and a memory 614 coupled to the processor 612 and storing instructions. The electronic device 610 may, for example, take the form of but not limited to the aforementioned electronic device 400 or computer system 500, and may be implemented as, for example, but not limited to, a smart phone, a smart camera, a computer, etc. The memory 614 may store instructions that, when executed by the processor 612, cause the processor 612 to execute the method for processing recognition results described in any of the aforementioned embodiments of the present disclosure. The maintenance system 600 also includes (one or more) maintenance devices (e.g., 6201, 6202, ..., 620 n ). It is configured to perform maintenance tasks in response to receiving a command from the electronic device 610.

[0141] Specifically, in some embodiments, the instructions stored in the memory 614, when executed by the processor 612, may cause the processor 612 to: obtain a species image from a user and characteristic information of the user; identify image features of the species image, and extract content information from a content database based on the identified image features; input the content information and characteristic information into a large language model to generate a recognition result page from the content information based on the characteristic information; generate a maintenance plan based on the recognition result page, the maintenance plan including one or more pairs, each of the one or more pairs including one or more maintenance tasks and an identifier of a maintenance device for performing the one or more maintenance tasks; send a maintenance request to a corresponding maintenance device (e.g., 6201, 6202, ..., 6203) based on the identifier of the maintenance device in each of the one or more pairs in the maintenance plan. n ) transmits commands to control corresponding maintenance devices (e.g., 6201, 6202, ..., 620 n ) to complete one or more maintenance tasks in the pair.

[0142] In some embodiments, the electronic device 610 includes a user interface (not shown). For example, a recognition result page may be displayed on the user interface, and / or a maintenance plan may be displayed on the user interface.

[0143] In some embodiments, the maintenance device (e.g., 6201, 6202, ..., 620 n ) is configured to transmit execution data to the electronic device 610 in response to the execution of the maintenance task. Thus, the instructions stored in the memory 614 may include instructions that, when executed by the processor 612, cause the processor 612 to perform the following operations: update the user's characteristic information based on the execution data received from the maintenance device. In this way, when the user's species image is acquired again, the identification result page and the corresponding maintenance plan can be generated in combination with the updated user's characteristic information.

[0144] In some embodiments, the maintenance system 600 may include a camera 630 communicatively coupled to the electronic device 610. The camera 630 may be configured to capture species images and transmit the captured species images to the electronic device 610. Thus, the instructions stored in the memory 614 may include instructions that, when executed by the processor 612, cause the processor 612 to perform the following operations: generate an identification result page based on the species image received from the camera 630.

[0145] For example, the species image received from the camera 630 may be input into the aforementioned multimodal model for processing. For example, as mentioned above, close-up images or videos of one or more characteristic parts of the target species may be required. In this case, the user may not need to input these close-up images or videos, but the camera 630 may automatically acquire these close-up images or videos. Alternatively, when the species image from the user cannot be recognized due to various reasons such as insufficient clarity, the user may not need to re-input the species image, but the camera 630 may automatically acquire the species image. In other words, the images captured by the camera 630 may be used to assist in generating recognition results.

[0146] The camera 630 may be any suitable imaging device for monitoring a target species. The target species may be positioned in the field of view of the camera 630. In some embodiments, the instructions stored in the memory 614, when executed by the processor 612, may cause the processor 612 to: obtain a species image from the camera 630 and characteristic information of a user; identify image features of the species image, and extract content information from a content database based on the identified image features; input the content information and characteristic information into a large language model to generate a recognition result page from the content information based on the characteristic information; generate a maintenance plan based on the recognition result page, the maintenance plan including one or more pairs, each of the one or more pairs including one or more maintenance tasks and an identification of a maintenance device for performing the one or more maintenance tasks; send a maintenance message to a corresponding maintenance device (e.g., 6201, 6202, ..., 6203) based on the identification of the maintenance device in each of the one or more pairs in the maintenance plan; n ) transmits commands to control corresponding maintenance devices (e.g., 6201, 6202, ..., 620 n ) to complete one or more maintenance tasks in the pair. That is, the images captured by the camera 630 can be used to autonomously generate recognition results and automatically execute maintenance plans to achieve fully automatic monitoring and maintenance of the target species.

[0147] Various embodiments of the maintenance system 600 can be similarly referred to any embodiment of the aforementioned aspects of the present disclosure, and will not be described in detail here.

[0148] One or more exemplary embodiments of the present disclosure are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0149] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present disclosure does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a game console, a tablet computer, a wearable device or any combination thereof.

[0150] The terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, product or apparatus. In the absence of further restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or apparatus comprising the elements. For example, if the words "first", "second" and the like are used to indicate names, they do not indicate any particular order.

[0151] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more embodiments of the present disclosure, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The present disclosure is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a particular manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0154] Those skilled in the art should understand that one or more embodiments of the present disclosure may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0155] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0156] The same or similar parts between the various embodiments of the present disclosure can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. In the description of the present disclosure, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", "exemplary", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present disclosure, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present disclosure and the features of the different embodiments or examples without contradiction.

[0157] In addition, when used in the present disclosure, the words "herein," "above," "below," "hereunder," "above," and words of similar meaning shall refer to the present disclosure as a whole rather than to any particular portion of the present disclosure. Furthermore, unless expressly stated otherwise or understood otherwise in the context of use, conditional language used herein, such as "may," "might," "for example," "such as," and the like, is generally intended to express that certain embodiments include, while other embodiments do not include, certain features, elements, and / or states. Thus, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or states in any way, or whether such features, elements, and / or states are included or performed in any particular embodiment.

[0158] The above description is only an example of one or more embodiments of the present disclosure, and is not intended to limit one or more embodiments of the present disclosure. For those skilled in the art, one or more embodiments of the present disclosure may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims.

Claims

1. A method for processing a recognition result, comprising: Obtaining species images from a user and characteristic information of the user; identifying image features of the species image, and extracting content information from a content database based on the identified image features; inputting the content information and the feature information into a large language model to generate a recognition result page from the content information based on the feature information; The recognition result page provided by the large language model is displayed on a user interface.

2. The method according to claim 1, wherein: The characteristic information includes first characteristic information obtained through historical data of the user.

3. The method according to claim 2, wherein: The first characteristic information includes at least one of the following: Property characteristic information, the property characteristic information including maintenance level information; Operation characteristic information, the operation characteristic information including maintenance history information.

4. The method according to claim 1, wherein: The characteristic information includes second characteristic information obtained through interaction data of the user.

5. The method according to claim 4, wherein: The second characteristic information includes demand characteristic information, and the demand characteristic information includes one or more of focus content information, detail preference information, typesetting preference information, and reading habit information.

6. The method according to claim 1, comprising: Obtaining one or more of the user's location information, time information, weather information, and climate information; inputting the species image, the one or more pieces of information, and the content information into a multimodal model to adjust the content information based on the species image and the one or more pieces of information; as well as The adjusted content information and the characteristic information are input into the large language model to obtain the recognition result page.

7. The method according to claim 1, comprising: displaying interactive questions on the user interface regarding identification of the image of the species; receiving a user input including a response to the interactive question, the user input including at least one of an image, text, audio, or video; inputting the species image, the user input, and the content information into a multimodal model to adjust the content information based on the species image and the user input; as well as The adjusted content information and the characteristic information are input into the large language model to obtain the recognition result page.

8. The method according to claim 1, comprising: Inputting the species image, the content information and the preset content framework into an artificial intelligence generated content AIGC model to supplement the content information, wherein the supplemented content information includes the missing content of the content information compared with the preset content framework; as well as The supplemented content information and the feature information are input into the large language model to obtain the recognition result page.

9. The method according to claim 1, wherein: The characteristic information includes attention content information, and the method includes: Inputting the content information of interest, the species image, and the content information into an AIGC model to supplement the content information, wherein the supplemented content information includes the missing content of the content information compared to the content information of interest; and The supplemented content information and the feature information are input into the large language model to obtain the recognition result page.

10. The method according to claim 1, wherein: The characteristic information includes detailed preference information, and the method includes: Inputting the detail preference information, the species image, and the content information into an AIGC model to regenerate the content information, wherein the regenerated content information has a detail level that matches the detail preference information; and The regenerated content information and the feature information are input to the large language model to obtain the recognition result page.

11. The method according to claim 1, wherein: The recognition result page includes one or more content modules.

12. The method according to claim 11, wherein: The one or more content modules are divided according to themes, and the recognition result page also includes a primary dividing line between every two adjacent content modules in the one or more content modules.

13. The method according to claim 12, wherein: The recognition result page also includes a first-level title located before each content module of the one or more content modules.

14. The method according to claim 13, wherein: The first-level title is determined based on the summary of the content module, or based on the main points of the content module.

15. The method according to claim 11, wherein: Each of the one or more content modules includes one or more paragraphs divided according to contextual relationships, and the recognition result page also includes a secondary dividing line between every two adjacent paragraphs of the one or more paragraphs.

16. The method according to claim 15, wherein: The recognition result page further includes a secondary heading located before each of the one or more paragraphs.

17. The method according to claim 16, wherein: The secondary heading is determined based on the summary of the paragraph or based on the main points of the paragraph.

18. The method according to claim 15, wherein: Key words and / or key sentences in the one or more paragraphs are highlighted.

19. The method according to claim 15, wherein: The one or more paragraphs are arranged in an ordered list or an unordered list.

20. The method according to claim 11, wherein: The recognition result page does not include a first-level title before a content module located at the front of the recognition result page among the one or more content modules.

21. The method according to claim 1, wherein: The large language model is trained using first training data including a combination of a first text indicating content information and a second text indicating feature information as samples, and the first training data also includes a recognition result page as a label of the sample.

22. The method according to claim 6 or 7, wherein: The multimodal model is trained using second training data, which includes a species image as a sample, a first text indicating content information as a processing object, and a combination of data of any one or more modalities indicating reference information as a processing reference, and the second training data also includes a second text indicating content information as a processing result as a label of the sample.

23. The method according to claim 8, 9 or 10, wherein: The AIGC model is trained using third training data, the third training data including a combination of a species image as a sample, a first text indicating content information as a processing object, and a second text indicating reference information as a processing reference, and the third training data also includes a third text indicating content information as a processing result as a label of the sample.

24. The method according to claim 1, comprising: generating a maintenance plan according to the recognition result page, the maintenance plan comprising one or more pairs, each of the one or more pairs comprising one or more maintenance tasks and an identification of a maintenance device for performing the one or more maintenance tasks; The maintenance plan is displayed on the user interface.

25. The method according to claim 24, comprising: According to the identification of the maintenance device in each of the one or more pairs in the maintenance plan, the corresponding maintenance device is controlled to complete one or more maintenance tasks in the pair.

26. An electronic device comprising: one or more processors; as well as A memory storing computer executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform the method for processing recognition results according to any one of claims 1 to 25.

27. A non-transitory storage medium having computer executable instructions stored thereon, which, when executed by a computer, causes the computer to perform the method for processing a recognition result according to any one of claims 1 to 25.

28. A computer program product, comprising instructions, which, when executed by a processor, implement the method for processing a recognition result according to any one of claims 1 to 25.

29. A maintenance system comprising: An electronic device comprising a processor and a memory coupled to the processor and storing instructions, the instructions, when executed by the processor, causing the processor to: obtaining a species image from a user and characteristic information of the user, identifying image features of the species image, and extracting content information from a content database based on the identified image features, inputting the content information and the feature information into a large language model to generate a recognition result page from the content information based on the feature information, generating a maintenance plan according to the recognition result page, the maintenance plan comprising one or more pairs, each of the one or more pairs comprising one or more maintenance tasks and an identification of a maintenance device for performing the one or more maintenance tasks, transmitting, according to the identification of the maintenance device in each of the one or more pairs in the maintenance plan, a command to the corresponding maintenance device to control the corresponding maintenance device to complete one or more maintenance tasks in the pair; as well as A maintenance device is communicatively coupled to the electronic device, and is configured to perform a maintenance task in response to receiving a command from the electronic device.

30. The maintenance system according to claim 29, wherein: The maintenance device is configured to transmit execution data to the electronic device in response to execution of the maintenance task, The instructions include instructions that, when executed by the processor, cause the processor to perform the following operations: The characteristic information of the user is updated based on the execution data received from the maintenance device.

31. The maintenance system according to claim 29, comprising: a camera communicatively coupled to the electronic device, the camera being configured to capture an image of the species and transmit the captured image of the species to the electronic device, The instructions include instructions that, when executed by the processor, cause the processor to perform the following operations: The identification result page is generated based on the species image received from the camera.

Images