Language model dynamic processing method and system and medium

Abstract

The invention provides a language model dynamic processing method and system and a medium, and relates to the data processing technology, the method comprises the steps that after it is judged that a user selects to carry out language model dynamic processing on a first file in a preset mode, the first file is segmented according to the content to generate a plurality of content areas, and the first file is an image; generating a path recording layer corresponding to the first file; after it is judged that the user triggers any pixel point in the path recording layer, the dynamic interaction content is determined based on the time-space relation of the triggered pixel point; and determining at least one content or at least one content region output language result based on the dynamic interaction content. According to the method, dynamic data can be intelligently generated in combination with the dynamic interaction behavior of the user.

Description

Technical Field

[0001] The present invention relates to data processing card technology, and in particular to a language model dynamic processing method, system and medium. Background Art

[0002] Driven by modern information technology, visual information processing has become the focus of attention in various industries. With the popularization of smart devices, users expect to extract the required information from complex image data through simple interactions. For example, in the field of autonomous driving, the system needs to analyze road images in real time to make decisions; in e-commerce, users want to obtain detailed descriptions of products through pictures. This trend has prompted developers to explore more intelligent image understanding and natural language generation technologies to enhance the interactive experience between images and users.

[0003] Existing visual processing technologies mainly rely on static image recognition and predefined text generation. Although these technologies have made progress in some aspects, they still have many shortcomings. For example, traditional methods are usually limited to recognizing certain specific objects in the image, and it is difficult to dynamically capture the user's interactive behavior on the image. In addition, the generated text is often a fixed pattern and lacks the flexibility to match the user's personalized operations. This limitation cannot meet the user's demand for dynamic interaction.

[0004] Therefore, how to combine users' dynamic interactive behaviors and intelligently generate dynamic data has become an urgent problem that needs to be solved. Summary of the invention

[0005] The embodiments of the present invention provide a language model dynamic processing method, system and medium, which can intelligently generate dynamic data in combination with the dynamic interactive behavior of users.

[0006] A first aspect of an embodiment of the present invention provides a language model dynamic processing method, comprising: After determining that the user chooses to perform language model dynamic processing on a first file of a preset modality, the first file is segmented according to content to generate a plurality of content areas, wherein the first file is an image; Generate a path record layer corresponding to the first file; After determining that a user triggers any pixel point in the path record layer, the dynamic interaction content is determined based on the spatiotemporal relationship of the triggered pixel point; At least one content or at least one content area outputs a language result based on the dynamic interactive content.

[0007] Optionally, the generating multiple content areas by segmenting the first file according to the content includes: Performing graphic comparison on the first file based on a preset element database to determine a first image element in the first file that passes the comparison; After coordinate processing of the first file, a coordinate interval of each first image element is obtained, and after processing the coordinate interval, a content coordinate interval of the first image element is generated; All pixels within the content coordinate interval are regarded as the corresponding content area.

[0008] Optionally, the processing of the coordinate interval to generate a content coordinate interval of the first image element includes: Get the horizontal and vertical extreme values ​​of the coordinate interval; The maximum value among the extreme values ​​of the horizontal coordinate and the maximum value among the extreme values ​​of the vertical coordinate are increased by a preset value, and the minimum value among the extreme values ​​of the vertical coordinate and the minimum value among the extreme values ​​of the vertical coordinate are decreased by a preset value to obtain a content coordinate interval.

[0009] Optionally, generating a path record layer corresponding to the first file includes: Get the coordinates of each pixel in the first file; Establish the layer coordinates corresponding to the coordinates of each pixel point and generate a path record layer.

[0010] Optionally, after determining that the user triggers any pixel point in the path record layer, determining the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel point includes: Acquire the first pixel point in the trigger path record layer in real time, and extract the second pixel points of all historical triggers within a preset time period; Generate a pixel point sequence based on the time sequence of the first pixel point and the second pixel point, and sort the first pixel point and the second pixel point from near to far in time based on the pixel point sequence, so that the first pixel point and the second pixel point have corresponding sorting numbers respectively; The first pixel point and the second pixel point are traced in sequence based on the sequence number to determine the dynamic interactive content.

[0011] Optionally, the sequentially tracking the first pixel point and the second pixel point based on the sequence number to determine the dynamic interactive content includes: Connecting the first pixel point and the second pixel point in sequence based on the sequence number to generate a tracing route; Extracting the graph formed by the tracing route and comparing it with the preset instruction database, determining the first instruction that passes the comparison, each instruction having a preset response strategy; The dynamic interactive content corresponding to the current instruction is extracted based on the first instruction.

[0012] Optionally, extracting the dynamic interactive content corresponding to the current instruction based on the first instruction includes: If the preset response strategy of the first instruction is a positioning strategy, extracting the horizontal coordinate extreme values ​​and the vertical coordinate extreme values ​​of all the first pixel points and the second pixel points in the tracing route; Determine the middle value of the abscissa and the middle value of the ordinate based on the extreme value of the abscissa and the extreme value of the ordinate to obtain a positioning middle point; Determine the initial segmentation area of ​​the path record layer where the positioning intermediate point is located, and output the corresponding output language result according to the initial segmentation area where the positioning intermediate point is located.

[0013] Optionally, the initial segmentation area of ​​the path record layer is obtained by the following steps, including: Get the number of horizontal and vertical pixels in the path record layer; The number of horizontal pixel points is equally divided according to a preset horizontal equal division value to obtain a plurality of horizontal equally divided points, and the number of vertical pixel points is equally divided according to a preset vertical equal division value to obtain a plurality of vertical equally divided points; Based on the horizontal and vertical dividing points, the dividing lines are generated to divide the path record layer into equal parts to obtain the initial segmentation area of ​​the path record layer, and corresponding language orientation information is added to each initial segmentation area. The output language result includes the language orientation information.

[0014] Optionally, extracting the dynamic interactive content corresponding to the current instruction based on the first instruction includes: If the preset response strategy of the first instruction is a content recognition strategy, extracting the horizontal coordinate extreme values ​​and the vertical coordinate extreme values ​​of all the first pixel points and the second pixel points in the tracing route; Generate a selected area based on the abscissa extreme value and the ordinate extreme value; The pixels in the selected area are mapped to the first file to determine the dynamic interactive content.

[0015] Optionally, the determining, based on the dynamic interactive content, at least one content or at least one content area to output a language result includes: Acquire a first image element that intersects with the coordinate position of the pixel point in the selected area, determine the number of the pixel points of the intersection to obtain a first intersection number, and count a first total number of the pixel points in the first image element; Calculating a ratio of the first intersection quantity to the first total quantity to obtain a first proportion, and if the first proportion is greater than a preset value, using the content of the corresponding first image element as the output language result; If it is determined that all first proportions are less than or equal to a preset value, a content area is determined to output a language result.

[0016] Optionally, determining a content area output language result includes: Obtaining a content area that intersects with the pixel coordinate positions of the selected area, and determining the number of pixels of the intersection to obtain a second intersection number; If it is determined that there is only one content area that has an intersection with the selected area, the corresponding content area will be determined to output the language result; If it is determined that there are multiple content areas that have intersections with the selected area, then the content area with the second largest number of intersections is selected to output the language result, and the output language result includes all the content in the content area.

[0017] A second aspect of an embodiment of the present invention provides a language model dynamic processing system, including: a judgment module, configured to generate a plurality of content regions by segmenting the first file according to the content after judging that the user chooses to perform language model dynamic processing on the first file of the preset modality, wherein the first file is an image; A generation module, used for generating a path record layer corresponding to the first file; A determination module, for determining the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel points after determining that the user has triggered any pixel point in the path record layer; The output module is used to determine at least one content or at least one content area based on the dynamic interactive content and output a language result.

[0018] According to a third aspect of an embodiment of the present invention, a storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the method of the first aspect of the present invention and various possible designs of the first aspect.

[0019] The present invention can accurately determine the first image element in the image by performing graphic comparison on the first file based on the preset element database, and then generate the content coordinate interval through coordinate processing, and then determine the pixel points in the content coordinate interval as the content area. This series of operations enables the image to be finely divided into multiple content areas, and the user can interact with the path record layer to obtain information about the area of ​​interest in a targeted manner. The accuracy and efficiency of information acquisition are improved, and the problem that traditional technology cannot accurately locate the focus of user attention is effectively solved.

[0020] The present invention obtains the first pixel point in the trigger path record layer in real time, extracts the second pixel point of the historical trigger within a preset time period, generates and sorts the pixel point sequence according to the time sequence, and then determines the first instruction by comparing it with the preset instruction database through tracing processing. This process can deeply understand the user's operation intention and generate corresponding language results according to different instruction response strategies. For example, when the user draws a circular trajectory on the image, the system can accurately identify it as a "content recognition" instruction and provide accurate language feedback accordingly, which greatly enhances the interactive experience between the user and the image and makes up for the defect that the existing system does not understand the user's interactive behavior.

[0021] When determining the output language result, the present invention obtains the first image element that has an intersection with the coordinate position of the pixel points in the selected area according to the content recognition strategy, calculates the ratio of the number of intersection pixels to the total number of pixels in the first image element, and determines whether to use the corresponding image element content as the output language result; if all ratios do not meet the conditions, a content area output language result is determined. This intersection analysis of the image element and the selected area and the comprehensive judgment of the content area can accurately output the language result closely related to the user operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for dynamic processing of a language model provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a tracking route provided by an embodiment of the present invention; Figure 3 It is a structural diagram of a language model dynamic processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 , is a flow chart of a language model dynamic processing method provided by an embodiment of the present invention, including: S1, after determining that the user chooses to perform language model dynamic processing on a first file of a preset modality, the first file is segmented according to content to generate multiple content areas, and the first file is an image.

[0025] When the user chooses to dynamically process the language model for the first file of the preset modality, the system will quickly start the content segmentation process. This operation is the basis for subsequent accurate interaction and language result output. By dividing the image into multiple areas according to content, it provides users with a more targeted way to obtain information.

[0026] In some embodiments, the step of segmenting the first file according to the content to generate a plurality of content areas includes: S11, performing graphic comparison on the first file based on a preset element database to determine a first image element in the first file that passes the comparison.

[0027] The system will import the first file and perform a deep graphic comparison with the preset element database. The preset element database covers a large number of common image element templates, from tables, chairs, and electrical appliances in daily life to trees, flowers, and plants in natural scenes, and various geometric figures. During the comparison process, the system uses advanced image recognition algorithms to match each part of the image with the templates in the database one by one. For example, for an image of a "park scene", the system successfully identified the first image elements such as "bench", "tree", and "flower" after comparison. This graphic comparison technology can quickly and accurately locate various elements in the image, providing a key basis for subsequent coordinate processing and content area determination.

[0028] S12, obtaining a coordinate interval of each first image element after coordinate processing of the first file, and generating a content coordinate interval of the first image element after processing the coordinate interval.

[0029] After completing the image element recognition, the system coordinates the first file. In a digital image, each pixel corresponds to a unique horizontal and vertical coordinate. By establishing a coordinate system for the entire image, the coordinate range of each first image element can be accurately obtained. Taking the image element "bench" as an example, assuming that after coordinate processing, its horizontal coordinate range is [100, 200] and the vertical coordinate range is [150, 250].

[0030] The step of processing the coordinate interval to generate a content coordinate interval of the first image element includes: S121, obtaining the horizontal coordinate extreme value and the vertical coordinate extreme value of the coordinate interval.

[0031] In order to ensure that all information of the image elements can be fully covered and to avoid partial information loss due to boundary interception, the system will further process the acquired coordinate interval to generate a content coordinate interval.

[0032] S122, increasing the maximum value among the extreme values ​​of the horizontal coordinate and the maximum value among the extreme values ​​of the vertical coordinate by a preset value, and decreasing the minimum value among the extreme values ​​of the vertical coordinate and the minimum value among the extreme values ​​of the vertical coordinate by a preset value, to obtain a content coordinate interval.

[0033] For example, the system will increase the maximum value (here 200) of the horizontal coordinate by a preset value (assuming 10) to obtain 210; and decrease the minimum value (100) of the horizontal coordinate by a preset value (10) to obtain 90. Similarly, the maximum value (250) of the vertical coordinate is increased by a preset value (10) to obtain 260; and the minimum value (150) of the vertical coordinate is decreased by a preset value (10) to obtain 140. Finally, the content coordinate range of "bench" is [90, 210] horizontal coordinate range and [140, 260] vertical coordinate range.

[0034] S13: taking all pixel points within the content coordinate interval as corresponding content areas.

[0035] The content coordinate interval processed through the above steps can more comprehensively include the relevant pixel points of the image elements, providing a more reliable range definition for the subsequent determination of the content area.

[0036] S2, generating a path record layer corresponding to the first file.

[0037] In the language model dynamic processing method of the present invention, generating a path record layer corresponding to the first file is an extremely important step. It provides an indispensable basis for determining the dynamic interaction content based on the user's interaction behavior. By recording the pixel point information triggered during the user's interaction with the image, the system can deeply analyze the user's operation trajectory, thereby generating accurate and personalized language results.

[0038] In some embodiments, generating a path record layer corresponding to the first file includes: S21, obtaining the coordinates of each pixel point in the first file.

[0039] In the scope of digital images, each image is composed of a large number of pixels, which are the basic carriers of image information. The system will traverse each pixel in the first file (i.e., image), and use mature image analysis technology to accurately obtain the horizontal and vertical coordinate values ​​of each pixel in the image coordinate system. Taking a common color image with a resolution of 1920×1080 as an example, the system will start from the first pixel in the upper left corner of the image, and read its horizontal coordinate as 0 and vertical coordinate as 0 in sequence; then read the horizontal coordinate of the adjacent pixel on the right as 1 and the vertical coordinate as 0, and so on, until the last pixel in the lower right corner of the image is traversed, that is, the horizontal coordinate is 1919 and the vertical coordinate is 1079. Through this comprehensive and detailed traversal method, the system can fully grasp the position information of each pixel in the image.

[0040] S22, establishing layer coordinates corresponding to the coordinates of each pixel point, and generating a path record layer.

[0041] After successfully obtaining the coordinates of each pixel in the first file, the system will establish a corresponding layer coordinate system for these coordinates. This layer coordinate system is interrelated and one-to-one with the coordinate system of the image itself. Its essence is to create a mapping record for the coordinates of each pixel in an independent layer structure. This path record layer is like a transparent "cover layer" that fits tightly on the original image, ready to record every interaction between the user and the image, providing key data support for subsequent in-depth analysis of user behavior and determination of dynamic interactive content.

[0042] S3, after determining that the user triggers any pixel point in the path record layer, determine the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel point.

[0043] In the language model dynamic processing system of the present invention, step S3 is in the core interaction analysis link, which undertakes the key task of converting the interaction operation between the user and the path record layer into dynamic interaction content that can be understood by the system.

[0044] In some embodiments, after determining that a user triggers any pixel point in the path record layer, determining the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel point includes: S31, acquiring the first pixel point in the trigger path record layer in real time, and extracting the second pixel points of all historical triggers within a preset time period.

[0045] The system has the ability to monitor the path record layer in real time. Once a user trigger operation is detected, it will quickly capture the first pixel point in the triggered path record layer. For example, when a user clicks a certain position on the image interface, the pixel point in the path record layer corresponding to the position is the first pixel point, assuming its coordinates are (150, 200). At the same time, the system will trace back all the second pixel points triggered historically within the preset time period. The preset time period can be flexibly set according to the actual application scenario and user interaction habits, such as the common setting of the past 5 seconds or 10 seconds. During this time period, the system will retrieve and extract all the pixel point information previously triggered by the user. For example, in the past 5 seconds, the user has also triggered pixels with coordinates of (120, 180), (130, 190), etc. These pixels are the second pixels. In this way, the system comprehensively collects the key information of the user's interaction with the image over a period of time.

[0046] S32, generating a pixel point sequence based on the time sequence of the first pixel point and the second pixel point, and sorting the first pixel point and the second pixel point from near to far in time based on the pixel point sequence, so that the first pixel point and the second pixel point have corresponding sorting numbers respectively.

[0047] Based on the acquired first pixel and second pixel, the system generates a pixel sequence according to their triggering timing. The system arranges these pixels in chronological order to form an ordered sequence. Then, the system further processes this pixel sequence and sorts the first pixel and the second pixel in order from near to far in time. In this way, each pixel is assigned a corresponding sorting number, which clearly shows the user's operation sequence. For example, the latest triggered first pixel (150, 200) has a sorting number of 1, while the previously triggered (130,190) has a sorting number of 2, (120, 180) has a sorting number of 3, and so on. This sorting method provides a clear order basis for subsequent tracing processing and command recognition, allowing the system to accurately restore the user's interaction process.

[0048] S33: trace the first pixel point and the second pixel point in sequence based on the sequence number to determine dynamic interactive content.

[0049] The step of sequentially tracking the first pixel point and the second pixel point based on the sequence number to determine the dynamic interactive content includes: S331, connecting the first pixel point and the second pixel point in sequence based on the sorting number to generate a tracing route.

[0050] The system connects the first pixel and the second pixel in sequence based on the sorting number to generate a trace route. Taking the previous pixel as an example, the system will start from the pixel with sorting number 1 (150, 200), and sequentially connect the pixel with sorting number 2 (130, 190) and the pixel with sorting number 3 (120, 180) to form a continuous track. This trace route intuitively reflects the user's operation trajectory on the image.

[0051] S332, extracting the graph formed by the tracing route and performing a graph comparison with a preset instruction database to determine the first instruction that passes the comparison, each instruction having a preset response strategy.

[0052] After the tracking route is generated, the system will extract the graphic features formed by the tracking route and compare the graphics with the preset instruction database. The preset instruction database pre-stores various common user operation instruction graphic templates and their corresponding instruction meanings. For example, a circle may correspond to the "content recognition" instruction, a straight line may correspond to the "positioning" instruction, and so on. The system uses advanced graphic matching algorithms to compare the tracking route graphics with the templates in the database one by one to determine the first instruction that passes the comparison. In this way, the system can accurately identify the user's operation intention and provide direction for subsequent responses.

[0053] S333: Extract the dynamic interactive content corresponding to the current instruction based on the first instruction.

[0054] This solution will analyze the first instruction to obtain the corresponding dynamic interactive content.

[0055] The extracting the dynamic interactive content corresponding to the current instruction based on the first instruction includes: S3331: If the preset response strategy of the first instruction is a positioning strategy, the horizontal coordinate extreme values ​​and the vertical coordinate extreme values ​​of all the first pixel points and the second pixel points in the tracing route are extracted.

[0056] When the system determines that the preset response strategy of the first instruction is the positioning strategy, it will first conduct a detailed analysis of all the first pixel points and second pixel points involved in the tracing route. The system will traverse these pixel points and accurately extract their horizontal coordinate extreme values ​​and vertical coordinate extreme values. For example, assuming that the tracing route is a "cross", the system will find the minimum and maximum values ​​of the horizontal coordinate and the minimum and maximum values ​​of the vertical coordinate. These extreme value coordinates will serve as the key data basis for the subsequent determination of the intermediate positioning point.

[0057] S3332: Determine the middle value of the horizontal coordinate and the middle value of the vertical coordinate based on the extreme value of the horizontal coordinate and the extreme value of the vertical coordinate to obtain a positioning middle point.

[0058] Based on the extreme values ​​of the horizontal and vertical coordinates extracted in the previous step, the system will perform precise calculations. For the horizontal coordinate, add the maximum and minimum values ​​and divide by 2 to get the middle value of the horizontal coordinate. Similarly, for the vertical coordinate, get the middle value of the vertical coordinate. The coordinate point formed by the combination of these two middle values ​​is the positioning middle point. The positioning middle point represents the center position of the user's operation trajectory and provides a core reference for the subsequent determination of the area.

[0059] S3333, determine the initial segmentation area of ​​the path record layer where the positioning middle point is located, and output the corresponding output language result according to the initial segmentation area where the positioning middle point is located.

[0060] The initial segmentation area is, for example, to divide the path record layer into multiple areas, so as to obtain multiple initial segmentation areas. This solution will output the corresponding output language result in combination with the initial segmentation area where the positioning intermediate point is located.

[0061] It is worth mentioning that for blind users, obtaining image information has always been a difficult problem. With the help of the language model dynamic processing method of the present invention, blind users can interact with images by touching the screen and other means. In the field of children's education, children have a strong demand for understanding and exploring images. Taking an image of a fairy tale scene as an example, a child uses a device equipped with this system to view the image. This solution is not limited to the above example.

[0062] See also Figure 2 In some embodiments, after obtaining the tracing route, the present solution may further perform the following steps, including: Determine whether the first pixel point and the second pixel point in the trigger path recording layer are continuously triggered or segmented triggered. Determine whether the first pixel point and the second pixel point in the trigger path recording layer are continuously triggered or segmented triggered. The system makes the judgment by analyzing the spatial position relationship of the pixels in the path recording layer. If the spatial positions of adjacent pixels are adjacent (for example, within a certain pixel distance threshold), it is determined to be a continuous trigger; otherwise, it is determined to be a segmented trigger.

[0063] If it is a continuous trigger, the tracking route is used as the last determined tracking route. If it is a continuous trigger, the tracking route is used as the last determined tracking route. This means that the user's operation is coherent and no further processing of the path is required.

[0064] If it is a segmented trigger, determine each segmented trigger path segment, sort them according to the generation order of the segmented trigger path segment, and obtain a path segment sequence. If it is a segmented trigger, determine each segmented trigger path segment, sort them according to the generation order of the segmented trigger path segment, and obtain a path segment sequence. For example, if the user first draws a straight line, pauses for a while, and then draws another straight line, the system will identify these two segmented trigger path segments and sort them according to their generation time.

[0065] The adjacent segmented trigger path segments in the path segment sequence are grouped in pairs, and the last pixel point of the previous segmented trigger path segment in each group is determined as the breakaway pixel point, and the first pixel point of the next segmented trigger path segment in each group is determined as the start pixel point. The adjacent segmented trigger path segments in the path segment sequence are grouped in pairs, and the last pixel point of the previous segmented trigger path segment in each group is determined as the breakaway pixel point, and the first pixel point of the next segmented trigger path segment in each group is determined as the start pixel point.

[0066] The route between the detached pixel point and the starting pixel point in the corresponding group in the tracing route is updated as a dotted line or deleted to obtain the final tracing route. The route between the detached pixel point and the starting pixel point in the corresponding group in the tracing route is updated as a dotted line or deleted to obtain the final tracing route. Through this processing method, the user's effective operation path can be presented more clearly, avoiding path confusion caused by pauses or misoperations during the user's operation, which affects subsequent instruction recognition and language result output.

[0067] In the above embodiment, the initial segmentation area of ​​the path record layer is obtained by the following steps, including: Get the number of horizontal and vertical pixels in the path record layer. To determine the initial segmentation area of ​​the path record layer where the positioning middle point is located, the system first obtains the number of horizontal and vertical pixels in the path record layer. Assume that the number of horizontal pixels in the path record layer is 800 and the number of vertical pixels is 600.

[0068] The number of horizontal pixel points is equally divided according to the preset horizontal equal division value to obtain a plurality of horizontal equally divided points, and the number of vertical pixel points is equally divided according to the preset vertical equal division value to obtain a plurality of vertical equally divided points. Exemplarily, the preset horizontal equal division value is set to 8, and the preset vertical equal division value is set to 6. Then the number of horizontal pixel points is equally divided, 800÷8 = 100, and 8 horizontal equally divided points are obtained, and each equally divided point is separated by 100 pixels; the number of vertical pixel points is equally divided, 600÷6 = 100, and 6 vertical equally divided points are obtained, and each equally divided point is separated by 100 pixels.

[0069] Based on the horizontal and vertical dividing points, the dividing lines are generated to divide the path record layer into equal parts to obtain the initial segmentation area of ​​the path record layer, and corresponding language orientation information is added to each initial segmentation area. The output language result includes the language orientation information.

[0070] Based on these horizontal and vertical dividing points, the system generates equal dividing lines and divides the path record layer into 48 initial segmentation areas. At the same time, the system will add corresponding language orientation information to each initial segmentation area, such as the upper left corner area can be marked as "upper left corner area", the upper right corner area can be marked as "upper right corner area", etc. By judging the coordinates of the positioning middle point (115, 165), it can be known that it is located in the second horizontal segmentation area from left to right and the second vertical segmentation area from top to bottom, and the corresponding language orientation information is "upper left middle area". The system outputs the corresponding output language result based on this information, such as "the position you are operating is located in the upper left middle area of ​​the image."

[0071] In some other embodiments, extracting the dynamic interactive content corresponding to the current instruction based on the first instruction includes: If the preset response strategy of the first instruction is the content recognition strategy, the horizontal coordinate extreme values ​​and the vertical coordinate extreme values ​​of all the first pixel points and the second pixel points in the tracing route are extracted.

[0072] If the preset response strategy of the first instruction is the content recognition strategy, the system will also extract the horizontal coordinate extreme values ​​and the vertical coordinate extreme values ​​of all the first pixel points and the second pixel points in the tracking route.

[0073] A selected area is generated based on the abscissa extreme value and the ordinate extreme value.

[0074] For example, suppose the coordinates of the pixel points on the tracing route determine that the extreme values ​​of the horizontal coordinate are [90, 140] and the extreme values ​​of the vertical coordinate are [130, 170]. Based on these extreme values, the system generates a selected area, which on the path record layer is a rectangular area with horizontal coordinates from 90 to 140 and vertical coordinates from 130 to 170.

[0075] The pixels in the selected area are mapped to the first file to determine the dynamic interactive content.

[0076] The system maps the pixels in the generated selected area to the first file (i.e., the original image). Since there is a corresponding relationship between the coordinates of the path record layer and the original image, through this mapping, the system can determine the part of the original image that corresponds to the selected area. For example, in the "park scene" image, the selected area may cover the bench and the surrounding grass. The system uses this part of the image content as dynamic interactive content to provide a basis for the subsequent determination of the output language results, such as analyzing the image elements in the area to determine the intersection with the "bench" image elements, and further determining the final output language results based on the intersection.

[0077] S4: Determine at least one content or at least one content area to output a language result based on the dynamic interactive content.

[0078] In the language model dynamic processing flow of the present invention, step S4 is the key link to convert the dynamic interaction content determined in the early stage into a language result understandable to the end user. Through the careful analysis of the selected area, image elements, and content area, the system can accurately output a language description that meets the user's interaction intention, realizing the key conversion from image interaction operation to natural language feedback.

[0079] In some embodiments, the determining, based on the dynamic interactive content, at least one content or at least one content area output language result comprises: S41, obtaining a first image element that intersects with the coordinate position of the pixel point in the selected area, determining the number of the intersecting pixel points to obtain a first intersection number, and counting a first total number of the pixel points in the first image element.

[0080] When the system determines that it needs to output language results based on dynamic interactive content, it will first focus on the selected area. In the scenario based on the content recognition strategy, the system will obtain the first image element that has an intersection with the coordinate position of the pixel point in the selected area. For example, in the "park scene" image, if the coordinate range of the selected area is the horizontal coordinate [110, 160] and the vertical coordinate [170, 210], the system will use the coordinate matching algorithm to retrieve the first image element that has been identified in the preset element database and find that the "bench" image element has an intersection with the selected area. Then, the system will accurately count the number of pixels in the intersection to obtain the first intersection number. Assume that through traversal calculation, it is determined that the number of pixels in the intersection is 600. At the same time, the system will count the first total number of pixels in the first image element "bench", which is assumed to be 1000. These quantitative statistical data provide a quantitative basis for subsequent proportion calculation and language result judgment.

[0081] S42, calculating the ratio of the first intersection quantity to the first total quantity to obtain a first proportion, and if the first proportion is greater than a preset value, taking the content of the corresponding first image element as the output language result.

[0082] The system will calculate the ratio of the first intersection number to the first total number, that is, 600÷1000 = 0.6, to obtain the first proportion. The preset value can be flexibly set according to the actual application scenario and needs, for example, set to 0.5. When the first proportion is greater than the preset value, it means that the selected area has a high degree of overlap with the first image element, which can more accurately reflect the user's attention to the image element. At this time, the system will use the content of the corresponding first image element as the output language result. Taking the "bench" image element as an example, the system may output "This is a brown wooden bench for two people to rest." Such a language description accurately feedbacks the image element information that the user is concerned about.

[0083] S43: If it is determined that all first proportions are less than or equal to a preset value, a content area is determined to output a language result.

[0084] If it is determined that all first proportions are less than or equal to the preset value, it means that the overlap between the selected area and a single image element is not high enough and cannot clearly point to a specific image element.

[0085] The step of determining a content area output language result includes: S431, obtaining a content area that intersects with the pixel coordinate positions of the selected area, and determining the number of intersecting pixels to obtain a second intersection number.

[0086] The system changes its strategy to obtain content areas that intersect with the pixel coordinates of the selected area. For example, in the "park scene" image, in addition to the "bench" image element, there may also be a content area containing a "flower bed" that intersects with the selected area. The system determines the number of pixels in these intersections and obtains the second intersection number.

[0087] S432: If it is determined that there is only one content area that has an intersection with the selected area, the corresponding content area is determined to output a language result.

[0088] If it is determined that there is only one content area that intersects with the selected area, the system will output the language result for the corresponding content area. For example, if the content area contains "bench", the system may output "the area contains a bench".

[0089] S433: If it is determined that there are multiple content areas that have intersections with the selected area, select a content area with a larger number of second intersections to output a language result, and the output language result includes all the content in the content area.

[0090] If it is determined that there are multiple content areas that have intersections with the selected area, the system will further compare the degree of intersection between these content areas and the selected area.

[0091] By comparing the second intersection number, the content area with the larger second intersection number is selected to output the language result. Assuming that there is a content area containing "bench" with a second intersection number of 400, and another content area containing "flowers" with a second intersection number of 300, the system will select the content area containing "bench" with a larger second intersection number to output the language result, and the output language result will include all the content in the content area, such as "this area contains a bench", thereby providing language feedback to the user.

[0092] See also Figure 3 , is a structural diagram of a language model dynamic processing system provided by an embodiment of the present invention, the system comprising: a judgment module, configured to generate a plurality of content regions by segmenting the first file according to the content after judging that the user chooses to perform language model dynamic processing on the first file of the preset modality, wherein the first file is an image; A generation module, used for generating a path record layer corresponding to the first file; A determination module, for determining the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel points after determining that the user has triggered any pixel point in the path record layer; The output module is used to determine at least one content or at least one content area based on the dynamic interactive content and output a language result.

[0093] The present invention also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the methods provided by the various embodiments described above.

[0094] Among them, the storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, the storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also exist in a communication device as discrete components. The storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0095] The present invention also provides a program product, which includes an execution instruction, which is stored in a storage medium. At least one processor of a device can read the execution instruction from the storage medium, and at least one processor executes the execution instruction so that the device implements the methods provided in the above various embodiments.

[0096] In the above-mentioned terminal or server embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A language model dynamic processing method, characterized in that: include: After determining that the user chooses to perform language model dynamic processing on a first file of a preset modality, the first file is segmented according to content to generate a plurality of content areas, wherein the first file is an image; Generate a path record layer corresponding to the first file; After determining that a user triggers any pixel point in the path record layer, the dynamic interaction content is determined based on the spatiotemporal relationship of the triggered pixel point; At least one content or at least one content area outputs a language result based on the dynamic interactive content.

2. The method according to claim 1, characterized in that The step of segmenting the first file according to the content to generate a plurality of content areas includes: Performing graphic comparison on the first file based on a preset element database to determine a first image element in the first file that passes the comparison; After coordinate processing of the first file, a coordinate interval of each first image element is obtained, and after processing the coordinate interval, a content coordinate interval of the first image element is generated; All pixels within the content coordinate interval are regarded as the corresponding content area.

3. The method according to claim 2, characterized in that The processing of the coordinate interval to generate a content coordinate interval of the first image element includes: Get the horizontal and vertical extreme values ​​of the coordinate interval; The maximum value among the extreme values ​​of the horizontal coordinate and the maximum value among the extreme values ​​of the vertical coordinate are increased by a preset value, and the minimum value among the extreme values ​​of the vertical coordinate and the minimum value among the extreme values ​​of the vertical coordinate are decreased by a preset value to obtain a content coordinate interval.

4. The method according to claim 2, characterized in that: The generating of the path record layer corresponding to the first file includes: Get the coordinates of each pixel in the first file; Establish the layer coordinates corresponding to the coordinates of each pixel point and generate a path record layer.

5. The method according to claim 2, characterized in that: After determining that a user triggers any pixel point in the path record layer, determining the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel point includes: Acquire the first pixel point in the trigger path record layer in real time, and extract the second pixel points of all historical triggers within a preset time period; Generate a pixel point sequence based on the time sequence of the first pixel point and the second pixel point, and sort the first pixel point and the second pixel point from near to far in time based on the pixel point sequence, so that the first pixel point and the second pixel point have corresponding sorting numbers respectively; The first pixel point and the second pixel point are traced in sequence based on the sequence number to determine the dynamic interactive content.

6. The method according to claim 5, characterized in that The step of tracking the first pixel point and the second pixel point in sequence based on the sequence number to determine the dynamic interactive content includes: Connecting the first pixel point and the second pixel point in sequence based on the sequence number to generate a tracing route; Extracting the graph formed by the tracing route and comparing it with the preset instruction database, determining the first instruction that passes the comparison, each instruction having a preset response strategy; The dynamic interactive content corresponding to the current instruction is extracted based on the first instruction.

7. The method according to claim 6, characterized in that The extracting the dynamic interactive content corresponding to the current instruction based on the first instruction includes: If the preset response strategy of the first instruction is a positioning strategy, extracting the extreme values ​​of the horizontal coordinates and the vertical coordinates of all the first pixel points and the second pixel points in the tracing route; Determine the middle value of the abscissa and the middle value of the ordinate based on the extreme value of the abscissa and the extreme value of the ordinate to obtain a positioning middle point; Determine the initial segmentation area of ​​the path record layer where the positioning intermediate point is located, and output the corresponding output language result according to the initial segmentation area where the positioning intermediate point is located.

8. The method according to claim 7, characterized in that The initial segmentation area of ​​the path record layer is obtained by the following steps, including: Get the number of horizontal and vertical pixels in the path record layer; The number of horizontal pixel points is equally divided according to a preset horizontal equal division value to obtain a plurality of horizontal equally divided points, and the number of vertical pixel points is equally divided according to a preset vertical equal division value to obtain a plurality of vertical equally divided points; Based on the horizontal and vertical dividing points, the dividing lines are generated to divide the path record layer into equal parts to obtain the initial segmentation area of ​​the path record layer, and the corresponding language orientation information is added to each initial segmentation area. The output language result includes the language orientation information.

9. The method according to claim 6, characterized in that The extracting the dynamic interactive content corresponding to the current instruction based on the first instruction includes: If the preset response strategy of the first instruction is a content recognition strategy, extracting the extreme values ​​of the horizontal coordinates and the vertical coordinates of all the first pixel points and the second pixel points in the tracing route; Generate a selected area based on the abscissa extreme value and the ordinate extreme value; The pixels in the selected area are mapped to the first file to determine the dynamic interactive content.

10. The method according to claim 9, characterized in that The step of determining at least one content or at least one content area output language result based on the dynamic interactive content comprises: Acquire a first image element that intersects with the coordinate position of the pixel point in the selected area, determine the number of the pixel points of the intersection to obtain a first intersection number, and count a first total number of the pixel points in the first image element; Calculating a ratio of the first intersection quantity to the first total quantity to obtain a first proportion, and if the first proportion is greater than a preset value, using the content of the corresponding first image element as the output language result; If it is determined that all first proportions are less than or equal to a preset value, a content area is determined to output a language result.

11. The method according to claim 10, characterized in that The step of determining a content area output language result includes: Obtaining a content area that intersects with the pixel coordinate positions of the selected area, and determining the number of intersecting pixels to obtain a second intersection number; If it is determined that there is only one content area that has an intersection with the selected area, the corresponding content area will be determined to output the language result; If it is determined that there are multiple content areas that have intersections with the selected area, then the content area with the second largest number of intersections is selected to output the language result, and the output language result includes all the content in the content area.

12. A language model dynamic processing system, characterized in that: a judgment module, configured to generate a plurality of content regions by segmenting the first file according to the content after judging that the user chooses to perform language model dynamic processing on the first file of the preset modality, wherein the first file is an image; A generation module, used for generating a path record layer corresponding to the first file; A determination module, for determining the dynamic interaction content based on the spatiotemporal relationship of the triggered pixel points after determining that the user has triggered any pixel point in the path record layer; The output module is used to determine at least one content or at least one content area based on the dynamic interactive content and output a language result.

13. A storage medium, characterized in that The storage medium stores a computer program, which is used to implement the method according to any one of claims 1 to 11 when executed by a processor.

Images