Data processing method and device, equipment and storage medium

Abstract

This disclosure provides a data processing method, apparatus, device, and storage medium applicable to the fields of big data and data retrieval. The method includes: determining the request type of initial request data; extracting retrieval data from the initial request data if the request type is a retrieval request type; determining a target retrieval container from multiple retrieval containers based on index update attribute information of the target index data corresponding to the retrieval data and the container update time attributes of each of the multiple retrieval containers, wherein the target retrieval container is suitable for performing a retrieval task based on the retrieval data; and processing the retrieval data based on the target retrieval container to obtain the target retrieval result corresponding to the initial request data.

Description

Technical Field

[0001] This disclosure relates to the fields of big data and data retrieval, and in particular to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of technology, businesses are increasingly relying on data retrieval services to retrieve data. These services can return search results to clients based on search keywords. For example, they can perform searches based on keywords and return documents, reports, and other files containing those keywords, helping users quickly find the data they need.

[0003] The inventors discovered that the retrieval efficiency of related search services was low, and there were potential issues such as incorrect search results, making it difficult to meet the actual needs of users. Summary of the Invention

[0004] In view of the above problems, this disclosure provides data processing methods, apparatus, devices, media and program products.

[0005] According to a first aspect of this disclosure, a data processing method is provided, comprising:

[0006] Determine the request type for the initial request data;

[0007] If the above request type is a retrieval request type, extract the retrieval data from the above initial request data;

[0008] Based on the index update attribute information of the target index data corresponding to the aforementioned search data, and the container update time attributes of each of the multiple search containers, a target search container is determined from the multiple search containers, wherein the target search container is suitable for performing a search task based on the aforementioned search data; and

[0009] The search data is processed by the target search container to obtain the target search results corresponding to the initial request data.

[0010] According to embodiments of this disclosure, the plurality of search containers include a source search container and at least one mirror search container, and the target search container is determined from at least one of the mirror search containers.

[0011] The above data processing methods also include:

[0012] If the above request type is an index update request type, extract the updated index data from the above initial request data;

[0013] Based on the updated index data, update the source retrieval container to obtain the updated target source retrieval container; and

[0014] The target source retrieval container is mirrored to generate a new mirror retrieval container.

[0015] According to embodiments of this disclosure, the index update attribute information includes the index update frequency, the container update time attribute includes the container generation time of generating the image retrieval container, and the image retrieval container includes multiple containers.

[0016] The process of determining the target retrieval container from the multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data, and the container update time attributes of each of the multiple retrieval containers, includes:

[0017] By comparing the index update frequency of the target index data with a preset update frequency threshold, a first comparison result is obtained, wherein the first comparison result represents the update frequency level of the target index data;

[0018] Based on the first comparison result above, the target container's creation time is determined from the creation times of the respective image containers of the multiple image retrieval containers mentioned above; and

[0019] Based on the time of the target container's creation, the target retrieval container is determined from among the multiple image retrieval containers.

[0020] According to embodiments of this disclosure, the index update attribute information includes the index update time, the container update time attribute includes the container generation time of generating the image retrieval container, and the image retrieval container includes multiple containers.

[0021] The process of determining the target retrieval container from the multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data, and the container update time attributes of each of the multiple retrieval containers, includes:

[0022] When the target index data is associated with multiple index update times, the target index update time closest to the current time is determined from among the multiple index update times;

[0023] Based on the target index update time mentioned above, the target container generation time is determined from the respective image container generation times of the multiple image retrieval containers mentioned above; and

[0024] Based on the time of the target container's creation, the target retrieval container is determined from among the multiple image retrieval containers.

[0025] According to embodiments of this disclosure, determining the target container generation time from the respective image container generation times of the plurality of image retrieval containers based on the target index update time includes:

[0026] Among the multiple image container generation times mentioned above, those image container generation times that are later than the target index update time are identified as candidate container generation times; and

[0027] When there are multiple candidate container generation times, the candidate container generation time whose duration between it and the target index update time is less than a preset duration threshold is determined as the target container generation time.

[0028] According to embodiments of this disclosure, the above data processing method further includes:

[0029] Based on the container update time attribute of each of the multiple retrieval containers, determine the duration of each retrieval container to be updated;

[0030] Based on the respective pending update duration of multiple search containers, the search containers to be deleted are determined from the multiple search containers mentioned above; and

[0031] Delete the above-mentioned search containers that are to be deleted.

[0032] According to embodiments of this disclosure, the request type for determining the initial request data includes:

[0033] Parse the initial request header data of the above initial request data to obtain the above request type.

[0034] A second aspect of this disclosure provides a data processing apparatus, comprising:

[0035] The first determining module is used to determine the request type of the initial request data;

[0036] The extraction module is used to extract the retrieval data from the initial request data when the above request type is a retrieval request type.

[0037] The second determining module is used to determine a target retrieval container from the multiple retrieval containers based on the index update attribute information of the target index data corresponding to the aforementioned retrieval data and the container update time attributes of each of the multiple retrieval containers, wherein the target retrieval container is suitable for performing a retrieval task based on the aforementioned retrieval data; and

[0038] The target retrieval result acquisition module is used to process the retrieval data based on the target retrieval container to obtain the target retrieval result corresponding to the initial request data.

[0039] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0040] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0041] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0042] According to the data processing method, apparatus, device, medium, and program product provided in this disclosure, by determining the target retrieval container suitable for performing retrieval tasks based on the index update attribute information of the target index data and the container update time attributes of each of the multiple retrieval containers, the container update time attribute of the target retrieval container can be adapted to the index update attribute information. This allows for the adaptive selection of a target retrieval container adapted to the retrieval data to perform the retrieval task, at least partially avoiding the technical problems of response delay and retrieval result errors caused by performing retrieval tasks through the same retrieval server in the case of high concurrency retrieval of massive request data, thereby improving retrieval efficiency. Attached Figure Description

[0043] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0044] Figure 1 This diagram illustrates an application scenario of the data processing method according to an embodiment of the present disclosure.

[0045] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0046] Figure 3 A flowchart illustrating the process of determining a target retrieval container from a plurality of retrieval containers according to an embodiment of the present disclosure is shown schematically.

[0047] Figure 4 This diagram illustrates an application scenario of the data processing method according to an embodiment of the present disclosure.

[0048] Figure 5 A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0049] Figure 6A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0050] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0051] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0052] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0053] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0054] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0055] To further and accurately explain the technical solutions provided in the embodiments of this disclosure, the following glossary of terms can be used to explain the technical terms provided in the embodiments of this disclosure.

[0056] Lucene: Lucene can refer to a full-text search engine development kit or a retrieval service resource. Lucene provides application programming interfaces (APIs) for query engines, indexing engines, and text analysis engines, enabling full-text indexing and searching.

[0057] Index, or indexed data:

[0058] In Lucene, index data can be stored in a folder. The index data in the same folder constitutes a Lucene index file.

[0059] Segment: An index file can contain multiple segments. Segments are independent of each other. Adding new documents generates new segments, and different segments can be merged. Files with the same prefix can belong to the same segment.

[0060] Document: A document is the basic unit for building index data. Different documents are stored in different segments, and a segment can contain multiple documents. Newly added documents are stored separately in a newly generated segment. As segments are merged, different documents are merged into the same segment.

[0061] Field: A document can contain different types of information, which can be indexed and retrieved separately. For example, index data based on title, time, body text, author, etc. can be stored in different fields for separate retrieval. Different fields can be indexed in different ways.

[0062] Term: A term can be the smallest unit of indexed data, and it can be a string after lexical analysis and language processing.

[0063] After a document is segmented by an analyzer, it becomes terms. These terms can be associated with document identifiers. Multiple terms are sorted, and identical terms are merged and their frequencies are calculated, along with the document identifiers of those terms. During the retrieval process, Lucene can save a term dictionary, a frequency file, and a position file. The term dictionary not only stores each keyword but also contains pointers to the frequency and position files, allowing users to find the keyword's frequency and position information.

[0064] When performing a search based on index data, for example, to find the word "sales", Lucene first performs a binary search on the dictionary to locate "sales", then reads all article numbers from the frequency file using a pointer, and finally returns the results. Since the dictionary is typically very small, the entire process takes only milliseconds.

[0065] Mirroring is a type of redundancy where data on one disk is mirrored to create an exact copy on another disk. Similarly, by mirroring the source data, a completely identical copy can be obtained.

[0066] The inventors made a groundbreaking discovery: Search services like Lucene, during the execution of search tasks based on index data, experience high-concurrency data I / O (input / output) during index data updates and search task execution. This can lead to synchronization issues among different Lucene replicas, resulting in inconsistent search results, as well as problems such as search blocking and slow response times.

[0067] Embodiments of this disclosure provide a data processing method, apparatus, device, medium, and program product. The data processing method includes: determining the request type of initial request data; if the request type is a retrieval request type, extracting retrieval data from the initial request data; determining a target retrieval container from a plurality of retrieval containers based on index update attribute information of target index data corresponding to the retrieval data and container update time attributes of each of the plurality of retrieval containers, wherein the target retrieval container is adapted to perform a retrieval task based on the retrieval data; and processing the retrieval data based on the target retrieval container to obtain a target retrieval result corresponding to the initial request data.

[0068] According to embodiments of this disclosure, by determining the target retrieval container suitable for performing retrieval tasks based on the index update attribute information of the target index data and the container update time attributes of each of the multiple retrieval containers, the container update time attribute of the target retrieval container can be adapted to the index update attribute information. This allows for the adaptive selection of the target retrieval container that is compatible with the retrieval data to perform the retrieval task, achieving dynamic scheduling and load balancing of retrieval resources. This at least partially avoids the technical problems of response delay and retrieval result errors caused by performing retrieval tasks through the same retrieval server in the case of high concurrency retrieval of massive request data, thereby improving retrieval efficiency.

[0069] Figure 1 The diagram illustrates an application scenario of the data processing method according to an embodiment of the present disclosure.

[0070] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0071] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0072] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0073] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0074] It should be noted that the data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the data processing device provided in this embodiment can generally be located in server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0075] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0076] The following will be based on Figure 1 The described scene, through Figures 2-4 The data processing method of the disclosed embodiments will be described in detail.

[0077] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0078] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S240.

[0079] In operation S210, the request type of the initial request data is determined.

[0080] According to embodiments of this disclosure, the request type can be determined by parsing the initial request data, but it is not limited thereto. The request type can also be determined by parsing the header of the initial request data. Embodiments of this disclosure do not limit this to the above.

[0081] In operation S220, if the request type is a retrieval request type, the retrieval data is extracted from the initial request data.

[0082] According to embodiments of this disclosure, the retrieval data may be retrieval keywords, retrieval terms, retrieval characters, etc. in the initial request data. Embodiments of this disclosure do not limit the specific type of retrieval data.

[0083] In operation S230, based on the index update attribute information of the target index data corresponding to the search data and the container update time attribute of each of the multiple search containers, a target search container is determined from the multiple search containers, wherein the target search container is suitable for performing search tasks based on the search data.

[0084] According to embodiments of this disclosure, index update attribute information can be information related to update attributes such as index update time and frequency. Index update attribute information can be, for example, the index update time, but is not limited to this; it can also be other types of index update attribute information such as index update frequency and index update importance level. Embodiments of this disclosure do not limit the specific type of index update attribute information, and those skilled in the art can select it according to actual needs.

[0085] According to embodiments of this disclosure, a retrieval container can be a service resource capable of performing retrieval tasks based on retrieval data and returning retrieval results corresponding to the retrieval data. The container update time attribute can include time attributes such as the update time of the corresponding retrieval container and the duration after the update. It should be understood that the update time of the retrieval container can be the time when the updated retrieval container is generated after updating the source retrieval container based on the configuration file or index. However, it is not limited to this; it can also be a retrieval container generated based on a mirror of the updated new retrieval container.

[0086] In operation S240, the retrieval data is processed based on the target retrieval container to obtain the target retrieval results corresponding to the initial request data.

[0087] According to embodiments of this disclosure, the container update time attribute of each retrieval container can be used to characterize the duration of the retrieval container after an update. Therefore, by determining a container update time attribute that matches the index update attribute of the target index, and further obtaining a target retrieval container with that container update time attribute, it is possible to adapt the retrieval resource (i.e., the target retrieval container) that is suitable for implementing the retrieval task corresponding to the retrieval data to the retrieval task corresponding to the initial request data. This allows for flexible and adaptive allocation of retrieval tasks to multiple retrieval containers, at least avoiding the direct completion of all retrieval tasks through the latest updated retrieval container, and evenly distributing the load generated by the retrieval tasks.

[0088] According to embodiments of this disclosure, by determining the target retrieval container suitable for performing retrieval tasks based on the index update attribute information of the target index data and the container update time attributes of each of the multiple retrieval containers, the container update time attribute of the target retrieval container can be adapted to the index update attribute information. This allows for the adaptive selection of the target retrieval container that is compatible with the retrieval data to perform the retrieval task, achieving dynamic scheduling and load balancing of retrieval resources. This at least partially avoids the technical problems of response delay and retrieval result errors caused by performing retrieval tasks through the same retrieval server in the case of high concurrency retrieval of massive request data, thereby improving retrieval efficiency.

[0089] According to an embodiment of this disclosure, in operation S210, determining the request type of the initial request data may include: parsing the initial request header data of the initial request data to obtain the request type.

[0090] According to embodiments of this disclosure, obtaining the request type by parsing the initial request header data can improve the timeliness of determining the request type, thereby improving the efficiency of retrieval task execution.

[0091] According to embodiments of this disclosure, the plurality of retrieval containers include a source retrieval container and at least one mirror retrieval container, and the target retrieval container is determined from the at least one mirror retrieval container.

[0092] Data processing methods may also include the following operations:

[0093] When the request type is an index update request, extract the updated index data from the initial request data; update the source retrieval container based on the updated index data to obtain the updated target source retrieval container; and perform mirroring on the target source retrieval container to generate a new mirrored retrieval container.

[0094] According to embodiments of this disclosure, index data update operations such as adding, deleting, and modifying can be performed on the source index data in the source retrieval container based on the updated index data, thereby obtaining the target source retrieval container constructed based on the updated index data.

[0095] According to embodiments of this disclosure, after obtaining the target source retrieval container, a mirror image of the target source retrieval container can be generated to obtain a mirror retrieval container corresponding to the target source retrieval container. Since the target retrieval container used to perform the retrieval task is determined from the mirror retrieval container, multiple retrieval containers can implement the index data resource update operation process (which can be characterized as a retrieval resource read operation process) through the source retrieval container, and implement the retrieval task execution operation process based on the target index data (which can be characterized as a retrieval resource write operation process) through at least one mirror retrieval container. Therefore, the data processing method provided by embodiments of this disclosure can fully decouple the index data resource update operation of the retrieval resource from the retrieval task execution operation, improving the operating efficiency of the retrieval resource and simultaneously improving the accuracy of the retrieval task.

[0096] According to embodiments of this disclosure, the index update attribute information includes the index update frequency, and the container update time attribute includes the container generation time of the generated image retrieval container, and the image retrieval container includes multiple containers.

[0097] In operation S230, determining the target retrieval container from multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data and the container update time attribute of each of the multiple retrieval containers may include the following operations.

[0098] The index update frequency of the target index data is compared with a preset update frequency threshold to obtain a first comparison result, wherein the first comparison result represents the update frequency level of the target index data; based on the first comparison result, the target container generation time is determined from the image container generation times of the multiple image retrieval containers; and based on the target container generation time, the target retrieval container is determined from the multiple image retrieval containers.

[0099] According to embodiments of this disclosure, the preset update frequency threshold may include one or more update frequency threshold data. By comparing the index update frequency of the target index data with one or more update frequency threshold data, it can be determined that the index update frequency of the target index data is in an update frequency region composed of different update frequency threshold data. The update frequency region represents the update frequency level, and then the update frequency level corresponding to the target index data can be mapped to the corresponding target container generation time, thereby realizing the determination of the target retrieval container from multiple mirror retrieval containers.

[0100] According to embodiments of this disclosure, by determining the target retrieval container whose update frequency matches that of the target index data based on the update frequency level of the target index data represented by the first comparison result, a mirror retrieval container whose creation time is closer to the current time can be selected to perform the retrieval task when the target index data corresponding to the retrieval data is updated frequently. Correspondingly, when the target index data corresponding to the retrieval data is updated infrequently, a mirror retrieval container whose creation time is farther from the current time can be selected to perform the retrieval task. This effectively adapts to the retrieval requirements corresponding to the retrieval data in the initial request data, enabling adaptive allocation of retrieval tasks to multiple mirror retrieval containers. This avoids excessive concentration of retrieval tasks on the newly generated mirror retrieval container and at least prevents retrieval tasks from being assigned to expired retrieval resources, thereby ensuring retrieval timeliness and accuracy while improving retrieval efficiency.

[0101] Figure 3 A flowchart illustrating the process of determining a target retrieval container from a plurality of retrieval containers according to an embodiment of the present disclosure is shown.

[0102] According to embodiments of this disclosure, the index update attribute information includes the index update time, and the container update time attribute includes the container generation time of the generated image retrieval container, and the image retrieval container includes multiple containers.

[0103] like Figure 3 As shown, in operation S230, determining the target retrieval container from multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data and the container update time attribute of each of the multiple retrieval containers may include operations S310 to S330.

[0104] In operation S310, when the target index data is associated with multiple index update times, the target index update time closest to the current time is determined from the multiple index update times.

[0105] In operation S320, the target container generation time is determined from the respective image container generation times of multiple image retrieval containers based on the target index update time.

[0106] In operation S330, the target container is determined from multiple image retrieval containers based on the target container's creation time.

[0107] According to embodiments of this disclosure, the target container generation time is determined from the image container generation times of multiple image retrieval containers based on the target index update time. This can be achieved by selecting the time closest to the target index update time among the multiple image container generation times, or by selecting a time period less than a preset duration threshold among the time periods between the multiple image container generation times and the target index update time, and then determining one or more target container generation times based on the target time period.

[0108] According to embodiments of this disclosure, the target retrieval container is determined from multiple mirror retrieval containers based on the target container's generation time. This allows for more precise determination of the target retrieval container that matches the retrieval data, thereby further improving the accuracy of determining retrieval resources and enhancing the precision and timeliness of subsequent retrieval results.

[0109] According to an embodiment of this disclosure, in operation S320, determining the target container generation time from the respective image container generation times of multiple image retrieval containers based on the target index update time may include the following operations.

[0110] Among multiple image container generation times, the image container generation time that is later than the target index update time is determined as the candidate container generation time; and when there are multiple candidate container generation times, the candidate container generation time whose duration between the candidate container generation time and the target index update time is less than a preset duration threshold is determined as the target container generation time.

[0111] According to embodiments of this disclosure, the candidate container is generated later than the target index update time. Therefore, it can be determined that the candidate retrieval container corresponding to the candidate container generation time is a mirror retrieval container generated after the last update of the target index data. This results in mirror retrieval containers that all contain the most recently updated target index data. Furthermore, by selecting the candidate container generation times from multiple candidate container generation times whose time interval with the target index update time is less than a preset time threshold as the target container generation times, the most recently updated mirror retrieval containers can be obtained. This at least avoids the same mirror retrieval container executing multiple retrieval tasks corresponding to different retrieval data, improving the matching accuracy between the target retrieval container and the retrieval data, and simultaneously improving the accuracy of subsequent target retrieval results.

[0112] According to embodiments of this disclosure, the data processing method may further include the following operations.

[0113] Based on the container update time attribute of each of the multiple search containers, determine the duration to be updated for each of the multiple search containers; based on the duration to be updated for each of the multiple search containers, determine the search containers to be deleted from the multiple search containers; and delete the search containers to be deleted.

[0114] According to embodiments of this disclosure, expired search resources can be accurately removed by deleting search containers whose pending update duration exceeds a preset pending update duration threshold. This avoids identifying expired mirror search containers as target search containers, which could lead to search errors in the target search results.

[0115] Figure 4 The diagram illustrates an application scenario of the data processing method according to an embodiment of the present disclosure.

[0116] like Figure 4 As shown, in the revised application scenario 400, there are a client 410 and a retrieval server 420. The retrieval server 420 may include a front-end gateway module 421 and a retrieval service container cluster 422. The retrieval service container cluster 422 may include an active retrieval container 4220 and N image retrieval containers, where the N image retrieval containers may be the first image retrieval container 4221, the second image retrieval container 4222, ..., up to the nth image retrieval container 422n.

[0117] The retrieval service container cluster 422 can be a retrieval service resource built on Lucene. The mirrored retrieval container can be set up on a single server, or it can be set up in a server cluster consisting of multiple servers.

[0118] Client 410 can send initial request data to front-end gateway module 421 of retrieval server 420, and front-end gateway module 421 can determine the request type of the initial request data.

[0119] When the request type is an index update request, the updated index data in the initial request data can be sent to the source retrieval container 4220. This allows the source retrieval container 4220 to be updated based on the updated index data, resulting in the updated target source retrieval container. The target retrieval container can then be mirrored to generate a new mirrored retrieval container, i.e., the first mirrored retrieval container 4221. It should be understood that the second mirrored retrieval container 4222…up to the nth mirrored retrieval container 422n can all be generated in the same or similar manner as the first mirrored retrieval container 4221; this will not be elaborated further in the embodiments of this disclosure.

[0120] When the request type is a retrieval request, the retrieval data can be extracted from the initial request data. Based on the index update time of the target index data corresponding to the retrieval data, the target container generation time that matches the index update time of the target index data is determined from the generation times of the respective mirror containers of the first mirror retrieval container 4221, the second mirror retrieval container 4222, ..., up to the nth mirror retrieval container 422n. For example, the target container generation time could be the mirror container generation time of the second mirror retrieval container 4222, thus determining the second mirror retrieval container 4222 as the target retrieval container.

[0121] The retrieval data in the initial request data is pushed to the second mirror retrieval container 4222. The second mirror retrieval container 4222 can perform a retrieval task based on the retrieval data to obtain the target retrieval result corresponding to the initial request data. The target retrieval result can be returned to the client 410 through the front-end gateway module 421.

[0122] According to the data processing method provided in the embodiments of this disclosure, the request type of the initial request data is determined by the front-end gateway module 421, and the initial request data is distributed to different retrieval containers according to the request type. This can fully decouple the index data resource update operation of the retrieval resource from the retrieval task execution operation, improve the running efficiency of the retrieval resource, and reduce the load pressure of the same retrieval container.

[0123] Furthermore, due to the full decoupling of index data resource update operations and retrieval task execution operations, simultaneous read and write operations on the same retrieval container resource can be avoided, thus preventing retrieval congestion caused by simultaneous read and write operations on the retrieval container and improving the response speed of retrieval tasks.

[0124] Based on the above data processing method, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0125] Figure 5 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0126] like Figure 5 As shown, the data processing device 500 of this embodiment includes a first determining module 510, an extraction module 520, a second determining module 530, and a target retrieval result obtaining module 540.

[0127] The first determining module 510 is used to determine the request type of the initial request data.

[0128] The extraction module 520 is used to extract retrieval data from the initial request data when the request type is a retrieval request type.

[0129] The second determining module 530 is used to determine a target retrieval container from multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data and the container update time attribute of each of the multiple retrieval containers, wherein the target retrieval container is suitable for performing retrieval tasks based on the retrieval data.

[0130] The target retrieval result acquisition module 540 is used to process retrieval data based on the target retrieval container to obtain the target retrieval results corresponding to the initial request data.

[0131] According to embodiments of this disclosure, the plurality of retrieval containers include a source retrieval container and at least one mirror retrieval container, and the target retrieval container is determined from the at least one mirror retrieval container.

[0132] The data processing device also includes: a third determination module, a target source retrieval container acquisition module, and an image processing module.

[0133] The third determination module is used to extract the updated index data from the initial request data when the request type is an index update request type.

[0134] The target source retrieval container acquisition module is used to update the source retrieval container based on the updated index data, and obtain the updated target source retrieval container.

[0135] The image processing module is used to perform image processing on the target source retrieval container to generate a new image retrieval container.

[0136] According to embodiments of this disclosure, the index update attribute information includes the index update frequency, and the container update time attribute includes the container generation time of the generated image retrieval container, and the image retrieval container includes multiple containers.

[0137] The second determining module includes: a first comparison unit, a first target container generation time acquisition unit, and a first target retrieval container acquisition unit.

[0138] The first comparison unit is used to compare the index update frequency of the target index data with the preset update frequency threshold to obtain the first comparison result, wherein the first comparison result represents the update frequency level of the target index data.

[0139] The first target container generation time acquisition unit is used to determine the target container generation time from the respective image container generation times of multiple image retrieval containers based on the first comparison result.

[0140] The first target retrieval container acquisition unit is used to determine the target retrieval container from multiple mirror retrieval containers based on the target container's generation time.

[0141] According to embodiments of this disclosure, the index update attribute information includes the index update time, and the container update time attribute includes the container generation time of the generated image retrieval container, and the image retrieval container includes multiple containers.

[0142] The second determining module includes: a target index update time acquisition unit, a second target container generation time acquisition unit, and a second target retrieval container acquisition unit.

[0143] The target index update time acquisition unit is used to determine the target index update time closest to the current time from among the multiple index update times when the target index data is associated with multiple index update times.

[0144] The second target container generation time acquisition unit is used to determine the target container generation time from the respective image container generation times of multiple images based on the target index update time.

[0145] The second target retrieval container acquisition unit is used to determine the target retrieval container from multiple image retrieval containers based on the target container's generation time.

[0146] According to embodiments of this disclosure, the second target container generation time acquisition unit includes: a candidate container generation time acquisition subunit and a target container generation time acquisition subunit.

[0147] The candidate container generation time acquisition sub-unit is used to determine the image container generation time that is later than the target index update time among multiple image container generation times as the candidate container generation time.

[0148] The target container generation time acquisition sub-unit is used to determine the target container generation time when there are multiple candidate container generation times, and the candidate container generation time whose time interval between the candidate container generation time and the target index update time is less than a preset time threshold.

[0149] According to embodiments of this disclosure, the data processing apparatus further includes: a module for determining the duration to be updated, a module for determining the retrieval container to be deleted, and a deletion module.

[0150] The module for determining the duration of pending updates is used to determine the duration of pending updates for each of the multiple retrieval containers based on their respective container update time attributes.

[0151] The module for determining search containers to be deleted is used to determine the search containers to be deleted from multiple search containers based on the duration of each search container's pending update.

[0152] The delete module is used to delete the search containers that are to be deleted.

[0153] According to embodiments of this disclosure, the first determining module includes a parsing unit.

[0154] The parsing unit is used to parse the initial request header data of the initial request data to obtain the request type.

[0155] According to embodiments of this disclosure, any plurality of modules among the first determining module 510, the extraction module 520, the second determining module 530, and the target retrieval result obtaining module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first determining module 510, the extraction module 520, the second determining module 530, and the target retrieval result obtaining module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first determining module 510, the extraction module 520, the second determining module 530, and the target retrieval result obtaining module 540 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0156] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0157] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0158] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0159] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0160] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0161] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0162] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the item recommendation method provided in the embodiments of this disclosure.

[0163] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0164] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0165] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0166] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0168] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0169] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, comprising: Determine the request type for the initial request data; If the request type is a retrieval request type, extract the retrieval data from the initial request data; Based on the index update attribute information of the target index data corresponding to the search data, and the container update time attributes of each of the multiple search containers, a target search container is determined from the multiple search containers, wherein the target search container is suitable for performing a search task based on the search data; and The retrieval data is processed based on the target retrieval container to obtain the target retrieval result corresponding to the initial request data. The multiple retrieval containers include a source retrieval container and at least one mirror retrieval container. The target retrieval container is determined from at least one of the mirror retrieval containers. The index update attribute information includes the index update frequency. The container update time attribute includes the container generation time of generating the mirror retrieval container. The mirror retrieval containers include multiple containers. The step of determining the target retrieval container from the multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data and the container update time attributes of each of the multiple retrieval containers includes: The index update frequency of the target index data is compared with a preset update frequency threshold to obtain a first comparison result, wherein the first comparison result characterizes the update frequency level of the target index data; Based on the first comparison result, the target container's creation time is determined from the creation times of the respective image containers of the multiple image retrieval containers; and The target retrieval container is determined from among the multiple image retrieval containers based on the target container's generation time.

2. The method according to claim 1, wherein, The data processing method further includes: If the request type is an index update request type, extract the updated index data from the initial request data; Based on the updated index data, the source retrieval container is updated to obtain the updated target source retrieval container; and The target source retrieval container is mirrored to generate a new mirror retrieval container.

3. The method according to claim 2, wherein, The index update attribute information includes the index update time, and the container update time attribute includes the container generation time of the container that generated the image retrieval container; The step of determining the target retrieval container from the multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data and the container update time attributes of each of the multiple retrieval containers includes: When the target index data is associated with multiple index update times, the target index update time closest to the current time is determined from the multiple index update times; Based on the target index update time, the target container generation time is determined from the respective image container generation times of the multiple image retrieval containers; and The target retrieval container is determined from among the multiple image retrieval containers based on the target container's generation time.

4. The method according to claim 3, wherein, The step of determining the target container generation time from the respective image container generation times of the multiple image retrieval containers based on the target index update time includes: Among the multiple image container generation times, those image container generation times that are later than the target index update time are determined as candidate container generation times; and When there are multiple candidate container generation times, the candidate container generation time whose duration between it and the target index update time is less than a preset duration threshold is determined as the target container generation time.

5. The method according to claim 1, further comprising: Based on the container update time attribute of each of the multiple retrieval containers, determine the duration of each retrieval container to be updated; Based on the duration of each of the multiple search containers to be updated, a search container to be deleted is determined from the multiple search containers; as well as Delete the search container to be deleted.

6. The method according to claim 1, wherein, The request types for determining the initial request data include: The initial request header data of the initial request data is parsed to obtain the request type.

7. A data processing apparatus, comprising: The first determining module is used to determine the request type of the initial request data; The extraction module is used to extract retrieval data from the initial request data when the request type is a retrieval request type. The second determining module is configured to determine a target retrieval container from among the multiple retrieval containers based on the index update attribute information of the target index data corresponding to the retrieval data and the container update time attributes of each of the multiple retrieval containers, wherein the target retrieval container is adapted to perform a retrieval task based on the retrieval data; and The target retrieval result acquisition module is used to process the retrieval data based on the target retrieval container to obtain the target retrieval result corresponding to the initial request data. The multiple retrieval containers include a source retrieval container and at least one mirror retrieval container. The target retrieval container is determined from at least one of the mirror retrieval containers. The index update attribute information includes the index update frequency. The container update time attribute includes the container generation time of generating the mirror retrieval container. The mirror retrieval containers include multiple containers. The second determining module includes: The first comparison unit is used to compare the index update frequency of the target index data with a preset update frequency threshold to obtain a first comparison result, wherein the first comparison result represents the update frequency level of the target index data; The first target container generation time acquisition unit is used to determine the target container generation time from the respective image container generation times of the multiple image retrieval containers based on the first comparison result; and The first target retrieval container obtaining unit is used to determine the target retrieval container from a plurality of mirror retrieval containers according to the target container generation time.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images